Project

General

Profile

Bug #12470

Updated by Loïc Dachary over 8 years ago

Steps While running hammer v0.94.3 integration tests (currently tracked at http://tracker.ceph.com/issues/11990), a single radosgw test failed as radosgw-agent couldn't be installed
http://pulpito.ceph.com/loic-2015-07-20_16:46:17-rgw-hammer-backports---basic-multi/980731/

More detailed log below:
<pre>
2015-07-23T08:15:42.744 INFO:teuthology.orchestra.run.plana76:Running: 'cd /home/ubuntu/cephtest && cd radosgw-agent.client.0 && ./bootstrap'
2015-07-23T08:15:43.451 INFO:teuthology.orchestra.run.plana76.stdout:New python executable in virtualenv/bin/python
2015-07-23T08:15:45.344 INFO:teuthology.orchestra.run.plana76.stdout:Installing distribute.............................................................................................................................................................................................done.
2015-07-23T08:15:45.814 INFO:teuthology.orchestra.run.plana76.stdout:Installing pip...............done.
2015-07-23T08:15:45.884 INFO:teuthology.orchestra.run.plana76.stderr:/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_requires'
2015-07-23T08:15:45.885 INFO:teuthology.orchestra.run.plana76.stderr: warnings.warn(msg)
2015-07-23T08:15:45.903 INFO:teuthology.orchestra.run.plana76.stdout:running develop
2015-07-23T08:15:45.917 INFO:teuthology.orchestra.run.plana76.stdout:running egg_info
2015-07-23T08:15:45.917 INFO:teuthology.orchestra.run.plana76.stdout:creating radosgw_agent.egg-info
2015-07-23T08:15:45.918 INFO:teuthology.orchestra.run.plana76.stdout:writing requirements
to reproduce

* docker run -t -i ubuntu:12.04 bash
radosgw_agent.egg-info/requires.txt
* apt-get install git python-virtualenv 2015-07-23T08:15:45.918 INFO:teuthology.orchestra.run.plana76.stdout:writing radosgw_agent.egg-info/PKG-INFO
* git clone https://github.com/ceph/radosgw-agent 2015-07-23T08:15:45.919 INFO:teuthology.orchestra.run.plana76.stdout:writing top-level names to radosgw_agent.egg-info/top_level.txt
* cd 2015-07-23T08:15:45.919 INFO:teuthology.orchestra.run.plana76.stdout:writing dependency_links to radosgw_agent.egg-info/dependency_links.txt
2015-07-23T08:15:45.919 INFO:teuthology.orchestra.run.plana76.stdout:writing entry points to radosgw_agent.egg-info/entry_points.txt
2015-07-23T08:15:45.920 INFO:teuthology.orchestra.run.plana76.stdout:writing manifest file 'radosgw_agent.egg-info/SOURCES.txt'
2015-07-23T08:15:45.923 INFO:teuthology.orchestra.run.plana76.stdout:reading manifest file 'radosgw_agent.egg-info/SOURCES.txt'
2015-07-23T08:15:45.923 INFO:teuthology.orchestra.run.plana76.stdout:reading manifest template 'MANIFEST.in'
2015-07-23T08:15:45.926 INFO:teuthology.orchestra.run.plana76.stdout:writing manifest file 'radosgw_agent.egg-info/SOURCES.txt'
2015-07-23T08:15:45.929 INFO:teuthology.orchestra.run.plana76.stdout:running build_ext
2015-07-23T08:15:45.929 INFO:teuthology.orchestra.run.plana76.stdout:Creating /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/lib/python2.7/site-packages/radosgw-agent.egg-link (link to .)
2015-07-23T08:15:45.929 INFO:teuthology.orchestra.run.plana76.stdout:Adding
radosgw-agent 1.2.3 to easy-install.pth file
* ./bootstrap

Expected output:

Successfully installed PyYAML-3.11 funcsigs-0.4 mock-1.3.0 pbr-1.3.0 pluggy-0.3.0 py-1.4.30 pytest-2.7.2 six-1.9.0 tox-2.1.1 virtualenv-13.1.0

What happens instead

<pre>
2015-07-23T08:15:45.930 INFO:teuthology.orchestra.run.plana76.stdout:Installing radosgw-agent script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
Downloading/unpacking 2015-07-23T08:15:45.930 INFO:teuthology.orchestra.run.plana76.stdout:
2015-07-23T08:15:45.930 INFO:teuthology.orchestra.run.plana76.stdout:Installed /home/ubuntu/cephtest/radosgw-agent.client.0
2015-07-23T08:15:45.931 INFO:teuthology.orchestra.run.plana76.stdout:Processing dependencies for radosgw-agent==1.2.3
2015-07-23T08:15:45.931 INFO:teuthology.orchestra.run.plana76.stdout:Searching for boto>=2.2.2,<3.0.0
2015-07-23T08:15:45.931 INFO:teuthology.orchestra.run.plana76.stdout:Reading http://pypi.python.org/simple/boto/
2015-07-23T08:15:46.142 INFO:teuthology.orchestra.run.plana76.stdout:Best match: boto 2.38.0
2015-07-23T08:15:46.142 INFO:teuthology.orchestra.run.plana76.stdout:Downloading https://pypi.python.org/packages/source/b/boto/boto-2.38.0.tar.gz#md5=28112f29e9c7b10e12b6917a325e70ce
2015-07-23T08:15:46.222 INFO:teuthology.orchestra.run.plana76.stdout:Processing boto-2.38.0.tar.gz
2015-07-23T08:15:46.647 INFO:teuthology.orchestra.run.plana76.stdout:Running boto-2.38.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ISV4eJ/boto-2.38.0/egg-dist-tmp-SZ8xxg
2015-07-23T08:15:46.804 INFO:teuthology.orchestra.run.plana76.stderr:warning: no files found matching 'boto/mturk/test/*.doctest'
2015-07-23T08:15:46.804 INFO:teuthology.orchestra.run.plana76.stderr:warning: no files found matching 'boto/mturk/test/.gitignore'
2015-07-23T08:15:47.673 INFO:teuthology.orchestra.run.plana76.stderr:zip_safe flag not set; analyzing archive contents...
2015-07-23T08:15:47.676 INFO:teuthology.orchestra.run.plana76.stderr:boto.__init__: module references __file__
2015-07-23T08:15:47.678 INFO:teuthology.orchestra.run.plana76.stderr:boto.connection: module references __file__
2015-07-23T08:15:47.722 INFO:teuthology.orchestra.run.plana76.stderr:boto.vendored.six: module references __path__
2015-07-23T08:15:48.964 INFO:teuthology.orchestra.run.plana76.stdout:Adding boto 2.38.0 to easy-install.pth file
2015-07-23T08:15:48.971 INFO:teuthology.orchestra.run.plana76.stdout:Installing sdbadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.971 INFO:teuthology.orchestra.run.plana76.stdout:Installing kill_instance script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.974 INFO:teuthology.orchestra.run.plana76.stdout:Installing s3put script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.975 INFO:teuthology.orchestra.run.plana76.stdout:Installing cfadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.977 INFO:teuthology.orchestra.run.plana76.stdout:Installing elbadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.978 INFO:teuthology.orchestra.run.plana76.stdout:Installing bundle_image script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.978 INFO:teuthology.orchestra.run.plana76.stdout:Installing dynamodb_dump script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.979 INFO:teuthology.orchestra.run.plana76.stdout:Installing cq script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.983 INFO:teuthology.orchestra.run.plana76.stdout:Installing mturk script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.984 INFO:teuthology.orchestra.run.plana76.stdout:Installing list_instances script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.986 INFO:teuthology.orchestra.run.plana76.stdout:Installing asadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.987 INFO:teuthology.orchestra.run.plana76.stdout:Installing cwutil script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.989 INFO:teuthology.orchestra.run.plana76.stdout:Installing route53 script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.991 INFO:teuthology.orchestra.run.plana76.stdout:Installing launch_instance script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.992 INFO:teuthology.orchestra.run.plana76.stdout:Installing lss3 script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.993 INFO:teuthology.orchestra.run.plana76.stdout:Installing instance_events script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.994 INFO:teuthology.orchestra.run.plana76.stdout:Installing taskadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.995 INFO:teuthology.orchestra.run.plana76.stdout:Installing pyami_sendmail script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.996 INFO:teuthology.orchestra.run.plana76.stdout:Installing dynamodb_load script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.996 INFO:teuthology.orchestra.run.plana76.stdout:Installing fetch_file script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.997 INFO:teuthology.orchestra.run.plana76.stdout:Installing glacier script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:48.998 INFO:teuthology.orchestra.run.plana76.stdout:
2015-07-23T08:15:48.998 INFO:teuthology.orchestra.run.plana76.stdout:Installed /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg
2015-07-23T08:15:49.033 INFO:teuthology.orchestra.run.plana76.stdout:Searching for distribute==0.6.24
2015-07-23T08:15:49.033 INFO:teuthology.orchestra.run.plana76.stdout:Best match: distribute 0.6.24
2015-07-23T08:15:49.033 INFO:teuthology.orchestra.run.plana76.stdout:Processing distribute-0.6.24-py2.7.egg
2015-07-23T08:15:49.033 INFO:teuthology.orchestra.run.plana76.stdout:distribute 0.6.24 is already the active version in easy-install.pth
2015-07-23T08:15:49.035 INFO:teuthology.orchestra.run.plana76.stdout:Installing easy_install script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:49.035 INFO:teuthology.orchestra.run.plana76.stdout:Installing easy_install-2.7 script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-07-23T08:15:49.036 INFO:teuthology.orchestra.run.plana76.stdout:
2015-07-23T08:15:49.036 INFO:teuthology.orchestra.run.plana76.stdout:Using /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg
2015-07-23T08:15:49.036 INFO:teuthology.orchestra.run.plana76.stdout:Finished processing dependencies for radosgw-agent==1.2.3
2015-07-23T08:15:55.409 INFO:teuthology.orchestra.run.plana76.stdout:Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.7 (from -r requirements.txt (line 1))
2015-07-23T08:15:55.409 INFO:teuthology.orchestra.run.plana76.stdout:Requirement already satisfied (use --upgrade to upgrade): boto>=2.2.2,<3.0.0 in ./virtualenv/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg (from -r requirements.txt (line 2))
2015-07-23T08:15:55.409 INFO:teuthology.orchestra.run.plana76.stdout:Downloading/unpacking PyYAML (from -r requirements.txt (line 3))
2015-07-23T08:15:55.410 INFO:teuthology.orchestra.run.plana76.stdout: Running setup.py egg_info for package PyYAML
2015-07-23T08:15:55.410 INFO:teuthology.orchestra.run.plana76.stdout:
2015-07-23T08:15:55.410 INFO:teuthology.orchestra.run.plana76.stdout:Downloading/unpacking
pytest>=2.1.3 (from -r requirements-dev.txt (line 1))
Downloading pytest-2.7.2.tar.gz (532Kb): 532Kb downloaded

2015-07-23T08:15:55.410 INFO:teuthology.orchestra.run.plana76.stdout:
Running setup.py egg_info for package pytest

Downloading/unpacking

2015-07-23T08:15:55.410 INFO:teuthology.orchestra.run.plana76.stdout:
2015-07-23T08:15:55.411 INFO:teuthology.orchestra.run.plana76.stdout:Downloading/unpacking
mock>=1.0 (from -r requirements-dev.txt (line 2))
Downloading mock-1.3.0.tar.gz (70Kb): 70Kb downloaded

2015-07-23T08:15:55.411 INFO:teuthology.orchestra.run.plana76.stdout:
Running setup.py egg_info for package mock

2015-07-23T08:15:55.411 INFO:teuthology.orchestra.run.plana76.stdout:
mock requires setuptools>=17.1. Aborting installation

2015-07-23T08:15:55.411 INFO:teuthology.orchestra.run.plana76.stdout:
Complete output from command python setup.py egg_info:

2015-07-23T08:15:55.411 INFO:teuthology.orchestra.run.plana76.stdout:
mock requires setuptools>=17.1. Aborting installation

----------------------------------------

Command 2015-07-23T08:15:55.411 INFO:teuthology.orchestra.run.plana76.stdout:
2015-07-23T08:15:55.411 INFO:teuthology.orchestra.run.plana76.stdout:----------------------------------------
2015-07-23T08:15:55.411 INFO:teuthology.orchestra.run.plana76.stdout:Command
python setup.py egg_info failed with error code 1 in /radosgw-agent/virtualenv/build/mock /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/build/mock
Storing 2015-07-23T08:15:55.412 INFO:teuthology.orchestra.run.plana76.stdout:Storing complete log in /root/.pip/pip.log /home/ubuntu/.pip/pip.log
2015-07-23T08:15:55.412 INFO:teuthology.orchestra.run.plana76.stderr:Traceback (most recent call last):
2015-07-23T08:15:55.412 INFO:teuthology.orchestra.run.plana76.stderr: File "./virtualenv/bin/pip", line 9, in <module>
2015-07-23T08:15:55.412 INFO:teuthology.orchestra.run.plana76.stderr: load_entry_point('pip==1.1', 'console_scripts', 'pip')()
2015-07-23T08:15:55.412 INFO:teuthology.orchestra.run.plana76.stderr: File "/home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/__init__.py", line 116, in main
2015-07-23T08:15:55.412 INFO:teuthology.orchestra.run.plana76.stderr: return command.main(args[1:], options)
2015-07-23T08:15:55.412 INFO:teuthology.orchestra.run.plana76.stderr: File "/home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/basecommand.py", line 141, in main
2015-07-23T08:15:55.413 INFO:teuthology.orchestra.run.plana76.stderr: log_fp = open_logfile(log_fn, 'w')
2015-07-23T08:15:55.413 INFO:teuthology.orchestra.run.plana76.stderr: File "/home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/basecommand.py", line 168, in open_logfile
2015-07-23T08:15:55.413 INFO:teuthology.orchestra.run.plana76.stderr: log_fp = open(filename, mode)
2015-07-23T08:15:55.413 INFO:teuthology.orchestra.run.plana76.stderr:IOError: [Errno 13] Permission denied: '/home/ubuntu/.pip/pip.log'
2015-07-23T08:15:55.421 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/var/lib/teuthworker/src/ceph-qa-suite_hammer/tasks/radosgw_agent.py", line 190, in task
procs = run_radosgw_agent(ctx, config)
File "/var/lib/teuthworker/src/ceph-qa-suite_hammer/tasks/radosgw_agent.py", line 65, in run_radosgw_agent
'./bootstrap',
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
r.wait()
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
label=self.label)
CommandFailedError: Command failed on plana76 with status 1: 'cd /home/ubuntu/cephtest && cd radosgw-agent.client.0 && ./bootstrap'
2015-07-23T08:15:55.604 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/search?q=4b1333c712bb40828572d792e65d5fd6
CommandFailedError: Command failed on plana76 with status 1: 'cd /home/ubuntu/cephtest && cd radosgw-agent.client.0 && ./bootstrap'
2015-07-23T08:15:55.604 DEBUG:teuthology.run_tasks:Unwinding manager radosgw-agent
2015-07-23T08:15:55.604 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2015-07-23T08:15:55.604 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 30, in nested
yield vars
File "/var/lib/teuthworker/src/ceph-qa-suite_hammer/tasks/rgw.py", line 846, in task
yield
CommandFailedError: Command failed on plana76 with status 1: 'cd /home/ubuntu/cephtest && cd radosgw-agent.client.0 && ./bootstrap'

</pre>

Back