Bug #17075
closedError "sudo yum install python-rados -y" in upgrade:firefly-x-master-distro-basic-mira
0%
Description
This is point release tests hammer 0.94.8 (08277b7bc7c0e533c3fd56a0040dc0ddc74637d6)
Run: http://pulpito.front.sepia.ceph.com/yuriw-2016-08-16_21:59:58-upgrade:firefly-x-master-distro-basic-mira/
Jobs: seems to be all centos specific
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2016-08-16_21:59:58-upgrade:firefly-x-master-distro-basic-mira/369235/teuthology.log
2016-08-16T23:32:55.639 INFO:teuthology.orchestra.run.mira037.stdout:24 packages excluded due to repository priority protections
2016-08-16T23:32:55.640 INFO:teuthology.orchestra.run.mira037.stdout:Package 1:librados2-0.80.11-8.g95c4287.el7.x86_64 already installed and latest version
2016-08-16T23:32:55.640 INFO:teuthology.orchestra.run.mira037.stdout:Nothing to do
2016-08-16T23:32:55.640 INFO:teuthology.orchestra.run.mira037:Running: 'sudo yum install librbd1 -y'
2016-08-16T23:32:58.495 INFO:teuthology.orchestra.run.mira037.stdout:Loaded plugins: fastestmirror, langpacks, priorities
2016-08-16T23:32:58.495 INFO:teuthology.orchestra.run.mira037.stdout:Loading mirror speeds from cached hostfile
2016-08-16T23:32:58.495 INFO:teuthology.orchestra.run.mira037.stdout: * base: mirror.cogentco.com
2016-08-16T23:32:58.495 INFO:teuthology.orchestra.run.mira037.stdout: * epel: ftp.linux.ncsu.edu
2016-08-16T23:32:58.495 INFO:teuthology.orchestra.run.mira037.stdout: * extras: mirror.umd.edu
2016-08-16T23:32:58.495 INFO:teuthology.orchestra.run.mira037.stdout: * updates: mirror.trouble-free.net
2016-08-16T23:32:58.495 INFO:teuthology.orchestra.run.mira037.stdout:24 packages excluded due to repository priority protections
2016-08-16T23:32:58.496 INFO:teuthology.orchestra.run.mira037.stdout:Package 1:librbd1-0.80.11-8.g95c4287.el7.x86_64 already installed and latest version
2016-08-16T23:32:58.496 INFO:teuthology.orchestra.run.mira037.stdout:Nothing to do
2016-08-16T23:32:58.496 INFO:teuthology.orchestra.run.mira037:Running: 'sudo yum install python-rados -y'
2016-08-16T23:33:01.586 INFO:teuthology.orchestra.run.mira037.stdout:Loaded plugins: fastestmirror, langpacks, priorities
2016-08-16T23:33:01.586 INFO:teuthology.orchestra.run.mira037.stdout:Loading mirror speeds from cached hostfile
2016-08-16T23:33:01.586 INFO:teuthology.orchestra.run.mira037.stdout: * base: mirror.cogentco.com
2016-08-16T23:33:01.586 INFO:teuthology.orchestra.run.mira037.stdout: * epel: ftp.linux.ncsu.edu
2016-08-16T23:33:01.586 INFO:teuthology.orchestra.run.mira037.stdout: * extras: mirror.umd.edu
2016-08-16T23:33:01.586 INFO:teuthology.orchestra.run.mira037.stdout: * updates: mirror.trouble-free.net
2016-08-16T23:33:01.586 INFO:teuthology.orchestra.run.mira037.stdout:24 packages excluded due to repository priority protections
2016-08-16T23:33:01.586 INFO:teuthology.orchestra.run.mira037.stdout:No package python-rados available.
2016-08-16T23:33:01.587 INFO:teuthology.orchestra.run.mira037.stderr:Error: Nothing to do
2016-08-16T23:33:01.587 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__
for result in self:
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 261, in _update_rpm_package_list_and_install
remote.run(args=['sudo', 'yum', 'install', cpack, '-y'])
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 196, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 402, in run
r.wait()
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 166, in wait
label=self.label)
CommandFailedError: Command failed on mira041 with status 1: 'sudo yum install python-rados -y
Updated by Yuri Weinstein over 7 years ago
Updated by Yuri Weinstein over 7 years ago
Updated by Nathan Cutler over 7 years ago
Note, this is caused by https://github.com/ceph/teuthology/commit/4543fb64 which was merged in preparation for https://github.com/ceph/ceph/pull/9757 which drops the python-ceph-compat package.
One way to fix this would be to add code to teuthology to make it use different package lists depending on which major version of ceph is being installed.
I seem to remember the tests themselves can override the default package list - that might be another approach.
Updated by Nathan Cutler over 7 years ago
https://github.com/ceph/teuthology/pull/814 adds the package list override feature.
I was thinking we could just add a package list override to the test, to have it install "python-ceph" instead of "python-rados", "python-rbd", and "python-cephfs". That would overcome this particular error - teuthology will install "python-ceph" and yum will find that package in the firefly repo. Later, the upgrades should work too thanks to the "Obsoletes" and "Provides" in the spec file.
A potential problem arises at the end of the job, though, when master will have been installed. That means "python-rados", "python-rbd", and "python-cephfs" will be present, and "python-ceph" will not. Will need testing to see if teuthology chokes on this, since it will want to remove the same packages that it installed at the beginning. Even if that works, the "python-rados", "python-rbd", and "python-cephfs" packages will be left behind on the test machines, potentially polluting the next job's environment.
Updated by Loïc Dachary over 7 years ago
@Nathan Weinberg: I think this should be a thread on ceph-devel
Updated by Vasu Kulkarni over 7 years ago
Testing this on firefly https://github.com/ceph/ceph-qa-suite/pull/1130
Updated by Vasu Kulkarni over 7 years ago
Install task will be fixed with above pr, any upgrade issues should be resolved at packaging side.
Updated by Nathan Cutler over 7 years ago
@Vasu Kulkarni That will fix only runs with --suite-branch firefly, though, right?
Updated by Nathan Cutler over 7 years ago
- Status changed from New to Fix Under Review
teuthology PR: https://github.com/ceph/teuthology/pull/939
Updated by Nathan Cutler over 7 years ago
Updated by Nathan Cutler over 7 years ago
- Status changed from Fix Under Review to Resolved
Yuri, all the PRs have been merged so the problem should be fixed. If not, reopen.
Updated by Nathan Cutler over 7 years ago
The failed tests get past firefly installation, so the failures are unrelated to this issue #17075.
Here is a preliminary analysis of the failures, but I guess it deserves a new issue.
The failures happen when workunit.py tries to "git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone" four times in a row (once for each client):
2016-08-24T00:47:00.022 INFO:teuthology.orchestra.run.vpm071:Running: 'git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone ; cd -- /home/ubuntu/cephtest/clone && git checkout firefly && mv qa/workunits /home/ubuntu/cephtest/workunit.client.2' 2016-08-24T00:47:00.056 INFO:teuthology.orchestra.run.vpm071:Running: 'git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone ; cd -- /home/ubuntu/cephtest/clone && git checkout firefly && mv qa/workunits /home/ubuntu/cephtest/workunit.client.3' 2016-08-24T00:47:00.057 INFO:teuthology.orchestra.run.vpm071:Running: 'git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone ; cd -- /home/ubuntu/cephtest/clone && git checkout firefly && mv qa/workunits /home/ubuntu/cephtest/workunit.client.1' 2016-08-24T00:47:00.077 INFO:teuthology.orchestra.run.vpm071:Running: 'git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone ; cd -- /home/ubuntu/cephtest/clone && git checkout firefly && mv qa/workunits /home/ubuntu/cephtest/workunit.client.4'
As far as I can tell, this happens here: https://github.com/ceph/ceph-qa-suite/blob/hammer/tasks/workunit.py#L312-L326
The variable "testdir" is the same each time (/home/ubuntu/cephtest in this case). The first command succeeds, the remaining three fail with "fatal: destination path '/home/ubuntu/cephtest/clone' already exists and is not an empty directory."