Bug #11104
dependency problem with Ceph "Firefly" upstream and EPEL
0%
Description
Seen on RHEL7. check_obsoletes
was enabled.
Error: Package: 1:ceph-0.87.1-0.el7.x86_64 (Ceph) Requires: python-ceph = 1:0.87.1-0.el7 Removing: 1:python-ceph-0.87-0.el7.x86_64 (@Ceph) python-ceph = 1:0.87-0.el7 Obsoleted By: 1:python-rados-0.80.7-0.4.el7.x86_64 (epel) Not found Updated By: 1:python-ceph-0.87.1-0.el7.x86_64 (Ceph) python-ceph = 1:0.87.1-0.el7 Available: 1:python-ceph-0.86-0.el7.x86_64 (Ceph) python-ceph = 1:0.86-0.el7 Error: Package: 1:ceph-common-0.87.1-0.el7.x86_64 (Ceph) Requires: python-ceph = 1:0.87.1-0.el7 Removing: 1:python-ceph-0.87-0.el7.x86_64 (@Ceph) python-ceph = 1:0.87-0.el7 Obsoleted By: 1:python-rados-0.80.7-0.4.el7.x86_64 (epel) Not found Updated By: 1:python-ceph-0.87.1-0.el7.x86_64 (Ceph) python-ceph = 1:0.87.1-0.el7 Available: 1:python-ceph-0.86-0.el7.x86_64 (Ceph) python-ceph = 1:0.86-0.el7 Error: Package: 1:python-rados-0.80.7-0.4.el7.x86_64 (epel) Requires: librados2 = 1:0.80.7 Removing: 1:librados2-0.87-0.el7.x86_64 (@Ceph) librados2 = 1:0.87-0.el7 Updated By: 1:librados2-0.87.1-0.el7.x86_64 (Ceph) librados2 = 1:0.87.1-0.el7 Available: 1:librados2-0.86-0.el7.x86_64 (Ceph) librados2 = 1:0.86-0.el7
Disabling epel helped for the moment.
Related issues
Associated revisions
Revert "Revert "ceph.spec.: add epoch""
Re-add the Epoch bump to the packaging on the Firefly branch.
Ceph Hammer and newer already have this change. It's time to add it to
Firefly as well. The reasons that I removed it back in December 2014 are
no longer valid, and this will fix some interactions with the Ceph
client packages that ship in Base RHEL 7.1+.
Fixes: #11104 http://tracker.ceph.com/issues/11104
Fixes: #13794 http://tracker.ceph.com/issues/13794
This reverts commit 7faae891aefa4c21c50430fa03d9204a86d082f8.
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
History
#2 Updated by Ken Dreyer about 9 years ago
- Subject changed from more dependency issues? to dependency problem with Ceph "Giant" upstream and EPEL
The server in this case was magna128.ceph.redhat.com, an internal server that runs our Ceph cluster for storing lab test results.
I looked around in /etc/yum.repos.d, and the two repositories in question here were EPEL (from the regular epel-release RPM) and Ceph's "Giant" release:
[Ceph] name=Ceph packages for $basearch baseurl=http://ceph.com/rpm-giant/rhel7/$basearch enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc priority=1
#3 Updated by Zack Cerza about 9 years ago
The above was seeing when attempting yum update
or yum update ceph-common
#4 Updated by Ken Dreyer about 9 years ago
Also, from IRC: this server was running 0.87 originally.
#5 Updated by Boris Ranto about 9 years ago
Zack Cerza wrote:
The above was seeing when attempting
yum update
oryum update ceph-common
Is this still an issue or can we close this? I don't think that call trace is possible with check_obsoletes turned on.
#6 Updated by Jan Harkes about 9 years ago
Hitting the same issue. the problem seems to be that epel has split python-ceph into separate python-rados and python-rbd packages. These packages claim to obsolete python-ceph which somehow propagates back to the official Ceph repo and it will not consider python-ceph an installation candidate.
My solution for now is to run 'yum install -x python-rados -x python-rbd ceph' to explicitly ignore the packages that claim a bad obsoletes dependency.
#7 Updated by Boris Ranto about 9 years ago
Jan Harkes wrote:
Hitting the same issue. the problem seems to be that epel has split python-ceph into separate python-rados and python-rbd packages. These packages claim to obsolete python-ceph which somehow propagates back to the official Ceph repo and it will not consider python-ceph an installation candidate.
My solution for now is to run 'yum install -x python-rados -x python-rbd ceph' to explicitly ignore the packages that claim a bad obsoletes dependency.
That is precisely what the yum plugin priorities with check_obsoletes set to 1 does. ceph-deploy should also be patched to do this at least before the patches for package split get propagated to the upstream repos.
#8 Updated by Ken Dreyer almost 9 years ago
User visbits
in #ceph complained about this happening today as well. check_obsoletes
was set.
#9 Updated by Ken Dreyer almost 9 years ago
Users who have been experiencing this problem:
- If you must run Firefly (0.80.x) or Giant (0.87.x), please try enabling the "epel-testing" repository on your system prior to installing Ceph. There is an update in epel-testing that should help with this. https://admin.fedoraproject.org/updates/FEDORA-EPEL-2015-1607/ceph-0.80.7-0.5.el7
- If you can run Hammer (0.94), please try testing that out. The Hammer release's packages have been split up to match the split that happened in EPEL.
#10 Updated by Yuri Weinstein over 8 years ago
- Regression set to No
#11 Updated by Ken Dreyer over 8 years ago
I have a theory on how to fix the packaging situation with EPEL + Firefly (http://www.spinics.net/lists/ceph-devel/msg24497.html), just haven't had time to test it
#12 Updated by Ken Dreyer over 8 years ago
- Subject changed from dependency problem with Ceph "Giant" upstream and EPEL to dependency problem with Ceph "Firefly" upstream and EPEL
#13 Updated by Vasu Kulkarni over 8 years ago
Ken,
I somehow think this is a test issue, when testing with CentOS/Fedora in mixed cluster mode, I think upgrade steps should be doing some orchestration to ensure old repo's disabled and new repo's enabled or a new function call can be provided to achieve this. right now I think both firefly and hammer repo's are enabled causing this.
#14 Updated by Vasu Kulkarni over 8 years ago
yuri,
this seems different to me,
2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:Error: Package: ceph-common-0.80.10-140.g028da25.el7.x86_64 (Ceph) 2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr: Requires: python-ceph = 0.80.10-140.g028da25.el7 2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr: Available: python-ceph-0.80.10-140.g028da25.el7.x86_64 (Ceph) 2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr: python-ceph = 0.80.10-140.g028da25.el7
#15 Updated by Boris Ranto over 8 years ago
@Vasu: The issue here still seems to be that the 'check_obsoletes = 1' is not set, see few lines above those messages:
2015-10-05T10:49:25.195 INFO:teuthology.orchestra.run.vpm171.stdout:Package python-ceph is obsoleted by python-rados, but obsoleting package does not provide for requirements.
That should not happen if the 'check_obsoletes = 1' hack is in place.
#16 Updated by Zack Cerza over 8 years ago
Boris Ranto wrote:
@Vasu: The issue here still seems to be that the 'check_obsoletes = 1' is not set, see few lines above those messages:
2015-10-05T10:49:25.195 INFO:teuthology.orchestra.run.vpm171.stdout:Package python-ceph is obsoleted by python-rados, but obsoleting package does not provide for requirements.
That should not happen if the 'check_obsoletes = 1' hack is in place.
Have you tested that, though? We have code in place to set that option and it looks like it is executing. From http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-05_10:28:07-upgrade:firefly-hammer-x:stress-split-infernalis-distro-basic-vps/1089764/teuthology.log :
2015-10-05T10:46:51.086 INFO:teuthology.orchestra.run.vpm034:Running: "echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf"
#17 Updated by Ken Dreyer over 8 years ago
We should stop appending with "tee -a" ... it ends up writing the same line to the file many times, and last I remember, teuthology-nuke doesn't clean that up.
Boris, maybe try with five check_obsoletes = 1
identical lines in priorities.conf?
#18 Updated by Yuri Weinstein over 8 years ago
Some checks we did with branto and he was able to reproduce it locally.
ubuntu@teuthology:~$ ssh vpm139 Warning: Permanently added 'vpm139,10.214.130.139' (ECDSA) to the list of known hosts. Last login: Tue Oct 6 11:07:38 2015 from teuthology.front.sepia.ceph.com [ubuntu@vpm139 ~]$ cat /etc/yum/pluginconf.d/priorities.conf [main] enabled = 1 check_obsoletes = 1 ........... [ubuntu@vpm139 ~]$ cat /etc/yum.repos.d/ceph.repo [Ceph] name=Ceph packages for $basearch baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/$basearch enabled=1 priority=1 gpgcheck=1 type=rpm-md gpgkey=http://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc [Ceph-noarch] name=Ceph noarch packages baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/noarch enabled=1 priority=1 gpgcheck=1 type=rpm-md gpgkey=http://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc [ceph-source] name=Ceph source packages baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/SRPMS enabled=1 priority=1 gpgcheck=1 type=rpm-md gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc
#19 Updated by Boris Ranto over 8 years ago
OK, the actual issue here is a bug in yum-plugin-priorities. The way it works now is that it won't exclude older versions of packages that are present in some of the repositories, only the newest one. :-/
This means that the python-rados/python-rbd packages that are in base rhel get excluded from the installation procedure but python-rados/python-rbd packages that are in epel do not get excluded from the installation procedure by yum-plugin-priorities plugin.
I'll try to come up with some solution tomorrow but I'm not sure there is one that everyone will be happy with. :-/
The best splution that comes to my mind right now is to remove the client packages that are in base rhel from epel. That way, there would always be only one package to exclude and yum-plugin-priorities would work properly. That would break epel ceph for rhel 7.0 users, though.
[EDIT] yum-plugin-priorities has this issue only with obsoleted packages so as a compromise, it would suffice to remove python-rados and python-rbd epel packages to fix this issue. I guess I'll do that, then.
#20 Updated by Zack Cerza over 8 years ago
Ken Dreyer wrote:
We should stop appending with "tee -a" ... it ends up writing the same line to the file many times, and last I remember, teuthology-nuke doesn't clean that up.
We do at least attempt to reverse the change:
https://github.com/ceph/teuthology/blob/master/teuthology/task/install.py#L199-L220
It looks like the cleanup isn't being run because ``task.install.install()`` is executing everything outside the ``try:`` block here:
https://github.com/ceph/teuthology/blob/master/teuthology/task/install.py#L639-L645
#21 Updated by Ken Dreyer over 8 years ago
[EDIT] yum-plugin-priorities has this issue only with obsoleted packages so as a compromise, it would suffice to remove python-rados and python-rbd epel packages to fix this issue. I guess I'll do that, then.
I'm concerned that doing something this drastic in EPEL is going to have other unforeseen consequences. IIRC RDO relies on Ceph from EPEL, for example.
http://www.spinics.net/lists/ceph-devel/msg24497.html describes a solution of bumping the epoch in firefly upstream, and adding additional version numbers to the Obsoletes that the ceph-devel-compat and python-ceph-compat EPEL packages have. That sounds like a much less disruptive change.
#22 Updated by Boris Ranto over 8 years ago
@Ken: Do they use it with rhel 7.0 or rhel 7.1? If they use rhel 7.1, there should be no change to the packages they use.
Anyway, afaik, the policy for EPEL is to remove any packages that are in base RHEL as soon as they get there so EPEL users should always use the latest RHEL to avoid dependency problems.
#23 Updated by Boris Ranto over 8 years ago
I've created a rhel bugzilla for the yum-plugin-priorities issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1269414
In the meantime, I'll push the updated ceph packages to epel-testing repo to see what the people using epel think about this.
FWIW: Only the users that use rhel 7.0 trying to install new ceph packages from epel should be affected by this. The upgrade path should not break.
#24 Updated by Yuri Weinstein over 8 years ago
Note to Yuri From Boris IRC chat:
yuriw: a temporary workaround would be to run 'yum install -x python-rados -x python-rbd ceph' on firefly
#25 Updated by Jan Schermer over 8 years ago
Aren't you running into yum-plugin-protectbase rather than yum-priorities? I believe that one is installed and protects everything from the base repositories by default. It has some funny behaviour when combined with priorities (some packages work, other don't).
Just an idea...
#26 Updated by Zack Cerza over 8 years ago
I don't think we're using yum-plugin-protectbase at all.
#27 Updated by Boris Ranto over 8 years ago
@Jan Schermer: I doubt that. I don't see how masking a non-bse (epel) repo would help in that case.
#28 Updated by Yuri Weinstein over 8 years ago
- Priority changed from Normal to Urgent
#29 Updated by Boris Ranto over 8 years ago
@Yuri: The update that should fix this issue was pushed to epel-testing [1] 12 days ago, it will still take some time for it to reach stable repo and actually fix this issue.
#30 Updated by Boris Ranto over 8 years ago
- Status changed from New to Fix Under Review
The update that should fix this issue was requested to be pushed to the stable epel repository. You can re-test if the update already hit your stable epel repository. It may still take few days for the update to show up in all the public epel mirrors, though.
#31 Updated by Ken Dreyer over 8 years ago
- Status changed from Fix Under Review to In Progress
ceph-0.80.7-0.6.el7 was pushed to EPEL stable, but this build did not handle the conflict between ceph-devel and librados-devel/librbd-devel.
Boris created another EPEL update today, https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-e71600c1f0 , that removes librados-devel/librbd-devel.
#32 Updated by Tamilarasi muthamizhan over 8 years ago
Boris and Ken, where are we with this? you say that the change is made in epel.
am confused about the bug status "in progress"
#33 Updated by Ken Dreyer over 8 years ago
I was asked on IRC, " epel is fixed, this would work , right?"
Yes, however the fix that's currently in the epel-testing repo is basically untested, and we don't know if there will be other problems lurking once that package moves to epel-stable.
#34 Updated by Boris Ranto over 8 years ago
Note: One way to speed up the deployment of the fix is to test and vote for the update, here:
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-e71600c1f0
Otherwise, we will have to wait for ~2 weeks until it makes it into stable.
#35 Updated by Yuri Weinstein over 8 years ago
Per Tamil's request ran single rgw job on ubuntu, rhel 7.0 and centos, here are results:
http://pulpito.ceph.com/teuthology-2015-11-03_13:26:20-rgw-firefly-distro-basic-vps/ - ubunutu passed http://pulpito.ceph.com/teuthology-2015-11-03_13:31:12-rgw-firefly-distro-basic-vps/ - rhel 7.0 failed http://pulpito.ceph.com/teuthology-2015-11-03_13:25:58-rgw-firefly-distro-basic-vps/ - centos 7.0 (still running expected to fail as rhel 7.0)
#36 Updated by Anonymous over 8 years ago
This is fixed by the changes in:
#37 Updated by Anonymous over 8 years ago
This works for Centos 7.0 (tested on sepia vpms) and on RHEL 7.1 (tested on magna), for rgw, rbd, and rados.
Centos runs:
http://pulpito.ceph.com/ubuntu-2015-11-12_13:23:41-rgw-firefly-distro-basic-vps/ http://pulpito.ceph.com/ubuntu-2015-11-12_13:39:07-rbd-firefly-distro-basic-vps/ http://pulpito.ceph.com/ubuntu-2015-11-12_13:45:18-rados-firefly-distro-basic-vps/
RHEL runs:
http://pulpito.ceph.redhat.com/wusui-2015-11-12_19:14:17-rgw-firefly-distro-basic-magna/ http://pulpito.ceph.redhat.com/wusui-2015-11-12_19:57:17-rbd-firefly-distro-basic-magna/ http://pulpito.ceph.redhat.com/wusui-2015-11-12_20:05:39-rados-firefly-distro-basic-magna/
#38 Updated by Nathan Cutler over 8 years ago
- Status changed from In Progress to Fix Under Review
#39 Updated by Ken Dreyer over 8 years ago
The EPEL update at https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-e71600c1f0 will solve half of the problem.
We should leave this ticket open to track the issue where upstream's librbd1 and librados2 packages get overridden by the ones in RHEL 7 Base. Upstream's firefly packages have Epoch: 0, RHEL Base has Epoch: 1. See #13794 for a related discussion.
#40 Updated by Yuri Weinstein over 8 years ago
- Related to Bug #13874: Missing `make` on openstack-provisioned nodes added
#41 Updated by Ken Dreyer over 8 years ago
- Status changed from Fix Under Review to 7
- Assignee changed from Boris Ranto to Ken Dreyer
https://github.com/ceph/ceph/tree/wip-11104-firefly-add-epoch bumps the Firefly packages' Epoch to "1".
#42 Updated by Yuri Weinstein over 8 years ago
See failed run on wip-11104-firefly-add-epoch
#43 Updated by Ken Dreyer over 8 years ago
Yuri Weinstein wrote:
See failed run on wip-11104-firefly-add-epoch
The error there is:
2015-12-01T12:21:14.945 INFO:teuthology.orchestra.run.vpm119.stderr:Error: Package: 1:ceph-0.80.11-1.g762d0e2.el7.x86_64 (Ceph) 2015-12-01T12:21:14.945 INFO:teuthology.orchestra.run.vpm119.stderr: Requires: python-flask
The python-flask package is in RHEL 7's "Extras" repository. But please use CentOS instead for upstream testing - it makes all our lives easier.
#45 Updated by Boris Ranto over 8 years ago
@Ken: I wonder, why did we revert the epoch patch in firefly in the first place? Is that no longer an issue?
Anyway, I suppose that the only way to fully solve the issues related to this would be to backport the spec file patches that do actually split the packages.
#46 Updated by Yuri Weinstein over 8 years ago
- Duplicated by Bug #13958: 'sudo yum install ceph-radosgw -y' errors in upgrade:firefly-hammer-x-infernalis-distro-basic-openstack added
#47 Updated by Ken Dreyer over 8 years ago
We were not sure how the product was going to be built a year ago, so I originally reverted the Epoch bump due to that uncertainty (see the earlier PR).
The bump is back in https://github.com/ceph/ceph/pull/6773
#48 Updated by Ken Dreyer over 8 years ago
https://github.com/ceph/ceph/pull/6773 was merged last week. I think that means this is finally resolved.
#49 Updated by Sage Weil about 8 years ago
- Status changed from 7 to Resolved