Project

General

Profile

Bug #11104

dependency problem with Ceph "Firefly" upstream and EPEL

Added by Zack Cerza about 9 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Seen on RHEL7. check_obsoletes was enabled.

Error: Package: 1:ceph-0.87.1-0.el7.x86_64 (Ceph)
           Requires: python-ceph = 1:0.87.1-0.el7
           Removing: 1:python-ceph-0.87-0.el7.x86_64 (@Ceph)
               python-ceph = 1:0.87-0.el7
           Obsoleted By: 1:python-rados-0.80.7-0.4.el7.x86_64 (epel)
               Not found
           Updated By: 1:python-ceph-0.87.1-0.el7.x86_64 (Ceph)
               python-ceph = 1:0.87.1-0.el7
           Available: 1:python-ceph-0.86-0.el7.x86_64 (Ceph)
               python-ceph = 1:0.86-0.el7
Error: Package: 1:ceph-common-0.87.1-0.el7.x86_64 (Ceph)
           Requires: python-ceph = 1:0.87.1-0.el7
           Removing: 1:python-ceph-0.87-0.el7.x86_64 (@Ceph)
               python-ceph = 1:0.87-0.el7
           Obsoleted By: 1:python-rados-0.80.7-0.4.el7.x86_64 (epel)
               Not found
           Updated By: 1:python-ceph-0.87.1-0.el7.x86_64 (Ceph)
               python-ceph = 1:0.87.1-0.el7
           Available: 1:python-ceph-0.86-0.el7.x86_64 (Ceph)
               python-ceph = 1:0.86-0.el7
Error: Package: 1:python-rados-0.80.7-0.4.el7.x86_64 (epel)
           Requires: librados2 = 1:0.80.7
           Removing: 1:librados2-0.87-0.el7.x86_64 (@Ceph)
               librados2 = 1:0.87-0.el7
           Updated By: 1:librados2-0.87.1-0.el7.x86_64 (Ceph)
               librados2 = 1:0.87.1-0.el7
           Available: 1:librados2-0.86-0.el7.x86_64 (Ceph)
               librados2 = 1:0.86-0.el7

Disabling epel helped for the moment.


Related issues

Related to devops - Bug #13874: Missing `make` on openstack-provisioned nodes Resolved 11/25/2015
Duplicated by devops - Bug #13958: 'sudo yum install ceph-radosgw -y' errors in upgrade:firefly-hammer-x-infernalis-distro-basic-openstack Resolved 12/02/2015

Associated revisions

Revision 344ec4be (diff)
Added by Ken Dreyer over 8 years ago

Revert "Revert "ceph.spec.: add epoch""

Re-add the Epoch bump to the packaging on the Firefly branch.

Ceph Hammer and newer already have this change. It's time to add it to
Firefly as well. The reasons that I removed it back in December 2014 are
no longer valid, and this will fix some interactions with the Ceph
client packages that ship in Base RHEL 7.1+.

Fixes: #11104 http://tracker.ceph.com/issues/11104
Fixes: #13794 http://tracker.ceph.com/issues/13794

This reverts commit 7faae891aefa4c21c50430fa03d9204a86d082f8.

Signed-off-by: Ken Dreyer <>

History

#1 Updated by Zack Cerza about 9 years ago

  • Assignee set to Boris Ranto

Sorry for the weak subject.

#2 Updated by Ken Dreyer about 9 years ago

  • Subject changed from more dependency issues? to dependency problem with Ceph "Giant" upstream and EPEL

The server in this case was magna128.ceph.redhat.com, an internal server that runs our Ceph cluster for storing lab test results.

I looked around in /etc/yum.repos.d, and the two repositories in question here were EPEL (from the regular epel-release RPM) and Ceph's "Giant" release:

[Ceph]
name=Ceph packages for $basearch
baseurl=http://ceph.com/rpm-giant/rhel7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
priority=1

#3 Updated by Zack Cerza about 9 years ago

The above was seeing when attempting yum update or yum update ceph-common

#4 Updated by Ken Dreyer about 9 years ago

Also, from IRC: this server was running 0.87 originally.

#5 Updated by Boris Ranto about 9 years ago

Zack Cerza wrote:

The above was seeing when attempting yum update or yum update ceph-common

Is this still an issue or can we close this? I don't think that call trace is possible with check_obsoletes turned on.

#6 Updated by Jan Harkes about 9 years ago

Hitting the same issue. the problem seems to be that epel has split python-ceph into separate python-rados and python-rbd packages. These packages claim to obsolete python-ceph which somehow propagates back to the official Ceph repo and it will not consider python-ceph an installation candidate.

My solution for now is to run 'yum install -x python-rados -x python-rbd ceph' to explicitly ignore the packages that claim a bad obsoletes dependency.

#7 Updated by Boris Ranto about 9 years ago

Jan Harkes wrote:

Hitting the same issue. the problem seems to be that epel has split python-ceph into separate python-rados and python-rbd packages. These packages claim to obsolete python-ceph which somehow propagates back to the official Ceph repo and it will not consider python-ceph an installation candidate.

My solution for now is to run 'yum install -x python-rados -x python-rbd ceph' to explicitly ignore the packages that claim a bad obsoletes dependency.

That is precisely what the yum plugin priorities with check_obsoletes set to 1 does. ceph-deploy should also be patched to do this at least before the patches for package split get propagated to the upstream repos.

#8 Updated by Ken Dreyer almost 9 years ago

User visbits in #ceph complained about this happening today as well. check_obsoletes was set.

http://pastebin.com/c7xaQWp5

#9 Updated by Ken Dreyer almost 9 years ago

Users who have been experiencing this problem:

- If you must run Firefly (0.80.x) or Giant (0.87.x), please try enabling the "epel-testing" repository on your system prior to installing Ceph. There is an update in epel-testing that should help with this. https://admin.fedoraproject.org/updates/FEDORA-EPEL-2015-1607/ceph-0.80.7-0.5.el7

- If you can run Hammer (0.94), please try testing that out. The Hammer release's packages have been split up to match the split that happened in EPEL.

#11 Updated by Ken Dreyer over 8 years ago

I have a theory on how to fix the packaging situation with EPEL + Firefly (http://www.spinics.net/lists/ceph-devel/msg24497.html), just haven't had time to test it

#12 Updated by Ken Dreyer over 8 years ago

  • Subject changed from dependency problem with Ceph "Giant" upstream and EPEL to dependency problem with Ceph "Firefly" upstream and EPEL

#13 Updated by Vasu Kulkarni over 8 years ago

Ken,

I somehow think this is a test issue, when testing with CentOS/Fedora in mixed cluster mode, I think upgrade steps should be doing some orchestration to ensure old repo's disabled and new repo's enabled or a new function call can be provided to achieve this. right now I think both firefly and hammer repo's are enabled causing this.

#14 Updated by Vasu Kulkarni over 8 years ago

yuri,

this seems different to me,

2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:Error: Package: ceph-common-0.80.10-140.g028da25.el7.x86_64 (Ceph)
2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:           Requires: python-ceph = 0.80.10-140.g028da25.el7
2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:           Available: python-ceph-0.80.10-140.g028da25.el7.x86_64 (Ceph)
2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:               python-ceph = 0.80.10-140.g028da25.el7

#15 Updated by Boris Ranto over 8 years ago

@Vasu: The issue here still seems to be that the 'check_obsoletes = 1' is not set, see few lines above those messages:

2015-10-05T10:49:25.195 INFO:teuthology.orchestra.run.vpm171.stdout:Package python-ceph is obsoleted by python-rados, but obsoleting package does not provide for requirements.

That should not happen if the 'check_obsoletes = 1' hack is in place.

#16 Updated by Zack Cerza over 8 years ago

Boris Ranto wrote:

@Vasu: The issue here still seems to be that the 'check_obsoletes = 1' is not set, see few lines above those messages:

2015-10-05T10:49:25.195 INFO:teuthology.orchestra.run.vpm171.stdout:Package python-ceph is obsoleted by python-rados, but obsoleting package does not provide for requirements.

That should not happen if the 'check_obsoletes = 1' hack is in place.

Have you tested that, though? We have code in place to set that option and it looks like it is executing. From http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-05_10:28:07-upgrade:firefly-hammer-x:stress-split-infernalis-distro-basic-vps/1089764/teuthology.log :

2015-10-05T10:46:51.086 INFO:teuthology.orchestra.run.vpm034:Running: "echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf" 

#17 Updated by Ken Dreyer over 8 years ago

We should stop appending with "tee -a" ... it ends up writing the same line to the file many times, and last I remember, teuthology-nuke doesn't clean that up.

Boris, maybe try with five check_obsoletes = 1 identical lines in priorities.conf?

#18 Updated by Yuri Weinstein over 8 years ago

Some checks we did with branto and he was able to reproduce it locally.

ubuntu@teuthology:~$ ssh vpm139
Warning: Permanently added 'vpm139,10.214.130.139' (ECDSA) to the list of known hosts.
Last login: Tue Oct  6 11:07:38 2015 from teuthology.front.sepia.ceph.com
[ubuntu@vpm139 ~]$ cat /etc/yum/pluginconf.d/priorities.conf
[main]
enabled = 1
check_obsoletes = 1
...........
[ubuntu@vpm139 ~]$ cat /etc/yum.repos.d/ceph.repo
[Ceph]
name=Ceph packages for $basearch
baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/$basearch
enabled=1
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/noarch
enabled=1
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

[ceph-source]
name=Ceph source packages
baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/SRPMS
enabled=1
priority=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

#19 Updated by Boris Ranto over 8 years ago

OK, the actual issue here is a bug in yum-plugin-priorities. The way it works now is that it won't exclude older versions of packages that are present in some of the repositories, only the newest one. :-/

This means that the python-rados/python-rbd packages that are in base rhel get excluded from the installation procedure but python-rados/python-rbd packages that are in epel do not get excluded from the installation procedure by yum-plugin-priorities plugin.

I'll try to come up with some solution tomorrow but I'm not sure there is one that everyone will be happy with. :-/

The best splution that comes to my mind right now is to remove the client packages that are in base rhel from epel. That way, there would always be only one package to exclude and yum-plugin-priorities would work properly. That would break epel ceph for rhel 7.0 users, though.

[EDIT] yum-plugin-priorities has this issue only with obsoleted packages so as a compromise, it would suffice to remove python-rados and python-rbd epel packages to fix this issue. I guess I'll do that, then.

#20 Updated by Zack Cerza over 8 years ago

Ken Dreyer wrote:

We should stop appending with "tee -a" ... it ends up writing the same line to the file many times, and last I remember, teuthology-nuke doesn't clean that up.

We do at least attempt to reverse the change:
https://github.com/ceph/teuthology/blob/master/teuthology/task/install.py#L199-L220

It looks like the cleanup isn't being run because ``task.install.install()`` is executing everything outside the ``try:`` block here:
https://github.com/ceph/teuthology/blob/master/teuthology/task/install.py#L639-L645

#21 Updated by Ken Dreyer over 8 years ago

[EDIT] yum-plugin-priorities has this issue only with obsoleted packages so as a compromise, it would suffice to remove python-rados and python-rbd epel packages to fix this issue. I guess I'll do that, then.

I'm concerned that doing something this drastic in EPEL is going to have other unforeseen consequences. IIRC RDO relies on Ceph from EPEL, for example.

http://www.spinics.net/lists/ceph-devel/msg24497.html describes a solution of bumping the epoch in firefly upstream, and adding additional version numbers to the Obsoletes that the ceph-devel-compat and python-ceph-compat EPEL packages have. That sounds like a much less disruptive change.

#22 Updated by Boris Ranto over 8 years ago

@Ken: Do they use it with rhel 7.0 or rhel 7.1? If they use rhel 7.1, there should be no change to the packages they use.

Anyway, afaik, the policy for EPEL is to remove any packages that are in base RHEL as soon as they get there so EPEL users should always use the latest RHEL to avoid dependency problems.

#23 Updated by Boris Ranto over 8 years ago

I've created a rhel bugzilla for the yum-plugin-priorities issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1269414

In the meantime, I'll push the updated ceph packages to epel-testing repo to see what the people using epel think about this.

FWIW: Only the users that use rhel 7.0 trying to install new ceph packages from epel should be affected by this. The upgrade path should not break.

#24 Updated by Yuri Weinstein over 8 years ago

Note to Yuri From Boris IRC chat:

yuriw: a temporary workaround would be to run 'yum install -x python-rados -x python-rbd ceph' on firefly    

#25 Updated by Jan Schermer over 8 years ago

Aren't you running into yum-plugin-protectbase rather than yum-priorities? I believe that one is installed and protects everything from the base repositories by default. It has some funny behaviour when combined with priorities (some packages work, other don't).
Just an idea...

#26 Updated by Zack Cerza over 8 years ago

I don't think we're using yum-plugin-protectbase at all.

#27 Updated by Boris Ranto over 8 years ago

@Jan Schermer: I doubt that. I don't see how masking a non-bse (epel) repo would help in that case.

#28 Updated by Yuri Weinstein over 8 years ago

  • Priority changed from Normal to Urgent

#29 Updated by Boris Ranto over 8 years ago

@Yuri: The update that should fix this issue was pushed to epel-testing [1] 12 days ago, it will still take some time for it to reach stable repo and actually fix this issue.

#30 Updated by Boris Ranto over 8 years ago

  • Status changed from New to Fix Under Review

The update that should fix this issue was requested to be pushed to the stable epel repository. You can re-test if the update already hit your stable epel repository. It may still take few days for the update to show up in all the public epel mirrors, though.

#31 Updated by Ken Dreyer over 8 years ago

  • Status changed from Fix Under Review to In Progress

ceph-0.80.7-0.6.el7 was pushed to EPEL stable, but this build did not handle the conflict between ceph-devel and librados-devel/librbd-devel.

Boris created another EPEL update today, https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-e71600c1f0 , that removes librados-devel/librbd-devel.

#32 Updated by Tamilarasi muthamizhan over 8 years ago

Boris and Ken, where are we with this? you say that the change is made in epel.
am confused about the bug status "in progress"

#33 Updated by Ken Dreyer over 8 years ago

I was asked on IRC, " epel is fixed, this would work , right?"

Yes, however the fix that's currently in the epel-testing repo is basically untested, and we don't know if there will be other problems lurking once that package moves to epel-stable.

#34 Updated by Boris Ranto over 8 years ago

Note: One way to speed up the deployment of the fix is to test and vote for the update, here:

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-e71600c1f0

Otherwise, we will have to wait for ~2 weeks until it makes it into stable.

#35 Updated by Yuri Weinstein over 8 years ago

Per Tamil's request ran single rgw job on ubuntu, rhel 7.0 and centos, here are results:

http://pulpito.ceph.com/teuthology-2015-11-03_13:26:20-rgw-firefly-distro-basic-vps/ - ubunutu passed
http://pulpito.ceph.com/teuthology-2015-11-03_13:31:12-rgw-firefly-distro-basic-vps/ - rhel 7.0 failed
http://pulpito.ceph.com/teuthology-2015-11-03_13:25:58-rgw-firefly-distro-basic-vps/ - centos 7.0 (still running expected to fail as rhel 7.0)

#36 Updated by Anonymous over 8 years ago

This is fixed by the changes in:

https://github.com/ceph/teuthology/pull/704

#37 Updated by Anonymous over 8 years ago

This works for Centos 7.0 (tested on sepia vpms) and on RHEL 7.1 (tested on magna), for rgw, rbd, and rados.

Centos runs:

http://pulpito.ceph.com/ubuntu-2015-11-12_13:23:41-rgw-firefly-distro-basic-vps/
http://pulpito.ceph.com/ubuntu-2015-11-12_13:39:07-rbd-firefly-distro-basic-vps/
http://pulpito.ceph.com/ubuntu-2015-11-12_13:45:18-rados-firefly-distro-basic-vps/

RHEL runs:

http://pulpito.ceph.redhat.com/wusui-2015-11-12_19:14:17-rgw-firefly-distro-basic-magna/
http://pulpito.ceph.redhat.com/wusui-2015-11-12_19:57:17-rbd-firefly-distro-basic-magna/
http://pulpito.ceph.redhat.com/wusui-2015-11-12_20:05:39-rados-firefly-distro-basic-magna/

#38 Updated by Nathan Cutler over 8 years ago

  • Status changed from In Progress to Fix Under Review

#39 Updated by Ken Dreyer over 8 years ago

The EPEL update at https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-e71600c1f0 will solve half of the problem.

We should leave this ticket open to track the issue where upstream's librbd1 and librados2 packages get overridden by the ones in RHEL 7 Base. Upstream's firefly packages have Epoch: 0, RHEL Base has Epoch: 1. See #13794 for a related discussion.

#40 Updated by Yuri Weinstein over 8 years ago

  • Related to Bug #13874: Missing `make` on openstack-provisioned nodes added

#41 Updated by Ken Dreyer over 8 years ago

  • Status changed from Fix Under Review to 7
  • Assignee changed from Boris Ranto to Ken Dreyer

https://github.com/ceph/ceph/tree/wip-11104-firefly-add-epoch bumps the Firefly packages' Epoch to "1".

#43 Updated by Ken Dreyer over 8 years ago

Yuri Weinstein wrote:

See failed run on wip-11104-firefly-add-epoch

The error there is:

2015-12-01T12:21:14.945 INFO:teuthology.orchestra.run.vpm119.stderr:Error: Package: 1:ceph-0.80.11-1.g762d0e2.el7.x86_64 (Ceph)
2015-12-01T12:21:14.945 INFO:teuthology.orchestra.run.vpm119.stderr:           Requires: python-flask

The python-flask package is in RHEL 7's "Extras" repository. But please use CentOS instead for upstream testing - it makes all our lives easier.

#45 Updated by Boris Ranto over 8 years ago

@Ken: I wonder, why did we revert the epoch patch in firefly in the first place? Is that no longer an issue?

Anyway, I suppose that the only way to fully solve the issues related to this would be to backport the spec file patches that do actually split the packages.

#46 Updated by Yuri Weinstein over 8 years ago

  • Duplicated by Bug #13958: 'sudo yum install ceph-radosgw -y' errors in upgrade:firefly-hammer-x-infernalis-distro-basic-openstack added

#47 Updated by Ken Dreyer over 8 years ago

We were not sure how the product was going to be built a year ago, so I originally reverted the Epoch bump due to that uncertainty (see the earlier PR).

The bump is back in https://github.com/ceph/ceph/pull/6773

#48 Updated by Ken Dreyer over 8 years ago

https://github.com/ceph/ceph/pull/6773 was merged last week. I think that means this is finally resolved.

#49 Updated by Sage Weil about 8 years ago

  • Status changed from 7 to Resolved

Also available in: Atom PDF