Ceph » devops

The above was seeing when attempting yum update or yum update ceph-common

Also, from IRC: this server was running 0.87 originally.

Zack Cerza wrote:

The above was seeing when attempting yum update or yum update ceph-common

Is this still an issue or can we close this? I don't think that call trace is possible with check_obsoletes turned on.

Hitting the same issue. the problem seems to be that epel has split python-ceph into separate python-rados and python-rbd packages. These packages claim to obsolete python-ceph which somehow propagates back to the official Ceph repo and it will not consider python-ceph an installation candidate.

My solution for now is to run 'yum install -x python-rados -x python-rbd ceph' to explicitly ignore the packages that claim a bad obsoletes dependency.

Jan Harkes wrote:

Hitting the same issue. the problem seems to be that epel has split python-ceph into separate python-rados and python-rbd packages. These packages claim to obsolete python-ceph which somehow propagates back to the official Ceph repo and it will not consider python-ceph an installation candidate.

My solution for now is to run 'yum install -x python-rados -x python-rbd ceph' to explicitly ignore the packages that claim a bad obsoletes dependency.

That is precisely what the yum plugin priorities with check_obsoletes set to 1 does. ceph-deploy should also be patched to do this at least before the patches for package split get propagated to the upstream repos.

User visbits in #ceph complained about this happening today as well. check_obsoletes was set.

http://pastebin.com/c7xaQWp5

Users who have been experiencing this problem:

- If you must run Firefly (0.80.x) or Giant (0.87.x), please try enabling the "epel-testing" repository on your system prior to installing Ceph. There is an update in epel-testing that should help with this. https://admin.fedoraproject.org/updates/FEDORA-EPEL-2015-1607/ceph-0.80.7-0.5.el7

- If you can run Hammer (0.94), please try testing that out. The Hammer release's packages have been split up to match the split that happened in EPEL.

I have a theory on how to fix the packaging situation with EPEL + Firefly (http://www.spinics.net/lists/ceph-devel/msg24497.html), just haven't had time to test it

Ken,

I somehow think this is a test issue, when testing with CentOS/Fedora in mixed cluster mode, I think upgrade steps should be doing some orchestration to ensure old repo's disabled and new repo's enabled or a new function call can be provided to achieve this. right now I think both firefly and hammer repo's are enabled causing this.

yuri,

this seems different to me,

2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:Error: Package: ceph-common-0.80.10-140.g028da25.el7.x86_64 (Ceph)
2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:           Requires: python-ceph = 0.80.10-140.g028da25.el7
2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:           Available: python-ceph-0.80.10-140.g028da25.el7.x86_64 (Ceph)
2015-10-05T10:49:25.881 INFO:teuthology.orchestra.run.vpm171.stderr:               python-ceph = 0.80.10-140.g028da25.el7

@Vasu Kulkarni: The issue here still seems to be that the 'check_obsoletes = 1' is not set, see few lines above those messages:

2015-10-05T10:49:25.195 INFO:teuthology.orchestra.run.vpm171.stdout:Package python-ceph is obsoleted by python-rados, but obsoleting package does not provide for requirements.

That should not happen if the 'check_obsoletes = 1' hack is in place.

Boris Ranto wrote:

@Vasu Kulkarni: The issue here still seems to be that the 'check_obsoletes = 1' is not set, see few lines above those messages:

2015-10-05T10:49:25.195 INFO:teuthology.orchestra.run.vpm171.stdout:Package python-ceph is obsoleted by python-rados, but obsoleting package does not provide for requirements.

That should not happen if the 'check_obsoletes = 1' hack is in place.

Have you tested that, though? We have code in place to set that option and it looks like it is executing. From http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-05_10:28:07-upgrade:firefly-hammer-x:stress-split-infernalis-distro-basic-vps/1089764/teuthology.log :

2015-10-05T10:46:51.086 INFO:teuthology.orchestra.run.vpm034:Running: "echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf" 

We should stop appending with "tee -a" ... it ends up writing the same line to the file many times, and last I remember, teuthology-nuke doesn't clean that up.

Boris, maybe try with five check_obsoletes = 1 identical lines in priorities.conf?

Some checks we did with branto and he was able to reproduce it locally.

ubuntu@teuthology:~$ ssh vpm139
Warning: Permanently added 'vpm139,10.214.130.139' (ECDSA) to the list of known hosts.
Last login: Tue Oct  6 11:07:38 2015 from teuthology.front.sepia.ceph.com
[ubuntu@vpm139 ~]$ cat /etc/yum/pluginconf.d/priorities.conf
[main]
enabled = 1
check_obsoletes = 1
...........
[ubuntu@vpm139 ~]$ cat /etc/yum.repos.d/ceph.repo
[Ceph]
name=Ceph packages for $basearch
baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/$basearch
enabled=1
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/noarch
enabled=1
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

[ceph-source]
name=Ceph source packages
baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/firefly/SRPMS
enabled=1
priority=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

OK, the actual issue here is a bug in yum-plugin-priorities. The way it works now is that it won't exclude older versions of packages that are present in some of the repositories, only the newest one. :-/

This means that the python-rados/python-rbd packages that are in base rhel get excluded from the installation procedure but python-rados/python-rbd packages that are in epel do not get excluded from the installation procedure by yum-plugin-priorities plugin.

I'll try to come up with some solution tomorrow but I'm not sure there is one that everyone will be happy with. :-/

The best splution that comes to my mind right now is to remove the client packages that are in base rhel from epel. That way, there would always be only one package to exclude and yum-plugin-priorities would work properly. That would break epel ceph for rhel 7.0 users, though.

[EDIT] yum-plugin-priorities has this issue only with obsoleted packages so as a compromise, it would suffice to remove python-rados and python-rbd epel packages to fix this issue. I guess I'll do that, then.

Ken Dreyer wrote:

We should stop appending with "tee -a" ... it ends up writing the same line to the file many times, and last I remember, teuthology-nuke doesn't clean that up.

We do at least attempt to reverse the change:
https://github.com/ceph/teuthology/blob/master/teuthology/task/install.py#L199-L220

It looks like the cleanup isn't being run because ``task.install.install()`` is executing everything outside the ``try:`` block here:
https://github.com/ceph/teuthology/blob/master/teuthology/task/install.py#L639-L645

[EDIT] yum-plugin-priorities has this issue only with obsoleted packages so as a compromise, it would suffice to remove python-rados and python-rbd epel packages to fix this issue. I guess I'll do that, then.

I'm concerned that doing something this drastic in EPEL is going to have other unforeseen consequences. IIRC RDO relies on Ceph from EPEL, for example.

http://www.spinics.net/lists/ceph-devel/msg24497.html describes a solution of bumping the epoch in firefly upstream, and adding additional version numbers to the Obsoletes that the ceph-devel-compat and python-ceph-compat EPEL packages have. That sounds like a much less disruptive change.

@Ken: Do they use it with rhel 7.0 or rhel 7.1? If they use rhel 7.1, there should be no change to the packages they use.

Anyway, afaik, the policy for EPEL is to remove any packages that are in base RHEL as soon as they get there so EPEL users should always use the latest RHEL to avoid dependency problems.

I've created a rhel bugzilla for the yum-plugin-priorities issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1269414

In the meantime, I'll push the updated ceph packages to epel-testing repo to see what the people using epel think about this.

FWIW: Only the users that use rhel 7.0 trying to install new ceph packages from epel should be affected by this. The upgrade path should not break.

Note to Yuri From Boris IRC chat:

yuriw: a temporary workaround would be to run 'yum install -x python-rados -x python-rbd ceph' on firefly    

Aren't you running into yum-plugin-protectbase rather than yum-priorities? I believe that one is installed and protects everything from the base repositories by default. It has some funny behaviour when combined with priorities (some packages work, other don't).
Just an idea...

I don't think we're using yum-plugin-protectbase at all.

@Jan Schermer: I doubt that. I don't see how masking a non-bse (epel) repo would help in that case.

@Yuri: The update that should fix this issue was pushed to epel-testing [1] 12 days ago, it will still take some time for it to reach stable repo and actually fix this issue.

Boris and Ken, where are we with this? you say that the change is made in epel.
am confused about the bug status "in progress"

I was asked on IRC, " epel is fixed, this would work , right?"

Yes, however the fix that's currently in the epel-testing repo is basically untested, and we don't know if there will be other problems lurking once that package moves to epel-stable.

Note: One way to speed up the deployment of the fix is to test and vote for the update, here:

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-e71600c1f0

Otherwise, we will have to wait for ~2 weeks until it makes it into stable.

Per Tamil's request ran single rgw job on ubuntu, rhel 7.0 and centos, here are results:

http://pulpito.ceph.com/teuthology-2015-11-03_13:26:20-rgw-firefly-distro-basic-vps/ - ubunutu passed
http://pulpito.ceph.com/teuthology-2015-11-03_13:31:12-rgw-firefly-distro-basic-vps/ - rhel 7.0 failed
http://pulpito.ceph.com/teuthology-2015-11-03_13:25:58-rgw-firefly-distro-basic-vps/ - centos 7.0 (still running expected to fail as rhel 7.0)

This is fixed by the changes in:

https://github.com/ceph/teuthology/pull/704

This works for Centos 7.0 (tested on sepia vpms) and on RHEL 7.1 (tested on magna), for rgw, rbd, and rados.

Centos runs:

http://pulpito.ceph.com/ubuntu-2015-11-12_13:23:41-rgw-firefly-distro-basic-vps/
http://pulpito.ceph.com/ubuntu-2015-11-12_13:39:07-rbd-firefly-distro-basic-vps/
http://pulpito.ceph.com/ubuntu-2015-11-12_13:45:18-rados-firefly-distro-basic-vps/

RHEL runs:

http://pulpito.ceph.redhat.com/wusui-2015-11-12_19:14:17-rgw-firefly-distro-basic-magna/
http://pulpito.ceph.redhat.com/wusui-2015-11-12_19:57:17-rbd-firefly-distro-basic-magna/
http://pulpito.ceph.redhat.com/wusui-2015-11-12_20:05:39-rados-firefly-distro-basic-magna/

The EPEL update at https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-e71600c1f0 will solve half of the problem.

We should leave this ticket open to track the issue where upstream's librbd1 and librados2 packages get overridden by the ones in RHEL 7 Base. Upstream's firefly packages have Epoch: 0, RHEL Base has Epoch: 1. See #13794 for a related discussion.

See failed run on wip-11104-firefly-add-epoch

http://pulpito.ceph.com/teuthology-2015-12-01_12:12:27-rgw-wip-11104-firefly-add-epoch-distro-basic-vps/

Yuri Weinstein wrote:

See failed run on wip-11104-firefly-add-epoch

The error there is:

2015-12-01T12:21:14.945 INFO:teuthology.orchestra.run.vpm119.stderr:Error: Package: 1:ceph-0.80.11-1.g762d0e2.el7.x86_64 (Ceph)
2015-12-01T12:21:14.945 INFO:teuthology.orchestra.run.vpm119.stderr:           Requires: python-flask

Better run http://pulpito.ceph.com/teuthology-2015-12-01_12:43:56-rgw-wip-11104-firefly-add-epoch-distro-basic-vps/

@Ken: I wonder, why did we revert the epoch patch in firefly in the first place? Is that no longer an issue?

Anyway, I suppose that the only way to fully solve the issues related to this would be to backport the spec file patches that do actually split the packages.

We were not sure how the product was going to be built a year ago, so I originally reverted the Epoch bump due to that uncertainty (see the earlier PR).

The bump is back in https://github.com/ceph/ceph/pull/6773

https://github.com/ceph/ceph/pull/6773 was merged last week. I think that means this is finally resolved.