Bug #48142
closed
rados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are mutually exclusive options
Added by Michael Fritch over 3 years ago.
Updated almost 3 years ago.
Description
https://pulpito.ceph.com/mgfritch-2020-11-04_22:57:20-rados:cephadm-wip-mgfritch-testing-2020-11-04-1200-distro-basic-smithi/5591020/
Upgrade test enables container cap ptrace using a v15.2.0 ceph image:
2020-11-04T23:19:13.340 INFO:teuthology.orchestra.run.smithi014:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8ca6492-1ef3-11eb-a2ae-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true
...
...
2020-11-04T23:19:54.511 INFO:tasks.cephadm:Adding mon.c on smithi014
2020-11-04T23:19:54.512 INFO:teuthology.orchestra.run.smithi014:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8ca6492-1ef3-11eb-a2ae-001a4aab830c -- ceph orch daemon add mon 'smithi014:[v2:172.21.15.14:3301,v1:172.21.15.14:6790]=c'
And fails to start due to conflicting CapAdd and privileged options:
2020-11-04T23:20:09.662 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c@mon.c.service: Service RestartSec=10s expired, scheduling restart.
2020-11-04T23:20:09.662 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c@mon.c.service: Scheduled restart job, restart counter is at 1.
2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: Stopped Ceph mon.c for f8ca6492-1ef3-11eb-a2ae-001a4aab830c.
2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: Starting Ceph mon.c for f8ca6492-1ef3-11eb-a2ae-001a4aab830c...
2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 podman[2847]: Error: no container with name or ID ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c-mon.c found: no such container
2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: Started Ceph mon.c for f8ca6492-1ef3-11eb-a2ae-001a4aab830c.
2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 bash[2893]: Error: invalid config provided: CapAdd and privileged are mutually exclusive options
2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c@mon.c.service: Main process exited, code=exited, status=125/n/a
2020-11-04T23:20:09.664 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c@mon.c.service: Failed with result 'exit-code'.
Fixed by this PR:
https://github.com/ceph/ceph/pull/37133
But that would require us to move the upgrade start point from v15.2.0, which would reduce test coverage
The mon_election tests are currently failing on RHEL >= 8.0 only, but they will likely fail on CentOS and Ubuntu when a newer version of podman is available.
Has this been fixed in octopus yet? It was seen here: a/yuriw-2020-11-13_16:25:11-rados-octopus-distro-basic-smithi/5619781
- Pull request ID set to 38066
- Assignee set to Michael Fritch
- Status changed from New to Fix Under Review
- Subject changed from rados:cephadm/upgrade/mon_election tests are failing to rados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are mutually exclusive options
- Priority changed from Normal to Urgent
- Related to Bug #48463: mon.c: Error: invalid config provided: CapAdd and privileged are mutually exclusive options added
- Related to Bug #48535: QA smoke test: cephadm is removing mgr.y added
Fascinatingly, this bug revealed #48535
upgrade test:
description: rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait
distro$/{centos_8.0} fixed-2}
/ceph/teuthology-archive/yuriw-2021-01-04_18:28:05-rados-wip-yuri2-testing-2021-01-04-0837-octopus-distro-basic-smithi/5754125/teuthology.log
/ceph/teuthology-archive/yuriw-2021-01-04_18:28:05-rados-wip-yuri2-testing-2021-01-04-0837-octopus-distro-basic-smithi/5754037/teuthology.log
/ceph/teuthology-archive/yuriw-2021-01-08_16:38:07-rados-wip-yuri4-testing-2021-01-07-1041-octopus-distro-basic-smithi/5766412/teuthology.log
[18:08:12] <SebastianW> so, I'd like to split this suite into ubuntu from 15.2.0 and rhel+centos form 15.2.5
[18:08:38] <SebastianW> I think this is a matter of moving files around in an elegant way
[18:09:54] <SebastianW> what I need is a way to run the upgrade suite from different starting points for different distros
[18:10:21] <SebastianW> as there is no way to fix 15.2.0 to support podman 2
/ceph/teuthology-archive/yuriw-2021-01-28_19:54:33-rados-wip-yuri4-testing-2021-01-28-0959-octopus-distro-basic-smithi/5835930/teuthology.log
/ceph/teuthology-archive/yuriw-2021-01-28_19:54:33-rados-wip-yuri4-testing-2021-01-28-0959-octopus-distro-basic-smithi/5836018/teuthology.log
description: rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade
4-wait distro$/{centos_8.0} fixed-2}
/ceph/teuthology-archive/yuriw-2021-02-07_16:10:58-rados-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864530/teuthology.log
/ceph/teuthology-archive/yuriw-2021-02-07_16:10:58-rados-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864366/teuthology.log
/ceph/teuthology-archive/yuriw-2021-02-07_16:10:58-rados-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864454/teuthology.log
I see the PR for fix being closed, is there some existing fix that needs backport?
- Assignee changed from Michael Fritch to Sage Weil
- Status changed from Fix Under Review to New
- Backport set to pacific,octopus
- Pull request ID changed from 38066 to 39430
- Status changed from New to Fix Under Review
- Status changed from Fix Under Review to Pending Backport
- Status changed from Pending Backport to Resolved
- Status changed from Resolved to Pending Backport
- Backport changed from pacific,octopus to octopus
needs octopus backport too, we can get back to it after 15.0.10
2021-05-14T00:00:17.701 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 14 00:00:17 smithi173 systemd[1]: Started Ceph mon.c for 13470a3a-b447-11eb-8c11-001a4aab830c.
2021-05-14T00:00:17.701 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 14 00:00:17 smithi173 bash[32957]: Error: failed to evict container: "": failed to find container "ceph-13470a3a-b447-11eb-8c11-001a4aab830c-mon.c" in state: no container with name or ID ceph-13470a3a-b447-11eb-8c11-001a4aab830c-mon.c found: no such container
2021-05-14T00:00:17.701 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 14 00:00:17 smithi173 bash[32957]: Error: failed to evict container: "": failed to find container "ceph-13470a3a-b447-11eb-8c11-001a4aab830c-mon.c" in state: no container with name or ID ceph-13470a3a-b447-11eb-8c11-001a4aab830c-mon.c found: no such container
2021-05-14T00:00:17.702 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 14 00:00:17 smithi173 bash[32957]: Error: invalid config provided: CapAdd and privileged are mutually exclusive options
/ceph/teuthology-archive/yuriw-2021-05-13_23:05:56-rados-wip-yuri4-testing-2021-05-13-1301-octopus-distro-basic-smithi/6114290/teuthology.log
- Status changed from Pending Backport to Resolved
Also available in: Atom
PDF