Bug #48142
closedrados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are mutually exclusive options
0%
Description
Upgrade test enables container cap ptrace using a v15.2.0 ceph image:
2020-11-04T23:19:13.340 INFO:teuthology.orchestra.run.smithi014:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8ca6492-1ef3-11eb-a2ae-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true ... ... 2020-11-04T23:19:54.511 INFO:tasks.cephadm:Adding mon.c on smithi014 2020-11-04T23:19:54.512 INFO:teuthology.orchestra.run.smithi014:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8ca6492-1ef3-11eb-a2ae-001a4aab830c -- ceph orch daemon add mon 'smithi014:[v2:172.21.15.14:3301,v1:172.21.15.14:6790]=c'
And fails to start due to conflicting CapAdd and privileged options:
2020-11-04T23:20:09.662 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c@mon.c.service: Service RestartSec=10s expired, scheduling restart. 2020-11-04T23:20:09.662 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c@mon.c.service: Scheduled restart job, restart counter is at 1. 2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: Stopped Ceph mon.c for f8ca6492-1ef3-11eb-a2ae-001a4aab830c. 2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: Starting Ceph mon.c for f8ca6492-1ef3-11eb-a2ae-001a4aab830c... 2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 podman[2847]: Error: no container with name or ID ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c-mon.c found: no such container 2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: Started Ceph mon.c for f8ca6492-1ef3-11eb-a2ae-001a4aab830c. 2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 bash[2893]: Error: invalid config provided: CapAdd and privileged are mutually exclusive options 2020-11-04T23:20:09.663 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c@mon.c.service: Main process exited, code=exited, status=125/n/a 2020-11-04T23:20:09.664 INFO:journalctl@ceph.mon.c.smithi014.stdout:Nov 04 23:20:09 smithi014 systemd[1]: ceph-f8ca6492-1ef3-11eb-a2ae-001a4aab830c@mon.c.service: Failed with result 'exit-code'.
Updated by Michael Fritch over 3 years ago
Updated by Michael Fritch over 3 years ago
It's also appearing in the nightly Octopus test runs:
https://pulpito.ceph.com/teuthology-2020-11-01_02:30:03-rados-octopus-distro-basic-smithi/5579214/
Updated by Michael Fritch over 3 years ago
Fixed by this PR:
https://github.com/ceph/ceph/pull/37133
But that would require us to move the upgrade start point from v15.2.0, which would reduce test coverage
The mon_election tests are currently failing on RHEL >= 8.0 only, but they will likely fail on CentOS and Ubuntu when a newer version of podman is available.
Updated by Neha Ojha over 3 years ago
Has this been fixed in octopus yet? It was seen here: a/yuriw-2020-11-13_16:25:11-rados-octopus-distro-basic-smithi/5619781
Updated by Sebastian Wagner over 3 years ago
- Status changed from New to Fix Under Review
Updated by Deepika Upadhyay over 3 years ago
2020-12-05T16:05:15.022 INFO:journalctl@ceph.mon.c.smithi077.stdout:Dec 05 16:05:14 smithi077 podman[20262]: Error: no container with name or ID ceph-5dd20ab6-3713-11eb-980d-001a4aab830c-mon.c found: no such container 2020-12-05T16:05:15.022 INFO:journalctl@ceph.mon.c.smithi077.stdout:Dec 05 16:05:14 smithi077 systemd[1]: Started Ceph mon.c for 5dd20ab6-3713-11eb-980d-001a4aab830c. 2020-12-05T16:05:15.023 INFO:journalctl@ceph.mon.c.smithi077.stdout:Dec 05 16:05:14 smithi077 bash[20302]: Error: invalid config provided: CapAdd and privileged are mutually exclusive options
worth mention: this issue was observed when https://tracker.ceph.com/issues/48157 existed, not sure if it's related or not
Updated by Sebastian Wagner over 3 years ago
- Subject changed from rados:cephadm/upgrade/mon_election tests are failing to rados:cephadm/upgrade/mon_election tests are failing: CapAdd and privileged are mutually exclusive options
Updated by Sebastian Wagner over 3 years ago
- Priority changed from Normal to Urgent
Updated by Sebastian Wagner over 3 years ago
- Related to Bug #48463: mon.c: Error: invalid config provided: CapAdd and privileged are mutually exclusive options added
Updated by Sebastian Wagner over 3 years ago
- Related to Bug #48535: QA smoke test: cephadm is removing mgr.y added
Updated by Sebastian Wagner over 3 years ago
Fascinatingly, this bug revealed #48535
Updated by Deepika Upadhyay over 3 years ago
upgrade test:
description: rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait
distro$/{centos_8.0} fixed-2}
/ceph/teuthology-archive/yuriw-2021-01-04_18:28:05-rados-wip-yuri2-testing-2021-01-04-0837-octopus-distro-basic-smithi/5754125/teuthology.log
/ceph/teuthology-archive/yuriw-2021-01-04_18:28:05-rados-wip-yuri2-testing-2021-01-04-0837-octopus-distro-basic-smithi/5754037/teuthology.log
Updated by Deepika Upadhyay over 3 years ago
/ceph/teuthology-archive/yuriw-2021-01-08_16:38:07-rados-wip-yuri4-testing-2021-01-07-1041-octopus-distro-basic-smithi/5766412/teuthology.log
Updated by Sebastian Wagner over 3 years ago
[18:08:12] <SebastianW> so, I'd like to split this suite into ubuntu from 15.2.0 and rhel+centos form 15.2.5 [18:08:38] <SebastianW> I think this is a matter of moving files around in an elegant way [18:09:54] <SebastianW> what I need is a way to run the upgrade suite from different starting points for different distros [18:10:21] <SebastianW> as there is no way to fix 15.2.0 to support podman 2
Updated by Deepika Upadhyay about 3 years ago
adding failure on master since not sure if issue is resolved or not, the added PR has been closed
Updated by Deepika Upadhyay about 3 years ago
/ceph/teuthology-archive/yuriw-2021-01-28_19:54:33-rados-wip-yuri4-testing-2021-01-28-0959-octopus-distro-basic-smithi/5835930/teuthology.log
/ceph/teuthology-archive/yuriw-2021-01-28_19:54:33-rados-wip-yuri4-testing-2021-01-28-0959-octopus-distro-basic-smithi/5836018/teuthology.log
Updated by Deepika Upadhyay about 3 years ago
description: rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade
4-wait distro$/{centos_8.0} fixed-2}
/ceph/teuthology-archive/yuriw-2021-02-07_16:10:58-rados-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864530/teuthology.log
/ceph/teuthology-archive/yuriw-2021-02-07_16:10:58-rados-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864366/teuthology.log
/ceph/teuthology-archive/yuriw-2021-02-07_16:10:58-rados-wip-yuri3-testing-2021-02-03-1046-octopus-distro-basic-smithi/5864454/teuthology.log
I see the PR for fix being closed, is there some existing fix that needs backport?
Updated by Sebastian Wagner about 3 years ago
- Assignee changed from Michael Fritch to Sage Weil
Updated by Sebastian Wagner about 3 years ago
- Status changed from Fix Under Review to New
Updated by Sage Weil about 3 years ago
- Backport set to pacific,octopus
- Pull request ID changed from 38066 to 39430
Updated by Sage Weil about 3 years ago
- Status changed from New to Fix Under Review
Updated by Sage Weil about 3 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Sebastian Wagner about 3 years ago
- Status changed from Pending Backport to Resolved
Updated by Deepika Upadhyay about 3 years ago
- Status changed from Resolved to Pending Backport
- Backport changed from pacific,octopus to octopus
needs octopus backport too, we can get back to it after 15.0.10
Updated by Deepika Upadhyay about 3 years ago
octopus backport: https://github.com/ceph/ceph/pull/40364
Updated by Deepika Upadhyay almost 3 years ago
Updated by Deepika Upadhyay almost 3 years ago
2021-05-14T00:00:17.701 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 14 00:00:17 smithi173 systemd[1]: Started Ceph mon.c for 13470a3a-b447-11eb-8c11-001a4aab830c. 2021-05-14T00:00:17.701 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 14 00:00:17 smithi173 bash[32957]: Error: failed to evict container: "": failed to find container "ceph-13470a3a-b447-11eb-8c11-001a4aab830c-mon.c" in state: no container with name or ID ceph-13470a3a-b447-11eb-8c11-001a4aab830c-mon.c found: no such container 2021-05-14T00:00:17.701 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 14 00:00:17 smithi173 bash[32957]: Error: failed to evict container: "": failed to find container "ceph-13470a3a-b447-11eb-8c11-001a4aab830c-mon.c" in state: no container with name or ID ceph-13470a3a-b447-11eb-8c11-001a4aab830c-mon.c found: no such container 2021-05-14T00:00:17.702 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 14 00:00:17 smithi173 bash[32957]: Error: invalid config provided: CapAdd and privileged are mutually exclusive options
/ceph/teuthology-archive/yuriw-2021-05-13_23:05:56-rados-wip-yuri4-testing-2021-05-13-1301-octopus-distro-basic-smithi/6114290/teuthology.log
Updated by Sebastian Wagner almost 3 years ago
- Status changed from Pending Backport to Resolved