Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2022-01-19T16:07:48Z
Ceph
Redmine
Orchestrator - Bug #53939 (Resolved): ceph-nfs-upgrade, pacific: Upgrade Paused due to UPGRADE_RE...
https://tracker.ceph.com/issues/53939
2022-01-19T16:07:48Z
Sebastian Wagner
<pre>
mon[102341]: : cluster [WRN] Health check failed: Upgrading daemon osd.0 on host smithi103 failed. (UPGRADE_REDEPLOY_DAEMON)
mon[66897]: cephadm 2022-01-18T16:27:48.439275+0000 mgr.smithi103.wyeocw (mgr.14712) 129 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Redeploy daemon osd.0 ...
mon[66897]: Non-zero exit code 1 from systemctl start ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0
mon[66897]: systemctl: stderr Job for ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service failed because a timeout was exceeded.
mon[66897]: systemctl: stderr See "systemctl status ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service" and "journalctl -xe" for details.
mon[66897]: Traceback (most recent call last):
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 8615, in <module>
mon[66897]: main()
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 8603, in main
mon[66897]: r = ctx.func(ctx)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 1790, in _default_image
mon[66897]: return func(ctx)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 4603, in command_deploy
mon[66897]: ports=daemon_ports)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 2715, in deploy_daemon
mon[66897]: c, osd_fsid=osd_fsid, ports=ports)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 2960, in deploy_daemon_units
mon[66897]: call_throws(ctx, ['systemctl', 'start', unit_name])
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 1469, in call_throws
mon[66897]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}')
mon[66897]: RuntimeError: Failed command: systemctl start ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0: Job for ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service failed because a timeout was exceeded.
mon[66897]: See "systemctl status ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service" and "journalctl -xe" for details.
mon[66897]: Traceback (most recent call last):
mon[66897]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1402, in _remote_connection
mon[66897]: yield (conn, connr)
mon[66897]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1295, in _run_cephadm
mon[66897]: code, '\n'.join(err)))
mon[66897]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Redeploy daemon osd.0 ...
mon[66897]: Non-zero exit code 1 from systemctl start ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0
mon[66897]: systemctl: stderr Job for ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service failed because a timeout was exceeded.
mon[66897]: systemctl: stderr See "systemctl status ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service" and "journalctl -xe" for details.
mon[66897]: Traceback (most recent call last):
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 8615, in <module>
mon[66897]: main()
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 8603, in main
mon[66897]: r = ctx.func(ctx)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 1790, in _default_image
mon[66897]: return func(ctx)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 4603, in command_deploy
mon[66897]: ports=daemon_ports)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 2715, in deploy_daemon
mon[66897]: c, osd_fsid=osd_fsid, ports=ports)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 2960, in deploy_daemon_units
mon[66897]: call_throws(ctx, ['systemctl', 'start', unit_name])
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 1469, in call_throws
mon[66897]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}')
mon[66897]: RuntimeError: Failed command: systemctl start ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0: Job for ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service failed because a timeout was exceeded.
mon[66897]: See "systemctl status ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service" and "journalctl -xe" for details.
...
cephadm 2022-01-18T16:27:48.439412+0000 mgr.smithi103.wyeocw (mgr.14712) 130 : cephadm [ERR] Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.0 on host smithi103 failed.
</pre>
<p><a class="external" href="https://pulpito.ceph.com/swagner-2022-01-18_15:34:53-rados:cephadm-wip-swagner2-testing-2022-01-18-1242-pacific-distro-default-smithi/6624255">https://pulpito.ceph.com/swagner-2022-01-18_15:34:53-rados:cephadm-wip-swagner2-testing-2022-01-18-1242-pacific-distro-default-smithi/6624255</a></p>
Orchestrator - Bug #53904 (Duplicate): cephadm: ingress jobs stuck
https://tracker.ceph.com/issues/53904
2022-01-17T16:07:38Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2022-01-17_12:42:04-orch:cephadm-wip-swagner-testing-2022-01-17-1014-distro-default-smithi/">https://pulpito.ceph.com/swagner-2022-01-17_12:42:04-orch:cephadm-wip-swagner-testing-2022-01-17-1014-distro-default-smithi/</a></p>
<pre>
2022-01-17T13:17:17.053 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:1cdf02ebbbdd98a055173cbac4d0171328a564dc shell -c /etc/ceph/ceph.conf -k />
2022-01-17T13:17:17.054 DEBUG:teuthology.orchestra.run.smithi155:> for haproxy in `ceph orch ps | grep ^haproxy.nfs.foo. | awk '"'"'{print $1}'"'"'`; do
2022-01-17T13:17:17.054 DEBUG:teuthology.orchestra.run.smithi155:> ceph orch daemon stop $haproxy
2022-01-17T13:17:17.054 DEBUG:teuthology.orchestra.run.smithi155:> while ! ceph orch ps | grep $haproxy | grep stopped; do sleep 1 ; done
2022-01-17T13:17:17.055 DEBUG:teuthology.orchestra.run.smithi155:> cat /mnt/foo/testfile
2022-01-17T13:17:17.055 DEBUG:teuthology.orchestra.run.smithi155:> echo $haproxy > /mnt/foo/testfile
2022-01-17T13:17:17.055 DEBUG:teuthology.orchestra.run.smithi155:> sync
2022-01-17T13:17:17.055 DEBUG:teuthology.orchestra.run.smithi155:> ceph orch daemon start $haproxy
2022-01-17T13:17:17.056 DEBUG:teuthology.orchestra.run.smithi155:> while ! ceph orch ps | grep $haproxy | grep running; do sleep 1 ; done
2022-01-17T13:17:17.056 DEBUG:teuthology.orchestra.run.smithi155:> done
2022-01-17T13:17:17.056 DEBUG:teuthology.orchestra.run.smithi155:> '
</pre><br />...snip...<br /><pre>
2022-01-17T13:17:20.571 INFO:teuthology.orchestra.run.smithi155.stdout:Check with each haproxy down in turn...
2022-01-17T13:17:21.281 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled to stop haproxy.nfs.foo.smithi155.xhswck on host 'smithi155'
</pre><br />...snip...
<pre>
2022-01-17T13:17:36.893 INFO:teuthology.orchestra.run.smithi155.stdout:haproxy.nfs.foo.smithi155.xhswck smithi155 *:2049,9002 stopped 0s ago 79s - - <unknown> <un>
2022-01-17T13:17:36.898 INFO:teuthology.orchestra.run.smithi155.stdout:test
2022-01-17T13:17:37.528 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled to start haproxy.nfs.foo.smithi155.xhswck on host 'smithi155'
</pre><br />...snip...<br /><pre>
2022-01-17T13:17:53.182 INFO:teuthology.orchestra.run.smithi155.stdout:haproxy.nfs.foo.smithi155.xhswck smithi155 *:2049,9002 running (5s) 0s ago 95s - - 2.3.17-d1c9119 14b>
2022-01-17T13:17:53.519 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled to stop haproxy.nfs.foo.smithi162.mahcqs on host 'smithi162'
</pre><br />...snip...<br /><pre>
2022-01-17T13:18:07.810 INFO:teuthology.orchestra.run.smithi155.stdout:haproxy.nfs.foo.smithi162.mahcqs smithi162 *:2049,9002 stopped 0s ago 102s - - <unknown> <unk>
</pre><br />...snip..<br /><pre>
h[14066]: cephadm 2022-01-17T13:17:53.516345+0000 mgr.smithi155.uoijyc (mgr.14206) 339 : cephadm [INF] Schedule stop daemon haproxy.nfs.foo.smithi162.mahcqs
</pre>
<p>But I never see a start of haproxy.nfs.foo.smithi162.mahcqs again.</p>
Orchestrator - Bug #51272 (Resolved): upgrade job: mgr.x getting removed by cephadm task: UPGRADE...
https://tracker.ceph.com/issues/51272
2021-06-18T08:47:37Z
Sebastian Wagner
<p>I think this bug is not yet merged.</p>
<ul>
<li><a class="external" href="https://github.com/ceph/ceph/pull/41478/">https://github.com/ceph/ceph/pull/41478/</a></li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/41568">https://github.com/ceph/ceph/pull/41568</a></li>
</ul>
<pre>
rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-start-upgrade 4-wait fixed-2}
</pre>
<pre>
roles:
- - mon.a
- mon.c
- mgr.y
- osd.0
- osd.1
- osd.2
- osd.3
- client.0
- node-exporter.a
- alertmanager.a
- - mon.b
- mgr.x
- osd.4
- osd.5
- osd.6
- osd.7
- client.1
- prometheus.a
- grafana.a
- node-exporter.b
</pre>
<p><strong>then</strong></p>
<pre>
: audit 2021-06-15T20:14:24.260141+0000 mgr.y (mgr.14138) 64 : audit [DBG] from='client.34106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi143=x", "target">
</pre>
<p>notice the placement only contains <strong>2;smithi143=x</strong></p>
<pre>
2021-06-15T20:14:29.203 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 15 20:14:29 smithi135 systemd[1]: Stopping Ceph mgr.y for e2a4517e-ce15-11eb-8c13-001a4aab830c...
</pre>
<p>*resulting in *</p>
<pre>
cluster 2021-06-15T20:21:09.388112+0000 mgr.x (mgr.34112) 238 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 3.7 MiB used, 707 GiB / 715 GiB avail
: debug 2021-06-15T20:21:11.241+0000 7ffa34117700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_NO_STANDBY_MGR: Upgrade: Need standby mgr daemon
: audit 2021-06-15T20:21:11.239485+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.34112 ' entity='mgr.x'
: audit 2021-06-15T20:21:11.241293+0000 mon.c (mon.1) 207 : audit [DBG] from='mgr.34112 172.21.15.143:0/2430240313' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
: cephadm 2021-06-15T20:21:11.241839+0000 mgr.x (mgr.34112) 239 : cephadm [INF] Upgrade: Target is quay.ceph.io/ceph-ci/ceph:da5e8184007182fa3cd5c8385fee4e08c5620fe2 with id 219a75e51380d5cdf3af7b1fa194d1bedd11>
: cephadm 2021-06-15T20:21:11.244338+0000 mgr.x (mgr.34112) 240 : cephadm [INF] Upgrade: Checking mgr daemons...
: cephadm 2021-06-15T20:21:11.244711+0000 mgr.x (mgr.34112) 241 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x)
: cephadm 2021-06-15T20:21:11.247775+0000 mgr.x (mgr.34112) 242 : cephadm [ERR] Upgrade: Paused due to UPGRADE_NO_STANDBY_MGR: Upgrade: Need standby mgr daemon
: audit 2021-06-15T20:21:11.253146+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.34112 ' entity='mgr.x'
: cluster 2021-06-15T20:21:11.255641+0000 mgr.x (mgr.34112) 243 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 3.7 MiB used, 707 GiB / 715 GiB avail
: audit 2021-06-15T20:21:11.259712+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.34112 ' entity='mgr.x'
</pre>
<pre>
2021-06-15T20:21:16.892 INFO:teuthology.orchestra.run.smithi135.stdout:NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID
2021-06-15T20:21:16.892 INFO:teuthology.orchestra.run.smithi135.stdout:alertmanager.a smithi135 running (117s) 107s ago 2m 0.20.0 docker.io/prom/alertmanager:v0.20.0 0881eb8f169f d7ab1fc469b4
2021-06-15T20:21:16.892 INFO:teuthology.orchestra.run.smithi135.stdout:grafana.a smithi143 running (2m) 107s ago 2m 6.6.2 docker.io/ceph/ceph-grafana:6.6.2 a0dce381714a bdf08596362b
2021-06-15T20:21:16.892 INFO:teuthology.orchestra.run.smithi135.stdout:mgr.x smithi143 running (6m) 107s ago 6m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 bf659290d1ab
2021-06-15T20:21:16.893 INFO:teuthology.orchestra.run.smithi135.stdout:mon.a smithi135 running (8m) 107s ago 9m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 a0083afbce6f
2021-06-15T20:21:16.893 INFO:teuthology.orchestra.run.smithi135.stdout:mon.b smithi143 running (7m) 107s ago 7m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 177430b8b423
2021-06-15T20:21:16.893 INFO:teuthology.orchestra.run.smithi135.stdout:mon.c smithi135 running (7m) 107s ago 7m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 881e672542be
2021-06-15T20:21:16.893 INFO:teuthology.orchestra.run.smithi135.stdout:node-exporter.a smithi135 running (2m) 107s ago 2m 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf acd96e0cc12e
2021-06-15T20:21:16.894 INFO:teuthology.orchestra.run.smithi135.stdout:node-exporter.b smithi143 running (2m) 107s ago 2m 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf a3c897228c6d
2021-06-15T20:21:16.894 INFO:teuthology.orchestra.run.smithi135.stdout:osd.0 smithi135 running (5m) 107s ago 5m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 9805ecc9628d
2021-06-15T20:21:16.894 INFO:teuthology.orchestra.run.smithi135.stdout:osd.1 smithi135 running (5m) 107s ago 5m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 29d8fc3fbb7f
2021-06-15T20:21:16.894 INFO:teuthology.orchestra.run.smithi135.stdout:osd.2 smithi135 running (5m) 107s ago 5m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 193e0a2a0487
2021-06-15T20:21:16.895 INFO:teuthology.orchestra.run.smithi135.stdout:osd.3 smithi135 running (4m) 107s ago 4m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 e2dea4bf5490
2021-06-15T20:21:16.895 INFO:teuthology.orchestra.run.smithi135.stdout:osd.4 smithi143 running (4m) 107s ago 4m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 e0e19361a64a
2021-06-15T20:21:16.895 INFO:teuthology.orchestra.run.smithi135.stdout:osd.5 smithi143 running (3m) 107s ago 3m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 71c57f8c0e3d
2021-06-15T20:21:16.895 INFO:teuthology.orchestra.run.smithi135.stdout:osd.6 smithi143 running (3m) 107s ago 3m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 4da5baa064d1
2021-06-15T20:21:16.895 INFO:teuthology.orchestra.run.smithi135.stdout:osd.7 smithi143 running (3m) 107s ago 3m 15.2.9 docker.io/ceph/ceph:v15.2.9 dfc483079636 098193d20e10
2021-06-15T20:21:16.896 INFO:teuthology.orchestra.run.smithi135.stdout:prometheus.a smithi143 running (110s) 107s ago 2m 2.18.1 docker.io/prom/prometheus:v2.18.1 de242295e225 fb7dd6cd2280
</pre>
<p><a class="external" href="http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-15_18:44:29-rados-wip-yuri8-testing-2021-06-15-0839-octopus-distro-basic-smithi/6174184/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-15_18:44:29-rados-wip-yuri8-testing-2021-06-15-0839-octopus-distro-basic-smithi/6174184/teuthology.log</a></p>
Orchestrator - Bug #50759 (Rejected): Redeploying daemon prometheus.a on host smithi159 failed: '...
https://tracker.ceph.com/issues/50759
2021-05-11T14:17:47Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2021-05-11_09:16:20-rados:cephadm-wip-swagner-testing-2021-05-06-1235-distro-basic-smithi/">https://pulpito.ceph.com/swagner-2021-05-11_09:16:20-rados:cephadm-wip-swagner-testing-2021-05-06-1235-distro-basic-smithi/</a></p>
<pre>
cluster 2021-05-11T09:58:40.820539+0000 mgr.y (mgr.44106) 332 cluster [DBG] pgmap v241132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s
cephadm 2021-05-11T09:58:41.309082+0000 mgr.y (mgr.44106) 333 cephadm [INF] UpgradeUpdating prometheus.a
cephadm 2021-05-11T09:58:41.326009+0000 mgr.y (mgr.44106) 334 cephadm [INF] Deploying daemon prometheus.a on smithi159
cluster 2021-05-11T09:58:40.820539+0000 mgr.y (mgr.44106) 332 cluster [DBG] pgmap v241132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s
cephadm 2021-05-11T09:58:41.309082+0000 mgr.y (mgr.44106) 333 cephadm [INF] UpgradeUpdating prometheus.a
cephadm 2021-05-11T09:58:41.326009+0000 mgr.y (mgr.44106) 334 cephadm [INF] Deploying daemon prometheus.a on smithi159
cluster 2021-05-11T09:58:40.820539+0000 mgr.y (mgr.44106) 332 cluster [DBG] pgmap v241132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s
cephadm 2021-05-11T09:58:41.309082+0000 mgr.y (mgr.44106) 333 cephadm [INF] UpgradeUpdating prometheus.a
cephadm 2021-05-11T09:58:41.326009+0000 mgr.y (mgr.44106) 334 cephadm [INF] Deploying daemon prometheus.a on smithi159
cluster 2021-05-11T09:58:42.821549+0000 mgr.y (mgr.44106) 335 cluster [DBG] pgmap v242132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s
cluster 2021-05-11T09:58:42.821549+0000 mgr.y (mgr.44106) 335 cluster [DBG] pgmap v242132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s
cluster 2021-05-11T09:58:42.821549+0000 mgr.y (mgr.44106) 335 cluster [DBG] pgmap v242132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s
debug 2021-05-11T09:58:45.577+0000 7f5afcd9b700 -1 log_channel(cephadm) log [ERR] cephadm exited with an error code1, stderr:Redeploy daemon prometheus.a ...
Traceback (most recent call last):
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 8187, in <module>
main()
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 8175, in main
r = ctx.func(ctx)
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 1760, in _default_image
return func(ctx)
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 4330, in command_deploy
deploy_daemon(ctx, ctx.fsid, daemon_type, daemon_id, c, uid, gid,
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 2598, in deploy_daemon
create_daemon_dirs(
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 2204, in create_daemon_dirs
f.write(content)
UnicodeEncodeError'latin-1' codec can't encode character '\u2265' in position 2023ordinal not in range(256)
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1216, in _remote_connection
yield (conn, connr)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1113, in _run_cephadm
code, '\n'.join(err)))
orchestrator._interface.OrchestratorErrorcephadm exited with an error code1, stderr:Redeploy daemon prometheus.a ...
Traceback (most recent call last):
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 8187, in <module>
main()
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 8175, in main
r = ctx.func(ctx)
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 1760, in _default_image
return func(ctx)
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 4330, in command_deploy
deploy_daemon(ctx, ctx.fsid, daemon_type, daemon_id, c, uid, gid,
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 2598, in deploy_daemon
create_daemon_dirs(
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 2204, in create_daemon_dirs
f.write(content)
UnicodeEncodeError'latin-1' codec can't encode character '\u2265' in position 2023ordinal not in range(256)
debug 2021-05-11T09:58:45.577+0000 7f5afcd9b700 -1 log_channel(cephadm) log [ERR] UpgradePaused due to UPGRADE_REDEPLOY_DAEMONRedeploying daemon prometheus.a on host smithi159 failed.
</pre>
<ul>
<li>\u2265' is the innocent-looking GREATER-THAN OR EQUAL TO sign, ≥</li>
</ul>
Orchestrator - Bug #49435 (Closed): cephadm: rgw not getting deployed due to HEALTH_WARN
https://tracker.ceph.com/issues/49435
2021-02-23T15:43:49Z
Sebastian Wagner
<p>We should provide a way for users to deploy RGW anyway and at the same time prevent radosgw-admin to block indefinitely.</p>
<p>idea: add a timeout.</p>
Orchestrator - Bug #48463 (Duplicate): mon.c: Error: invalid config provided: CapAdd and privileg...
https://tracker.ceph.com/issues/48463
2020-12-04T11:17:07Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2020-12-04_10:02:29-rados:cephadm-wip-jmolmo-testing-2020-12-02-1452-distro-basic-smithi/5680473/">https://pulpito.ceph.com/swagner-2020-12-04_10:02:29-rados:cephadm-wip-jmolmo-testing-2020-12-02-1452-distro-basic-smithi/5680473/</a></p>
<pre>
['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi135', '-v', '/var/log/ceph/c4502caa-3619-11eb-980d-001a4
aab830c:/var/log/ceph:z', '-v', '/tmp/ceph-tmpr_67xdiq:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpl0cckyum:/etc/ceph/ceph.conf:z', '-v', '/var/lib/ceph/c4502caa-3619-11eb-980d-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z', '--entrypoint', '/usr
/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'config', 'generate-minimal-conf', '-o', '/var/lib/ceph/mon/ceph-a/config']
</pre>
<p>Turns out, we're now installing podman 2 and then strting the upgrade from 15.2.0, which does not support podman 2.</p>
Orchestrator - Bug #47438 (Resolved): OSD.__init__ failes: the JSON object must be str, bytes or ...
https://tracker.ceph.com/issues/47438
2020-09-14T12:48:03Z
Sebastian Wagner
<pre>
After OSD deletion the orchestrator does not come up anymore.
From the log I can just find this:
debug 2020-09-14T12:09:37.105+0000 7fc932de3700 -1 mgr load Failed to construct class in 'cephadm'
debug 2020-09-14T12:09:37.105+0000 7fc932de3700 -1 mgr load Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/module.py", line 325, in __init__
self.rm_util.load_from_store()
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 465, in load_from_store
osd_obj = OSD.from_json(json.loads(osd), ctx=self)
File "/usr/lib64/python3.6/json/__init__.py", line 348, in loads
'not {!r}'.format(s.__class__.__name__))
TypeError: the JSON object must be str, bytes or bytearray, not 'dict'
debug 2020-09-14T12:09:37.105+0000 7fc932de3700 -1 mgr operator() Failed to run module in active mode ('cephadm')
</pre>
Orchestrator - Bug #47185 (Resolved): TypeError: _daemon_add_misc() got an unexpected keyword arg...
https://tracker.ceph.com/issues/47185
2020-08-28T10:28:01Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2020-08-28_09:46:34-rados:cephadm-wip-swagner-testing-2020-08-28-1004-distro-basic-smithi/5383116/">https://pulpito.ceph.com/swagner-2020-08-28_09:46:34-rados:cephadm-wip-swagner-testing-2020-08-28-1004-distro-basic-smithi/5383116/</a></p>
<pre>
2020-08-28T10:01:45.393 INFO:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:6258ea1dcfe72989baca3f3155cff7e60f2b9ac9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19d8afaa-e915-11ea-a074-001a4aab830c -- ceph orch daemon add mon 'smithi044:[v2:172.21.15.44:3301,v1:172.21.15.44:6790]=c'
2020-08-28T10:01:47.113 INFO:teuthology.orchestra.run.smithi044.stderr:Error EINVAL: Traceback (most recent call last):
2020-08-28T10:01:47.114 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 1191, in _handle_command
2020-08-28T10:01:47.114 INFO:teuthology.orchestra.run.smithi044.stderr: return self.handle_command(inbuf, cmd)
2020-08-28T10:01:47.114 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 141, in handle_command
2020-08-28T10:01:47.114 INFO:teuthology.orchestra.run.smithi044.stderr: return dispatch[cmd['prefix']].call(self, cmd, inbuf)
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 328, in call
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: return self.func(mgr, **kwargs)
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 103, in <lambda>
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 92, in wrapper
2020-08-28T10:01:47.116 INFO:teuthology.orchestra.run.smithi044.stderr: return func(*args, **kwargs)
2020-08-28T10:01:47.116 INFO:teuthology.orchestra.run.smithi044.stderr:TypeError: _daemon_add_misc() got an unexpected keyword argument 'smithi044:[v2:172.21.15.44:3301,v1:172.21.15.44:6790]'
2020-08-28T10:01:47.116 INFO:teuthology.orchestra.run.smithi044.stderr:
</pre>
<p>src: <a class="external" href="https://github.com/ceph/ceph-ci/blame/wip-swagner-testing-2020-08-28-1004/src/pybind/mgr/orchestrator/module.py#L769-L774">https://github.com/ceph/ceph-ci/blame/wip-swagner-testing-2020-08-28-1004/src/pybind/mgr/orchestrator/module.py#L769-L774</a></p>
<p>Possible cause: <a class="external" href="https://github.com/ceph/ceph-ci/commit/ee9dea6cbf9879208ca88786e7f3a944d479e9ed">https://github.com/ceph/ceph-ci/commit/ee9dea6cbf9879208ca88786e7f3a944d479e9ed</a></p>
Orchestrator - Bug #47170 (Resolved): cephadm "ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d-osd.3-ac...
https://tracker.ceph.com/issues/47170
2020-08-27T15:39:58Z
Sebastian Wagner
<pre>
Aug 27 11:36:50 r620-2 systemd[1]: Started Ceph osd.3 for c2f4ec26-c63c-11ea-80c1-90b11c20b87d.
Aug 27 11:36:50 r620-2 bash[9946]: Error: error creating container storage: the container name "ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d-osd.3-activate" is already in use by "c1b0b49f56035f4a1fb>
Aug 27 11:36:50 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Main process exited, code=exited, status=125/n/a
Aug 27 11:36:51 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Unit entered failed state.
Aug 27 11:36:51 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Failed with result 'exit-code'.
Aug 27 11:37:01 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Service RestartSec=10s expired, scheduling restart.
Aug 27 11:37:01 r620-2 systemd[1]: Stopped Ceph osd.3 for c2f4ec26-c63c-11ea-80c1-90b11c20b87d.
Aug 27 11:37:01 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Start request repeated too quickly.
Aug 27 11:37:01 r620-2 systemd[1]: Failed to start Ceph osd.3 for c2f4ec26-c63c-11ea-80c1-90b11c20b87d.
Aug 27 11:37:01 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Unit entered failed state.
Aug 27 11:37:01 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Failed with result 'exit-code'.
</pre>
<p>Workaround:</p>
<pre>
podman stop c1b0b49f56035f4a1fb
podman rm c1b0b49f56035f4a1fb
podman rm --storage c1b0b49f56035f4a1fb
</pre>
Orchestrator - Documentation #46701 (Resolved): remove `alias ceph='cephadm shell -- ceph'`
https://tracker.ceph.com/issues/46701
2020-07-24T08:16:42Z
Sebastian Wagner
<p>this will lead to unexpected behavior, like</p>
<pre>
$ ceph orch apply -i myfile.yaml
ERROR: no such file or directory: myfile.yaml
</pre>
RADOS - Bug #46178 (Duplicate): slow request osd_op(... (undecoded) ondisk+retry+read+ignore_over...
https://tracker.ceph.com/issues/46178
2020-06-24T12:57:47Z
Sebastian Wagner
<p>Saw this error yesterday for the first time:</p>
<p><a class="external" href="http://pulpito.ceph.com/swagner-2020-06-23_13:15:09-rados:cephadm-wip-swagner3-testing-2020-06-23-1058-distro-basic-smithi/5172444">http://pulpito.ceph.com/swagner-2020-06-23_13:15:09-rados:cephadm-wip-swagner3-testing-2020-06-23-1058-distro-basic-smithi/5172444</a></p>
<pre>
2020-06-23T14:14:24.479 INFO:tasks.cephadm:Deploying osd.1 on smithi140 with /dev/vg_nvme/lv_3...
...
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532712+0000 osd.1 (osd.1) 951804 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:02:28.014118+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532721+0000 osd.1 (osd.1) 951805 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:02:28.112645+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532732+0000 osd.1 (osd.1) 951806 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:17:28.017258+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532741+0000 osd.1 (osd.1) 951807 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:17:28.116826+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532749+0000 osd.1 (osd.1) 951808 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:32:28.021231+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532758+0000 osd.1 (osd.1) 951809 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:32:28.117176+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532770+0000 osd.1 (osd.1) 951810 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:47:28.021867+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532795+0000 osd.1 (osd.1) 951811 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:02:28.024273+0000 currently delayed
2020-06-24T01:44:38.779 INFO:ceph.osd.1.smithi140.stdout:Jun 24 01:44:38 smithi140 bash[20025]: debug 2020-06-24T01:44:38.512+0000 7f660a6f2700 -1 osd.1 49 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49)
2020-06-24T01:44:39.499 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.476728+0000 mgr.x (mgr.34109) 20737 : cluster [DBG] pgmap v20741: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.4 MiB used, 707 GiB / 715 GiB avail
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515272+0000 osd.1 (osd.1) 951812 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:17:28.028627+0000 currently delayed
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515294+0000 osd.1 (osd.1) 951813 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:32:28.033173+0000 currently delayed
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515312+0000 osd.1 (osd.1) 951814 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:47:28.037863+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515329+0000 osd.1 (osd.1) 951815 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:02:28.037117+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515344+0000 osd.1 (osd.1) 951816 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:17:28.041383+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515363+0000 osd.1 (osd.1) 951817 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:32:28.045582+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515379+0000 osd.1 (osd.1) 951818 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overla
</pre>
<p>now it happened again:</p>
<p><a class="external" href="http://pulpito.ceph.com/swagner-2020-06-24_11:29:20-rados:cephadm-wip-swagner-testing-2020-06-24-1032-distro-basic-smithi/5175427/">http://pulpito.ceph.com/swagner-2020-06-24_11:29:20-rados:cephadm-wip-swagner-testing-2020-06-24-1032-distro-basic-smithi/5175427/</a></p>
<pre>
2020-06-24T11:56:54.575 INFO:tasks.cephadm:Deploying osd.1 on smithi118 with /dev/vg_nvme/lv_3...
...
7f44d2a96700 -1 osd.1 49 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49)
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed
</pre>
<p>Unfortunately, I don't know where this comes from.</p>
Orchestrator - Bug #46157 (Resolved): cephadm upgrade test is broken: RGW: failed to bind address...
https://tracker.ceph.com/issues/46157
2020-06-23T15:28:13Z
Sebastian Wagner
<p><a class="external" href="http://pulpito.ceph.com/swagner-2020-06-23_11:55:14-rados:cephadm-wip-swagner-testing-2020-06-23-1057-distro-basic-smithi/5172315/">http://pulpito.ceph.com/swagner-2020-06-23_11:55:14-rados:cephadm-wip-swagner-testing-2020-06-23-1057-distro-basic-smithi/5172315/</a></p>
<pre>
smithi191 bash[19849]: debug 2020-06-23T12:12:23.014+0000 7fa16ade0240 -1 failed to bind address 0.0.0.0:80: Permission denied
smithi191 bash[19849]: debug 2020-06-23T12:12:23.014+0000 7fa16ade0240 -1 ERROR: failed initializing frontend
smithi191 systemd[1]: ceph-e5284d0a-b549-11ea-a06d-001a4aab830c@rgw.realm.zone.a.service: Main process exited, code=exited, status=13/n/a
smithi191 systemd[1]: ceph-e5284d0a-b549-11ea-a06d-001a4aab830c@rgw.realm.zone.a.service: Failed with result 'exit-code'.
NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID
alertmanager.a smithi191 running (88s) 79s ago 119s 0.21.0 prom/alertmanager c876f5897d7b 9bb35a708a2b
grafana.a smithi099 running (98s) 77s ago 98s 6.6.2 ceph/ceph-grafana:latest 87a51ecf0b1c 54054cba92af
mgr.x smithi099 running (5m) 77s ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 8f8d36e55746
mgr.y smithi191 running (7m) 79s ago 7m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 b44cf1b2e038
mon.a smithi191 running (8m) 79s ago 8m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 f80920a876de
mon.b smithi099 running (6m) 77s ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 97f5688a44c2
mon.c smithi191 running (6m) 79s ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 1bd09b77289e
node-exporter.a smithi191 running (2m) 79s ago 2m 1.0.1 prom/node-exporter 0e0218889c33 c2899707ead7
node-exporter.b smithi099 running (2m) 77s ago 2m 1.0.1 prom/node-exporter 0e0218889c33 f75e1a08b2c9
osd.0 smithi191 running (5m) 79s ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 b504633f19fb
osd.1 smithi191 running (5m) 79s ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 7e6502bdceaa
osd.2 smithi191 running (4m) 79s ago 4m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 8f12c72d90d9
osd.3 smithi191 running (4m) 79s ago 4m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d87273e6adf4
osd.4 smithi099 running (4m) 77s ago 4m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 e14cc4de3389
osd.5 smithi099 running (3m) 77s ago 3m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 81e03eaa21e1
osd.6 smithi099 running (3m) 77s ago 3m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d0b6c5f84249
osd.7 smithi099 running (2m) 77s ago 2m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 4afc94e89c54
prometheus.a smithi099 running (80s) 77s ago 2m 2.19.1 prom/prometheus:latest 396dc3b4e717 5395b028578a
rgw.realm.zone.a smithi191 error 79s ago 2m <unknown> docker.io/ceph/ceph:v15.2.0 <unknown> <unknown>
</pre>
<p>solution:</p>
<ul>
<li>either make the start version to 15.2.2 or 16.0.0</li>
</ul>
Orchestrator - Bug #45628 (Resolved): cephadm qa: smoke should verify daemons are actually running
https://tracker.ceph.com/issues/45628
2020-05-20T14:11:40Z
Sebastian Wagner
<p>RGW failed:</p>
<pre>
2020-05-20T13:08:09.186 INFO:teuthology.orchestra.run.smithi203.stdout:NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID
2020-05-20T13:08:09.186 INFO:teuthology.orchestra.run.smithi203.stdout:alertmanager.a smithi203 running (47s) 33s ago 75s 0.20.0 docker.io/prom/alertmanager:latest 0881eb8f169f 9bcf1765c9f6
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:grafana.a smithi060 running (58s) 31s ago 58s 6.6.2 docker.io/ceph/ceph-grafana:latest 87a51ecf0b1c 8731e3e51a0c
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:mgr.x smithi060 running (4m) 31s ago 4m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 1cd43976a17e
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:mgr.y smithi203 running (5m) 33s ago 5m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 678f88e3c420
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:mon.a smithi203 running (5m) 33s ago 6m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 68e1b9162747
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:mon.b smithi060 running (4m) 31s ago 4m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be d1383c8a0cf6
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:mon.c smithi203 running (4m) 33s ago 4m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 27a1a4d7af30
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:node-exporter.a smithi203 running (80s) 33s ago 85s 0.18.1 docker.io/prom/node-exporter:latest e5a616e4b9cf e725ba55bfd7
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:node-exporter.b smithi060 running (82s) 31s ago 86s 0.18.1 docker.io/prom/node-exporter:latest e5a616e4b9cf da71c458ed71
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:osd.0 smithi203 running (3m) 33s ago 3m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be fbd8df58b740
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:osd.1 smithi203 running (3m) 33s ago 3m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be f82a0984e8cb
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:osd.2 smithi203 running (3m) 33s ago 3m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 885fb5dfd287
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.3 smithi203 running (2m) 33s ago 2m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 4e6e5b008f2e
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.4 smithi060 running (2m) 31s ago 2m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be f1714bd9a240
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.5 smithi060 running (2m) 31s ago 2m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be e00f2801348c
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.6 smithi060 running (2m) 31s ago 2m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 73b01fddb7dd
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.7 smithi060 running (107s) 31s ago 110s 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be cebb5c6bf000
2020-05-20T13:08:09.190 INFO:teuthology.orchestra.run.smithi203.stdout:prometheus.a smithi060 running (42s) 31s ago 88s 2.18.1 docker.io/prom/prometheus:latest de242295e225 34d837c4f530
2020-05-20T13:08:09.190 INFO:teuthology.orchestra.run.smithi203.stdout:rgw.realm.zone.a smithi203 unknown 33s ago 102s <unknown> quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 <unknown> <unknown>
</pre>
<p>still the job succeeded:</p>
<p><a class="external" href="http://pulpito.ceph.com/swagner-2020-05-20_12:38:40-rados:cephadm-wip-swagner3-testing-2020-05-20-1009-distro-basic-smithi/5072816/">http://pulpito.ceph.com/swagner-2020-05-20_12:38:40-rados:cephadm-wip-swagner3-testing-2020-05-20-1009-distro-basic-smithi/5072816/</a></p>
Ceph - Bug #42528 (Resolved): python-common bulid failure: File not found: ceph-*.egg-info
https://tracker.ceph.com/issues/42528
2019-10-29T12:19:41Z
Sebastian Wagner
<pre>
PM build errors:
File not found: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python2.7/site-packages/ceph
File not found by glob: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python2.7/site-packages/ceph-*.egg-info
+ rm -fr /tmp/install-deps.1830
Build step 'Execute shell' marked build as failure
</pre>
<pre>
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/ceph
copying ceph/__init__.py -> build/lib/ceph
copying ceph/exceptions.py -> build/lib/ceph
creating build/lib/ceph/deployment
copying ceph/deployment/__init__.py -> build/lib/ceph/deployment
copying ceph/deployment/drive_group.py -> build/lib/ceph/deployment
copying ceph/deployment/ssh_orchestrator.py -> build/lib/ceph/deployment
creating build/lib/ceph/tests
copying ceph/tests/__init__.py -> build/lib/ceph/tests
copying ceph/tests/test_drive_group.py -> build/lib/ceph/tests
running install_lib
creating /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph
copying build/lib/ceph/__init__.py -> /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph
copying build/lib/ceph/exceptions.py -> /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph
creating /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/deployment
copying build/lib/ceph/deployment/__init__.py -> /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/deployment
copying build/lib/ceph/deployment/drive_group.py -> /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/deployment
copying build/lib/ceph/deployment/ssh_orchestrator.py -> /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/deployment
creating /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/tests
copying build/lib/ceph/tests/__init__.py -> /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/tests
copying build/lib/ceph/tests/test_drive_group.py -> /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/tests
byte-compiling /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/__init__.py to __init__.cpython-36.pyc
byte-compiling /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/exceptions.py to exceptions.cpython-36.pyc
byte-compiling /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/deployment/__init__.py to __init__.cpython-36.pyc
byte-compiling /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/deployment/drive_group.py to drive_group.cpython-36.pyc
byte-compiling /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/deployment/ssh_orchestrator.py to ssh_orchestrator.cpython-36.pyc
byte-compiling /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/tests/__init__.py to __init__.cpython-36.pyc
byte-compiling /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph/tests/test_drive_group.py to test_drive_group.cpython-36.pyc
running install_egg_info
running egg_info
creating ceph.egg-info
writing ceph.egg-info/PKG-INFO
writing dependency_links to ceph.egg-info/dependency_links.txt
writing requirements to ceph.egg-info/requires.txt
writing top-level names to ceph.egg-info/top_level.txt
writing manifest file 'ceph.egg-info/SOURCES.txt'
reading manifest file 'ceph.egg-info/SOURCES.txt'
writing manifest file 'ceph.egg-info/SOURCES.txt'
Copying ceph.egg-info to /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6557-g9c17ca0/rpm/el7/BUILDROOT/ceph-15.0.0-6557.g9c17ca0.el7.x86_64/usr/lib/python3.6/site-packages/ceph-1.0.0-py3.6.egg-info
running install_scripts
Traceback (most recent call last):
File "setup.py", line 45, in <module>
'Programming Language :: Python :: 3.6',
File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
parse_requirements(requires), installer=self.fetch_build_egg
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 618, in resolve
dist = best[req.key] = env.best_match(req, self, installer)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 874, in obtain
return installer(requirement)
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 339, in fetch_build_egg
return cmd.easy_install(req)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 653, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 69, in run_setup
lambda: execfile(
File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 120, in run
return func()
File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 71, in <lambda>
{'__file__':setup_script, '__name__':'__main__'}
File "setup.py", line 21, in <module>
packages=find_packages(),
File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 269, in __init__
_Distribution.__init__(self,attrs)
File "/usr/lib64/python2.7/distutils/dist.py", line 287, in __init__
self.finalize_options()
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 302, in finalize_options
ep.load()(self, ep.name, value)
File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line 9, in version_keyword
File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
</pre>
<p><a class="external" href="https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos7,DIST=centos7,MACHINE_SIZE=huge/31272//consoleFull">https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos7,DIST=centos7,MACHINE_SIZE=huge/31272//consoleFull</a><br /><a class="external" href="https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos7,DIST=centos7,MACHINE_SIZE=huge/31269//consoleFull">https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos7,DIST=centos7,MACHINE_SIZE=huge/31269//consoleFull</a></p>
mgr - Bug #39644 (Resolved): mgr/zabbix: ERROR: test_zabbix (tasks.mgr.test_module_selftest.TestM...
https://tracker.ceph.com/issues/39644
2019-05-09T08:32:02Z
Sebastian Wagner
<pre>
======================================================================
ERROR: test_zabbix (tasks.mgr.test_module_selftest.TestModuleSelftest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_module_selftest.py", line 41, in test_zabbix
self._selftest_plugin("zabbix")
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_module_selftest.py", line 34, in _selftest_plugin
"mgr", "self-test", "module", module_name)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/ceph_manager.py", line 1157, in raw_cluster_cmd
stdout=StringIO(),
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 435, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi023 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr self-test module zabbix'
</pre>
<pre>
2019-05-08 20:53:12.238 7fdcc2030700 -1 Remote method threw exception: Traceback (most recent call last):
File "/usr/share/ceph/mgr/zabbix/module.py", line 458, in self_test
data = self.get_data()
File "/usr/share/ceph/mgr/zabbix/module.py", line 209, in get_data
data['[{0},raw_bytes_used]'.format(pool['name'])] = pool['stats']['raw_bytes_used']
KeyError: ('raw_bytes_used',)
2019-05-08 20:53:12.238 7fdcc2030700 -1 mgr.server reply reply (1) Operation not permitted Test failed: Remote method threw exception: Traceback (most recent call last):
File "/usr/share/ceph/mgr/zabbix/module.py", line 458, in self_test
data = self.get_data()
File "/usr/share/ceph/mgr/zabbix/module.py", line 209, in get_data
data['[{0},raw_bytes_used]'.format(pool['name'])] = pool['stats']['raw_bytes_used']
KeyError: ('raw_bytes_used',)
</pre>
<p><a class="external" href="http://qa-proxy.ceph.com/teuthology/swagner-2019-05-08_15:36:11-rados:mgr-wip-swagner-testing-distro-basic-smithi/3941021/teuthology.log">http://qa-proxy.ceph.com/teuthology/swagner-2019-05-08_15:36:11-rados:mgr-wip-swagner-testing-distro-basic-smithi/3941021/teuthology.log</a></p>
<p>Introduced in <a class="external" href="https://github.com/ceph/ceph/pull/26152">https://github.com/ceph/ceph/pull/26152</a></p>
<p>Greg, I've assigned it to you, as Dmitriy Rabotjagov is not part of the mgr project</p>