Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2021-05-11T14:17:47Z
Ceph
Redmine
Orchestrator - Bug #50759 (Rejected): Redeploying daemon prometheus.a on host smithi159 failed: '...
https://tracker.ceph.com/issues/50759
2021-05-11T14:17:47Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2021-05-11_09:16:20-rados:cephadm-wip-swagner-testing-2021-05-06-1235-distro-basic-smithi/">https://pulpito.ceph.com/swagner-2021-05-11_09:16:20-rados:cephadm-wip-swagner-testing-2021-05-06-1235-distro-basic-smithi/</a></p>
<pre>
cluster 2021-05-11T09:58:40.820539+0000 mgr.y (mgr.44106) 332 cluster [DBG] pgmap v241132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s
cephadm 2021-05-11T09:58:41.309082+0000 mgr.y (mgr.44106) 333 cephadm [INF] UpgradeUpdating prometheus.a
cephadm 2021-05-11T09:58:41.326009+0000 mgr.y (mgr.44106) 334 cephadm [INF] Deploying daemon prometheus.a on smithi159
cluster 2021-05-11T09:58:40.820539+0000 mgr.y (mgr.44106) 332 cluster [DBG] pgmap v241132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s
cephadm 2021-05-11T09:58:41.309082+0000 mgr.y (mgr.44106) 333 cephadm [INF] UpgradeUpdating prometheus.a
cephadm 2021-05-11T09:58:41.326009+0000 mgr.y (mgr.44106) 334 cephadm [INF] Deploying daemon prometheus.a on smithi159
cluster 2021-05-11T09:58:40.820539+0000 mgr.y (mgr.44106) 332 cluster [DBG] pgmap v241132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s
cephadm 2021-05-11T09:58:41.309082+0000 mgr.y (mgr.44106) 333 cephadm [INF] UpgradeUpdating prometheus.a
cephadm 2021-05-11T09:58:41.326009+0000 mgr.y (mgr.44106) 334 cephadm [INF] Deploying daemon prometheus.a on smithi159
cluster 2021-05-11T09:58:42.821549+0000 mgr.y (mgr.44106) 335 cluster [DBG] pgmap v242132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s
cluster 2021-05-11T09:58:42.821549+0000 mgr.y (mgr.44106) 335 cluster [DBG] pgmap v242132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s
cluster 2021-05-11T09:58:42.821549+0000 mgr.y (mgr.44106) 335 cluster [DBG] pgmap v242132 pgs132 active+clean; 7.6 KiB data, 111 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s
debug 2021-05-11T09:58:45.577+0000 7f5afcd9b700 -1 log_channel(cephadm) log [ERR] cephadm exited with an error code1, stderr:Redeploy daemon prometheus.a ...
Traceback (most recent call last):
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 8187, in <module>
main()
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 8175, in main
r = ctx.func(ctx)
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 1760, in _default_image
return func(ctx)
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 4330, in command_deploy
deploy_daemon(ctx, ctx.fsid, daemon_type, daemon_id, c, uid, gid,
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 2598, in deploy_daemon
create_daemon_dirs(
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 2204, in create_daemon_dirs
f.write(content)
UnicodeEncodeError'latin-1' codec can't encode character '\u2265' in position 2023ordinal not in range(256)
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1216, in _remote_connection
yield (conn, connr)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1113, in _run_cephadm
code, '\n'.join(err)))
orchestrator._interface.OrchestratorErrorcephadm exited with an error code1, stderr:Redeploy daemon prometheus.a ...
Traceback (most recent call last):
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 8187, in <module>
main()
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 8175, in main
r = ctx.func(ctx)
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 1760, in _default_image
return func(ctx)
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 4330, in command_deploy
deploy_daemon(ctx, ctx.fsid, daemon_type, daemon_id, c, uid, gid,
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 2598, in deploy_daemon
create_daemon_dirs(
File "/var/lib/ceph/7e53a912-b23c-11eb-8c10-001a4aab830c/cephadm.1647ffb435456545022d2850dda95cc58ac4bce47ff1845094d2804873b551c2", line 2204, in create_daemon_dirs
f.write(content)
UnicodeEncodeError'latin-1' codec can't encode character '\u2265' in position 2023ordinal not in range(256)
debug 2021-05-11T09:58:45.577+0000 7f5afcd9b700 -1 log_channel(cephadm) log [ERR] UpgradePaused due to UPGRADE_REDEPLOY_DAEMONRedeploying daemon prometheus.a on host smithi159 failed.
</pre>
<ul>
<li>\u2265' is the innocent-looking GREATER-THAN OR EQUAL TO sign, ≥</li>
</ul>
Orchestrator - Bug #49435 (Closed): cephadm: rgw not getting deployed due to HEALTH_WARN
https://tracker.ceph.com/issues/49435
2021-02-23T15:43:49Z
Sebastian Wagner
<p>We should provide a way for users to deploy RGW anyway and at the same time prevent radosgw-admin to block indefinitely.</p>
<p>idea: add a timeout.</p>
Orchestrator - Bug #48463 (Duplicate): mon.c: Error: invalid config provided: CapAdd and privileg...
https://tracker.ceph.com/issues/48463
2020-12-04T11:17:07Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2020-12-04_10:02:29-rados:cephadm-wip-jmolmo-testing-2020-12-02-1452-distro-basic-smithi/5680473/">https://pulpito.ceph.com/swagner-2020-12-04_10:02:29-rados:cephadm-wip-jmolmo-testing-2020-12-02-1452-distro-basic-smithi/5680473/</a></p>
<pre>
['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi135', '-v', '/var/log/ceph/c4502caa-3619-11eb-980d-001a4
aab830c:/var/log/ceph:z', '-v', '/tmp/ceph-tmpr_67xdiq:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpl0cckyum:/etc/ceph/ceph.conf:z', '-v', '/var/lib/ceph/c4502caa-3619-11eb-980d-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z', '--entrypoint', '/usr
/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'config', 'generate-minimal-conf', '-o', '/var/lib/ceph/mon/ceph-a/config']
</pre>
<p>Turns out, we're now installing podman 2 and then strting the upgrade from 15.2.0, which does not support podman 2.</p>
Orchestrator - Bug #47438 (Resolved): OSD.__init__ failes: the JSON object must be str, bytes or ...
https://tracker.ceph.com/issues/47438
2020-09-14T12:48:03Z
Sebastian Wagner
<pre>
After OSD deletion the orchestrator does not come up anymore.
From the log I can just find this:
debug 2020-09-14T12:09:37.105+0000 7fc932de3700 -1 mgr load Failed to construct class in 'cephadm'
debug 2020-09-14T12:09:37.105+0000 7fc932de3700 -1 mgr load Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/module.py", line 325, in __init__
self.rm_util.load_from_store()
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 465, in load_from_store
osd_obj = OSD.from_json(json.loads(osd), ctx=self)
File "/usr/lib64/python3.6/json/__init__.py", line 348, in loads
'not {!r}'.format(s.__class__.__name__))
TypeError: the JSON object must be str, bytes or bytearray, not 'dict'
debug 2020-09-14T12:09:37.105+0000 7fc932de3700 -1 mgr operator() Failed to run module in active mode ('cephadm')
</pre>
Orchestrator - Bug #47185 (Resolved): TypeError: _daemon_add_misc() got an unexpected keyword arg...
https://tracker.ceph.com/issues/47185
2020-08-28T10:28:01Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2020-08-28_09:46:34-rados:cephadm-wip-swagner-testing-2020-08-28-1004-distro-basic-smithi/5383116/">https://pulpito.ceph.com/swagner-2020-08-28_09:46:34-rados:cephadm-wip-swagner-testing-2020-08-28-1004-distro-basic-smithi/5383116/</a></p>
<pre>
2020-08-28T10:01:45.393 INFO:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:6258ea1dcfe72989baca3f3155cff7e60f2b9ac9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19d8afaa-e915-11ea-a074-001a4aab830c -- ceph orch daemon add mon 'smithi044:[v2:172.21.15.44:3301,v1:172.21.15.44:6790]=c'
2020-08-28T10:01:47.113 INFO:teuthology.orchestra.run.smithi044.stderr:Error EINVAL: Traceback (most recent call last):
2020-08-28T10:01:47.114 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 1191, in _handle_command
2020-08-28T10:01:47.114 INFO:teuthology.orchestra.run.smithi044.stderr: return self.handle_command(inbuf, cmd)
2020-08-28T10:01:47.114 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 141, in handle_command
2020-08-28T10:01:47.114 INFO:teuthology.orchestra.run.smithi044.stderr: return dispatch[cmd['prefix']].call(self, cmd, inbuf)
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 328, in call
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: return self.func(mgr, **kwargs)
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 103, in <lambda>
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
2020-08-28T10:01:47.115 INFO:teuthology.orchestra.run.smithi044.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 92, in wrapper
2020-08-28T10:01:47.116 INFO:teuthology.orchestra.run.smithi044.stderr: return func(*args, **kwargs)
2020-08-28T10:01:47.116 INFO:teuthology.orchestra.run.smithi044.stderr:TypeError: _daemon_add_misc() got an unexpected keyword argument 'smithi044:[v2:172.21.15.44:3301,v1:172.21.15.44:6790]'
2020-08-28T10:01:47.116 INFO:teuthology.orchestra.run.smithi044.stderr:
</pre>
<p>src: <a class="external" href="https://github.com/ceph/ceph-ci/blame/wip-swagner-testing-2020-08-28-1004/src/pybind/mgr/orchestrator/module.py#L769-L774">https://github.com/ceph/ceph-ci/blame/wip-swagner-testing-2020-08-28-1004/src/pybind/mgr/orchestrator/module.py#L769-L774</a></p>
<p>Possible cause: <a class="external" href="https://github.com/ceph/ceph-ci/commit/ee9dea6cbf9879208ca88786e7f3a944d479e9ed">https://github.com/ceph/ceph-ci/commit/ee9dea6cbf9879208ca88786e7f3a944d479e9ed</a></p>
Orchestrator - Bug #47170 (Resolved): cephadm "ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d-osd.3-ac...
https://tracker.ceph.com/issues/47170
2020-08-27T15:39:58Z
Sebastian Wagner
<pre>
Aug 27 11:36:50 r620-2 systemd[1]: Started Ceph osd.3 for c2f4ec26-c63c-11ea-80c1-90b11c20b87d.
Aug 27 11:36:50 r620-2 bash[9946]: Error: error creating container storage: the container name "ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d-osd.3-activate" is already in use by "c1b0b49f56035f4a1fb>
Aug 27 11:36:50 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Main process exited, code=exited, status=125/n/a
Aug 27 11:36:51 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Unit entered failed state.
Aug 27 11:36:51 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Failed with result 'exit-code'.
Aug 27 11:37:01 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Service RestartSec=10s expired, scheduling restart.
Aug 27 11:37:01 r620-2 systemd[1]: Stopped Ceph osd.3 for c2f4ec26-c63c-11ea-80c1-90b11c20b87d.
Aug 27 11:37:01 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Start request repeated too quickly.
Aug 27 11:37:01 r620-2 systemd[1]: Failed to start Ceph osd.3 for c2f4ec26-c63c-11ea-80c1-90b11c20b87d.
Aug 27 11:37:01 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Unit entered failed state.
Aug 27 11:37:01 r620-2 systemd[1]: ceph-c2f4ec26-c63c-11ea-80c1-90b11c20b87d@osd.3.service: Failed with result 'exit-code'.
</pre>
<p>Workaround:</p>
<pre>
podman stop c1b0b49f56035f4a1fb
podman rm c1b0b49f56035f4a1fb
podman rm --storage c1b0b49f56035f4a1fb
</pre>
Orchestrator - Bug #46748 (Resolved): Module 'cephadm' has failed: auth get failed: failed to fin...
https://tracker.ceph.com/issues/46748
2020-07-29T09:53:48Z
Sebastian Wagner
<p>Was purged it yesterday:</p>
<pre>
ceph osd purge 32 --yes-i-really-mean-it
ceph osd tree | grep 32 => no match
ceph osd crush remove osd.32 => device 'osd.32' does not appear in the crush map
</pre>
Dashboard - Bug #46735 (New): FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
https://tracker.ceph.com/issues/46735
2020-07-28T10:55:14Z
Sebastian Wagner
<p>- <a class="external" href="https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/4429/">https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/4429/</a><br />- <a class="external" href="https://pulpito.ceph.com/jafaj-2020-08-26_09:07:46-rados-wip-jan-testing-2020-08-26-0905-distro-basic-smithi/5377674/">https://pulpito.ceph.com/jafaj-2020-08-26_09:07:46-rados-wip-jan-testing-2020-08-26-0905-distro-basic-smithi/5377674/</a><br />- <a class="external" href="https://pulpito.ceph.com/laura-2020-09-04_11:10:48-rados:dashboard-wip-laura-testing-34831-35785-distro-basic-smithi/">https://pulpito.ceph.com/laura-2020-09-04_11:10:48-rados:dashboard-wip-laura-testing-34831-35785-distro-basic-smithi/</a></p>
<pre>
2020-07-27 15:07:37,370.370 INFO:__main__:Starting test: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2020-07-27 15:07:37,371.371 INFO:__main__:Running ['./bin/ceph', 'log', 'Starting test tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all']
2020-07-27 15:07:38,636.636 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-07-27 15:07:44,230.230 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-07-27 15:07:53,170.170 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-07-27 15:07:53,761.761 INFO:tasks.mgr.dashboard.helper:Request POST to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2020-07-27 15:07:54,130.130 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2020-07-27 15:07:54,215.215 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2020-07-27 15:07:54,409.409 INFO:tasks.mgr.dashboard.helper:Request PUT to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2020-07-27 15:07:55,083.083 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2020-07-27 15:07:55,190.190 INFO:tasks.mgr.dashboard.helper:Request PUT to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2020-07-27 15:07:55,584.584 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2020-07-27 15:08:01,686.686 INFO:tasks.mgr.dashboard.helper:Request PUT to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2020-07-27 15:08:08,000.000 ERROR:tasks.mgr.dashboard.helper:Request response: {"detail": "Bad MFA credentials: RGW REST API failed request with status code 403\n(b'{\"Code\":\"AccessDenied\",\"BucketName\":\"teuth-test-bucket\",\"RequestId\":\"tx00000'\n b'0000000000000031-005f1eedd7-1158-default\",\"HostId\":\"1158-default-default\"}')", "component": "rgw"}
2020-07-27 15:08:08,001.001 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all']
2020-07-27 15:08:09,072.072 INFO:__main__:test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) ... FAIL
2020-07-27 15:08:09,072.072 INFO:__main__:Stopped test: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) in 31.701467s
2020-07-27 15:08:09,073.073 INFO:__main__:Running ['./bin/radosgw-admin', 'user', 'rm', '--tenant', 'testx', '--uid=teuth-test-user', '--purge-data']
2020-07-27 15:08:19,107.107 INFO:__main__:Running ['./bin/radosgw-admin', 'user', 'rm', '--tenant', 'testx2', '--uid=teuth-test-user2', '--purge-data']
2020-07-27 15:08:22,371.371 INFO:__main__:Running ['./bin/radosgw-admin', 'user', 'rm', '--uid', 'admin']
2020-07-27 15:08:25,605.605 INFO:__main__:Running ['./bin/radosgw-admin', 'user', 'rm', '--uid=teuth-test-user', '--purge-data']
2020-07-27 15:08:28,899.899 INFO:__main__:
2020-07-27 15:08:28,900.900 INFO:__main__:----------------------------------------------------------------------
2020-07-27 15:08:28,900.900 INFO:__main__:Traceback (most recent call last):
2020-07-27 15:08:28,901.901 INFO:__main__: File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_rgw.py", line 277, in test_all
2020-07-27 15:08:28,901.901 INFO:__main__: self.assertStatus(200)
2020-07-27 15:08:28,901.901 INFO:__main__: File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/helper.py", line 386, in assertStatus
2020-07-27 15:08:28,901.901 INFO:__main__: self.assertEqual(self._resp.status_code, status)
2020-07-27 15:08:28,901.901 INFO:__main__:AssertionError: 400 != 200
2020-07-27 15:08:28,902.902 INFO:__main__:
2020-07-27 15:08:28,902.902 INFO:__main__:----------------------------------------------------------------------
2020-07-27 15:08:28,902.902 INFO:__main__:Ran 193 tests in 5318.217s
2020-07-27 15:08:28,902.902 INFO:__main__:
2020-07-27 15:08:28,903.903 INFO:__main__:
</pre>
Orchestrator - Documentation #46701 (Resolved): remove `alias ceph='cephadm shell -- ceph'`
https://tracker.ceph.com/issues/46701
2020-07-24T08:16:42Z
Sebastian Wagner
<p>this will lead to unexpected behavior, like</p>
<pre>
$ ceph orch apply -i myfile.yaml
ERROR: no such file or directory: myfile.yaml
</pre>
RADOS - Bug #46178 (Duplicate): slow request osd_op(... (undecoded) ondisk+retry+read+ignore_over...
https://tracker.ceph.com/issues/46178
2020-06-24T12:57:47Z
Sebastian Wagner
<p>Saw this error yesterday for the first time:</p>
<p><a class="external" href="http://pulpito.ceph.com/swagner-2020-06-23_13:15:09-rados:cephadm-wip-swagner3-testing-2020-06-23-1058-distro-basic-smithi/5172444">http://pulpito.ceph.com/swagner-2020-06-23_13:15:09-rados:cephadm-wip-swagner3-testing-2020-06-23-1058-distro-basic-smithi/5172444</a></p>
<pre>
2020-06-23T14:14:24.479 INFO:tasks.cephadm:Deploying osd.1 on smithi140 with /dev/vg_nvme/lv_3...
...
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532712+0000 osd.1 (osd.1) 951804 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:02:28.014118+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532721+0000 osd.1 (osd.1) 951805 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:02:28.112645+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532732+0000 osd.1 (osd.1) 951806 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:17:28.017258+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532741+0000 osd.1 (osd.1) 951807 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:17:28.116826+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532749+0000 osd.1 (osd.1) 951808 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:32:28.021231+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532758+0000 osd.1 (osd.1) 951809 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:32:28.117176+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532770+0000 osd.1 (osd.1) 951810 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:47:28.021867+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532795+0000 osd.1 (osd.1) 951811 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:02:28.024273+0000 currently delayed
2020-06-24T01:44:38.779 INFO:ceph.osd.1.smithi140.stdout:Jun 24 01:44:38 smithi140 bash[20025]: debug 2020-06-24T01:44:38.512+0000 7f660a6f2700 -1 osd.1 49 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49)
2020-06-24T01:44:39.499 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.476728+0000 mgr.x (mgr.34109) 20737 : cluster [DBG] pgmap v20741: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.4 MiB used, 707 GiB / 715 GiB avail
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515272+0000 osd.1 (osd.1) 951812 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:17:28.028627+0000 currently delayed
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515294+0000 osd.1 (osd.1) 951813 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:32:28.033173+0000 currently delayed
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515312+0000 osd.1 (osd.1) 951814 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:47:28.037863+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515329+0000 osd.1 (osd.1) 951815 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:02:28.037117+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515344+0000 osd.1 (osd.1) 951816 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:17:28.041383+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515363+0000 osd.1 (osd.1) 951817 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:32:28.045582+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515379+0000 osd.1 (osd.1) 951818 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overla
</pre>
<p>now it happened again:</p>
<p><a class="external" href="http://pulpito.ceph.com/swagner-2020-06-24_11:29:20-rados:cephadm-wip-swagner-testing-2020-06-24-1032-distro-basic-smithi/5175427/">http://pulpito.ceph.com/swagner-2020-06-24_11:29:20-rados:cephadm-wip-swagner-testing-2020-06-24-1032-distro-basic-smithi/5175427/</a></p>
<pre>
2020-06-24T11:56:54.575 INFO:tasks.cephadm:Deploying osd.1 on smithi118 with /dev/vg_nvme/lv_3...
...
7f44d2a96700 -1 osd.1 49 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49)
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed
</pre>
<p>Unfortunately, I don't know where this comes from.</p>
Orchestrator - Bug #46157 (Resolved): cephadm upgrade test is broken: RGW: failed to bind address...
https://tracker.ceph.com/issues/46157
2020-06-23T15:28:13Z
Sebastian Wagner
<p><a class="external" href="http://pulpito.ceph.com/swagner-2020-06-23_11:55:14-rados:cephadm-wip-swagner-testing-2020-06-23-1057-distro-basic-smithi/5172315/">http://pulpito.ceph.com/swagner-2020-06-23_11:55:14-rados:cephadm-wip-swagner-testing-2020-06-23-1057-distro-basic-smithi/5172315/</a></p>
<pre>
smithi191 bash[19849]: debug 2020-06-23T12:12:23.014+0000 7fa16ade0240 -1 failed to bind address 0.0.0.0:80: Permission denied
smithi191 bash[19849]: debug 2020-06-23T12:12:23.014+0000 7fa16ade0240 -1 ERROR: failed initializing frontend
smithi191 systemd[1]: ceph-e5284d0a-b549-11ea-a06d-001a4aab830c@rgw.realm.zone.a.service: Main process exited, code=exited, status=13/n/a
smithi191 systemd[1]: ceph-e5284d0a-b549-11ea-a06d-001a4aab830c@rgw.realm.zone.a.service: Failed with result 'exit-code'.
NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID
alertmanager.a smithi191 running (88s) 79s ago 119s 0.21.0 prom/alertmanager c876f5897d7b 9bb35a708a2b
grafana.a smithi099 running (98s) 77s ago 98s 6.6.2 ceph/ceph-grafana:latest 87a51ecf0b1c 54054cba92af
mgr.x smithi099 running (5m) 77s ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 8f8d36e55746
mgr.y smithi191 running (7m) 79s ago 7m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 b44cf1b2e038
mon.a smithi191 running (8m) 79s ago 8m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 f80920a876de
mon.b smithi099 running (6m) 77s ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 97f5688a44c2
mon.c smithi191 running (6m) 79s ago 6m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 1bd09b77289e
node-exporter.a smithi191 running (2m) 79s ago 2m 1.0.1 prom/node-exporter 0e0218889c33 c2899707ead7
node-exporter.b smithi099 running (2m) 77s ago 2m 1.0.1 prom/node-exporter 0e0218889c33 f75e1a08b2c9
osd.0 smithi191 running (5m) 79s ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 b504633f19fb
osd.1 smithi191 running (5m) 79s ago 5m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 7e6502bdceaa
osd.2 smithi191 running (4m) 79s ago 4m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 8f12c72d90d9
osd.3 smithi191 running (4m) 79s ago 4m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d87273e6adf4
osd.4 smithi099 running (4m) 77s ago 4m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 e14cc4de3389
osd.5 smithi099 running (3m) 77s ago 3m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 81e03eaa21e1
osd.6 smithi099 running (3m) 77s ago 3m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 d0b6c5f84249
osd.7 smithi099 running (2m) 77s ago 2m 15.2.0 docker.io/ceph/ceph:v15.2.0 204a01f9b0b6 4afc94e89c54
prometheus.a smithi099 running (80s) 77s ago 2m 2.19.1 prom/prometheus:latest 396dc3b4e717 5395b028578a
rgw.realm.zone.a smithi191 error 79s ago 2m <unknown> docker.io/ceph/ceph:v15.2.0 <unknown> <unknown>
</pre>
<p>solution:</p>
<ul>
<li>either make the start version to 15.2.2 or 16.0.0</li>
</ul>
Orchestrator - Bug #45628 (Resolved): cephadm qa: smoke should verify daemons are actually running
https://tracker.ceph.com/issues/45628
2020-05-20T14:11:40Z
Sebastian Wagner
<p>RGW failed:</p>
<pre>
2020-05-20T13:08:09.186 INFO:teuthology.orchestra.run.smithi203.stdout:NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID
2020-05-20T13:08:09.186 INFO:teuthology.orchestra.run.smithi203.stdout:alertmanager.a smithi203 running (47s) 33s ago 75s 0.20.0 docker.io/prom/alertmanager:latest 0881eb8f169f 9bcf1765c9f6
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:grafana.a smithi060 running (58s) 31s ago 58s 6.6.2 docker.io/ceph/ceph-grafana:latest 87a51ecf0b1c 8731e3e51a0c
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:mgr.x smithi060 running (4m) 31s ago 4m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 1cd43976a17e
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:mgr.y smithi203 running (5m) 33s ago 5m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 678f88e3c420
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:mon.a smithi203 running (5m) 33s ago 6m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 68e1b9162747
2020-05-20T13:08:09.187 INFO:teuthology.orchestra.run.smithi203.stdout:mon.b smithi060 running (4m) 31s ago 4m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be d1383c8a0cf6
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:mon.c smithi203 running (4m) 33s ago 4m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 27a1a4d7af30
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:node-exporter.a smithi203 running (80s) 33s ago 85s 0.18.1 docker.io/prom/node-exporter:latest e5a616e4b9cf e725ba55bfd7
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:node-exporter.b smithi060 running (82s) 31s ago 86s 0.18.1 docker.io/prom/node-exporter:latest e5a616e4b9cf da71c458ed71
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:osd.0 smithi203 running (3m) 33s ago 3m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be fbd8df58b740
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:osd.1 smithi203 running (3m) 33s ago 3m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be f82a0984e8cb
2020-05-20T13:08:09.188 INFO:teuthology.orchestra.run.smithi203.stdout:osd.2 smithi203 running (3m) 33s ago 3m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 885fb5dfd287
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.3 smithi203 running (2m) 33s ago 2m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 4e6e5b008f2e
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.4 smithi060 running (2m) 31s ago 2m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be f1714bd9a240
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.5 smithi060 running (2m) 31s ago 2m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be e00f2801348c
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.6 smithi060 running (2m) 31s ago 2m 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be 73b01fddb7dd
2020-05-20T13:08:09.189 INFO:teuthology.orchestra.run.smithi203.stdout:osd.7 smithi060 running (107s) 31s ago 110s 16.0.0-1734-gc1cc5045b00 quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 06207838c6be cebb5c6bf000
2020-05-20T13:08:09.190 INFO:teuthology.orchestra.run.smithi203.stdout:prometheus.a smithi060 running (42s) 31s ago 88s 2.18.1 docker.io/prom/prometheus:latest de242295e225 34d837c4f530
2020-05-20T13:08:09.190 INFO:teuthology.orchestra.run.smithi203.stdout:rgw.realm.zone.a smithi203 unknown 33s ago 102s <unknown> quay.io/ceph-ci/ceph:c1cc5045b00842201e98ed965e87b16c8b2acec8 <unknown> <unknown>
</pre>
<p>still the job succeeded:</p>
<p><a class="external" href="http://pulpito.ceph.com/swagner-2020-05-20_12:38:40-rados:cephadm-wip-swagner3-testing-2020-05-20-1009-distro-basic-smithi/5072816/">http://pulpito.ceph.com/swagner-2020-05-20_12:38:40-rados:cephadm-wip-swagner3-testing-2020-05-20-1009-distro-basic-smithi/5072816/</a></p>
Orchestrator - Bug #45427 (Resolved): cephadm: auth get failed: invalid entity_auth mon
https://tracker.ceph.com/issues/45427
2020-05-07T10:13:25Z
Sebastian Wagner
<p><a class="external" href="http://pulpito.ceph.com/mgfritch-2020-05-07_02:27:06-rados-wip-mgfritch-testing-2020-05-06-1821-distro-basic-smithi/5029062">http://pulpito.ceph.com/mgfritch-2020-05-07_02:27:06-rados-wip-mgfritch-testing-2020-05-06-1821-distro-basic-smithi/5029062</a></p>
<pre>
cephadm 2020-05-07T03:43:08.989542+0000 mgr.smithi154.qjpiuj (mgr.27922) 6 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(host_pattern='*'), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False}): auth get failed: invalid entity_auth mon
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/module.py", line 2219, in _apply_all_services
if self._apply_service(spec):
File "/usr/share/ceph/mgr/cephadm/module.py", line 2190, in _apply_service
create_func(daemon_id, host) # type: ignore
File "/usr/share/ceph/mgr/cephadm/module.py", line 2967, in _create_node_exporter
return self._create_daemon('node-exporter', daemon_id, host)
File "/usr/share/ceph/mgr/cephadm/module.py", line 2021, in _create_daemon
extra_ceph_config=extra_config.pop('config', ''))
File "/usr/share/ceph/mgr/cephadm/module.py", line 1974, in _get_config_and_keyring
'entity': ename,
File "/usr/share/ceph/mgr/mgr_module.py", line 1096, in check_mon_command
raise MonCommandFailed(f'{cmd_dict["prefix"]} failed: {r.stderr}')
mgr_module.MonCommandFailed: auth get failed: invalid entity_auth mon
</pre>
<p>(as a side note, why do we need the mon keyrig for the node_exporter?)</p>
Orchestrator - Bug #45081 (Resolved): cephadm: `upgrade check 15.2.1` : OrchestratorError: Failed...
https://tracker.ceph.com/issues/45081
2020-04-14T10:33:34Z
Sebastian Wagner
<pre>
Apr 14 11:22:43 ceph1 bash[37629]: debug 2020-04-14T09:22:42.997+0000 7ff504d1f700 -1 Remote method threw exception: Traceback (most recent call last):
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in wrapper
Apr 14 11:22:43 ceph1 bash[37629]: return AsyncCompletion(value=f(*args, **kwargs), name=f.__name__)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 3046, in upgrade_check
Apr 14 11:22:43 ceph1 bash[37629]: target_id, target_version = self._get_container_image_id(target_name)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 3029, in _get_container_image_id
Apr 14 11:22:43 ceph1 bash[37629]: image_name, host, '\n'.join(out)))
Apr 14 11:22:43 ceph1 bash[37629]: orchestrator._interface.OrchestratorError: Failed to pull 15.2.1 on ceph0:
Apr 14 11:22:43 ceph1 bash[37629]: debug 2020-04-14T09:22:42.997+0000 7ff504d1f700 -1 mgr handle_command module 'orchestrator' command handler threw exception: Remote method threw exception: Traceback (most recent call last):
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in wrapper
Apr 14 11:22:43 ceph1 bash[37629]: return AsyncCompletion(value=f(*args, **kwargs), name=f.__name__)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 3046, in upgrade_check
Apr 14 11:22:43 ceph1 bash[37629]: target_id, target_version = self._get_container_image_id(target_name)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 3029, in _get_container_image_id
Apr 14 11:22:43 ceph1 bash[37629]: image_name, host, '\n'.join(out)))
Apr 14 11:22:43 ceph1 bash[37629]: orchestrator._interface.OrchestratorError: Failed to pull 15.2.1 on ceph0:
Apr 14 11:22:43 ceph1 bash[37629]: debug 2020-04-14T09:22:42.997+0000 7ff504d1f700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command
Apr 14 11:22:43 ceph1 bash[37629]: return self.handle_command(inbuf, cmd)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 110, in handle_command
Apr 14 11:22:43 ceph1 bash[37629]: return dispatch[cmd['prefix']].call(self, cmd, inbuf)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/mgr_module.py", line 308, in call
Apr 14 11:22:43 ceph1 bash[37629]: return self.func(mgr, **kwargs)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 72, in <lambda>
Apr 14 11:22:43 ceph1 bash[37629]: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 63, in wrapper
Apr 14 11:22:43 ceph1 bash[37629]: return func(*args, **kwargs)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 920, in _upgrade_check
Apr 14 11:22:43 ceph1 bash[37629]: completion = self.upgrade_check(image=image, version=ceph_version)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1510, in inner
Apr 14 11:22:43 ceph1 bash[37629]: completion = self._oremote(method_name, args, kwargs)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1581, in _oremote
Apr 14 11:22:43 ceph1 bash[37629]: return mgr.remote(o, meth, *args, **kwargs)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/mgr_module.py", line 1515, in remote
Apr 14 11:22:43 ceph1 bash[37629]: args, kwargs)
Apr 14 11:22:43 ceph1 bash[37629]: RuntimeError: Remote method threw exception: Traceback (most recent call last):
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in wrapper
Apr 14 11:22:43 ceph1 bash[37629]: return AsyncCompletion(value=f(*args, **kwargs), name=f.__name__)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 3046, in upgrade_check
Apr 14 11:22:43 ceph1 bash[37629]: target_id, target_version = self._get_container_image_id(target_name)
Apr 14 11:22:43 ceph1 bash[37629]: File "/usr/share/ceph/mgr/cephadm/module.py", line 3029, in _get_container_image_id
Apr 14 11:22:43 ceph1 bash[37629]: image_name, host, '\n'.join(out)))
Apr 14 11:22:43 ceph1 bash[37629]: orchestrator._interface.OrchestratorError: Failed to pull 15.2.1 on ceph0:
</pre>
<p>And according to `ceph config-key dump mgr`, container_image_base is still set to default.</p>
<p>Environment:</p>
<ul>
<li>15.2.0</li>
<li>debian + docker</li>
</ul>
mgr - Bug #39644 (Resolved): mgr/zabbix: ERROR: test_zabbix (tasks.mgr.test_module_selftest.TestM...
https://tracker.ceph.com/issues/39644
2019-05-09T08:32:02Z
Sebastian Wagner
<pre>
======================================================================
ERROR: test_zabbix (tasks.mgr.test_module_selftest.TestModuleSelftest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_module_selftest.py", line 41, in test_zabbix
self._selftest_plugin("zabbix")
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_module_selftest.py", line 34, in _selftest_plugin
"mgr", "self-test", "module", module_name)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/ceph_manager.py", line 1157, in raw_cluster_cmd
stdout=StringIO(),
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 435, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi023 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr self-test module zabbix'
</pre>
<pre>
2019-05-08 20:53:12.238 7fdcc2030700 -1 Remote method threw exception: Traceback (most recent call last):
File "/usr/share/ceph/mgr/zabbix/module.py", line 458, in self_test
data = self.get_data()
File "/usr/share/ceph/mgr/zabbix/module.py", line 209, in get_data
data['[{0},raw_bytes_used]'.format(pool['name'])] = pool['stats']['raw_bytes_used']
KeyError: ('raw_bytes_used',)
2019-05-08 20:53:12.238 7fdcc2030700 -1 mgr.server reply reply (1) Operation not permitted Test failed: Remote method threw exception: Traceback (most recent call last):
File "/usr/share/ceph/mgr/zabbix/module.py", line 458, in self_test
data = self.get_data()
File "/usr/share/ceph/mgr/zabbix/module.py", line 209, in get_data
data['[{0},raw_bytes_used]'.format(pool['name'])] = pool['stats']['raw_bytes_used']
KeyError: ('raw_bytes_used',)
</pre>
<p><a class="external" href="http://qa-proxy.ceph.com/teuthology/swagner-2019-05-08_15:36:11-rados:mgr-wip-swagner-testing-distro-basic-smithi/3941021/teuthology.log">http://qa-proxy.ceph.com/teuthology/swagner-2019-05-08_15:36:11-rados:mgr-wip-swagner-testing-distro-basic-smithi/3941021/teuthology.log</a></p>
<p>Introduced in <a class="external" href="https://github.com/ceph/ceph/pull/26152">https://github.com/ceph/ceph/pull/26152</a></p>
<p>Greg, I've assigned it to you, as Dmitriy Rabotjagov is not part of the mgr project</p>