Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2022-01-24T15:19:53Z
Ceph
Redmine
Orchestrator - Documentation #53997 (Resolved): cephadm spec: document "config" key
https://tracker.ceph.com/issues/53997
2022-01-24T15:19:53Z
Sebastian Wagner
<p>Users can specify config options for a particular service:</p>
<pre><code class="yaml syntaxhl"><span class="CodeRay"><span class="key">service_type</span>: <span class="string"><span class="content">mds</span></span>
<span class="key">service_id</span>: <span class="string"><span class="content">fsname</span></span>
<span class="key">placement</span>:
<span class="key">count</span>: <span class="string"><span class="content">2</span></span>
<span class="key">config</span>:
<span class="key">mds_cache_memory_limit</span>: <span class="string"><span class="content">8Gi</span></span>
</span></code></pre>
<p>This is also going to be cleaned up properly. Let's document it!</p>
Orchestrator - Bug #53939 (Resolved): ceph-nfs-upgrade, pacific: Upgrade Paused due to UPGRADE_RE...
https://tracker.ceph.com/issues/53939
2022-01-19T16:07:48Z
Sebastian Wagner
<pre>
mon[102341]: : cluster [WRN] Health check failed: Upgrading daemon osd.0 on host smithi103 failed. (UPGRADE_REDEPLOY_DAEMON)
mon[66897]: cephadm 2022-01-18T16:27:48.439275+0000 mgr.smithi103.wyeocw (mgr.14712) 129 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Redeploy daemon osd.0 ...
mon[66897]: Non-zero exit code 1 from systemctl start ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0
mon[66897]: systemctl: stderr Job for ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service failed because a timeout was exceeded.
mon[66897]: systemctl: stderr See "systemctl status ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service" and "journalctl -xe" for details.
mon[66897]: Traceback (most recent call last):
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 8615, in <module>
mon[66897]: main()
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 8603, in main
mon[66897]: r = ctx.func(ctx)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 1790, in _default_image
mon[66897]: return func(ctx)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 4603, in command_deploy
mon[66897]: ports=daemon_ports)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 2715, in deploy_daemon
mon[66897]: c, osd_fsid=osd_fsid, ports=ports)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 2960, in deploy_daemon_units
mon[66897]: call_throws(ctx, ['systemctl', 'start', unit_name])
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 1469, in call_throws
mon[66897]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}')
mon[66897]: RuntimeError: Failed command: systemctl start ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0: Job for ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service failed because a timeout was exceeded.
mon[66897]: See "systemctl status ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service" and "journalctl -xe" for details.
mon[66897]: Traceback (most recent call last):
mon[66897]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1402, in _remote_connection
mon[66897]: yield (conn, connr)
mon[66897]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1295, in _run_cephadm
mon[66897]: code, '\n'.join(err)))
mon[66897]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Redeploy daemon osd.0 ...
mon[66897]: Non-zero exit code 1 from systemctl start ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0
mon[66897]: systemctl: stderr Job for ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service failed because a timeout was exceeded.
mon[66897]: systemctl: stderr See "systemctl status ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service" and "journalctl -xe" for details.
mon[66897]: Traceback (most recent call last):
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 8615, in <module>
mon[66897]: main()
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 8603, in main
mon[66897]: r = ctx.func(ctx)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 1790, in _default_image
mon[66897]: return func(ctx)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 4603, in command_deploy
mon[66897]: ports=daemon_ports)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 2715, in deploy_daemon
mon[66897]: c, osd_fsid=osd_fsid, ports=ports)
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 2960, in deploy_daemon_units
mon[66897]: call_throws(ctx, ['systemctl', 'start', unit_name])
mon[66897]: File "/var/lib/ceph/e287ac0e-7879-11ec-8c34-001a4aab830c/cephadm.c659ab77cc705b8440c5bb10bf729dd981addbc618204d30ac82f427ecc4779d", line 1469, in call_throws
mon[66897]: raise RuntimeError(f'Failed command: {" ".join(command)}: {s}')
mon[66897]: RuntimeError: Failed command: systemctl start ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0: Job for ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service failed because a timeout was exceeded.
mon[66897]: See "systemctl status ceph-e287ac0e-7879-11ec-8c34-001a4aab830c@osd.0.service" and "journalctl -xe" for details.
...
cephadm 2022-01-18T16:27:48.439412+0000 mgr.smithi103.wyeocw (mgr.14712) 130 : cephadm [ERR] Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.0 on host smithi103 failed.
</pre>
<p><a class="external" href="https://pulpito.ceph.com/swagner-2022-01-18_15:34:53-rados:cephadm-wip-swagner2-testing-2022-01-18-1242-pacific-distro-default-smithi/6624255">https://pulpito.ceph.com/swagner-2022-01-18_15:34:53-rados:cephadm-wip-swagner2-testing-2022-01-18-1242-pacific-distro-default-smithi/6624255</a></p>
Orchestrator - Feature #53562 (New): cephadm doesn't support osd crush_location_hook
https://tracker.ceph.com/issues/53562
2021-12-09T11:59:53Z
Sebastian Wagner
<p>crush_location_hook is a path to an executable that is executed in order to update the current OSD's crush location. Executed like so:</p>
<pre>
$crush_location_hook --cluster {cluster-name} --id {ID} --type {daemon-type}
</pre>
<p>and prints out the current crush locations.</p>
<p>Workarounds:</p>
<ul>
<li>For a per-host based location, we have: <a class="external" href="https://docs.ceph.com/en/latest/cephadm/host-management/#setting-the-initial-crush-location-of-host">https://docs.ceph.com/en/latest/cephadm/host-management/#setting-the-initial-crush-location-of-host</a> which should cover a lot of use cases.</li>
<li>Build a new container image locally and add the crush_location_hook executable to it. Then set the config option to the file path within the container</li>
</ul>
mgr - Bug #53538 (Resolved): mgr/stats: ZeroDivisionError
https://tracker.ceph.com/issues/53538
2021-12-08T13:37:49Z
Sebastian Wagner
<pre>
root@service-01-08020:~# ceph osd status storage-01-08002
Error EINVAL: Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 1623, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
File "/usr/share/ceph/mgr/mgr_module.py", line 416, in call
return self.func(mgr, **kwargs)
File "/usr/share/ceph/mgr/status/module.py", line 338, in handle_osd_status
wr_ops_rate = (self.get_rate("osd", osd_id.__str__(), "osd.op_w") +
File "/usr/share/ceph/mgr/status/module.py", line 28, in get_rate
return (data[-1][1] - data[-2][1]) // int(data[-1][0] - data[-2][0])
ZeroDivisionError: integer division or modulo by zero
</pre>
<p>Since those PRs:</p>
<ul>
<li><a class="external" href="https://github.com/ceph/ceph/pull/25337">https://github.com/ceph/ceph/pull/25337</a></li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/26270">https://github.com/ceph/ceph/pull/26270</a></li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/26270/files#diff-dc6485f717f4dce4863733896375af75963412ebb2abc4b62fcd1f5233eee07dR44">https://github.com/ceph/ceph/pull/26270/files#diff-dc6485f717f4dce4863733896375af75963412ebb2abc4b62fcd1f5233eee07dR44</a></li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/28603">https://github.com/ceph/ceph/pull/28603</a> </li>
<li><a class="external" href="https://tracker.ceph.com/issues/43224#note-11">https://tracker.ceph.com/issues/43224#note-11</a></li>
</ul>
<p>no one had the patience to look into this all over again.</p>
Orchestrator - Bug #53154 (New): t8y: cephadm: error: unrecognized arguments: --keep-logs
https://tracker.ceph.com/issues/53154
2021-11-04T09:38:54Z
Sebastian Wagner
<pre>
2021-11-03T13:15:09.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f2abfd4e-3ca4-11ec-8c28-001a4aab830c --force --keep-logs
2021-11-03T13:15:09.584 INFO:teuthology.orchestra.run.smithi191.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR]
2021-11-03T13:15:09.584 INFO:teuthology.orchestra.run.smithi191.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR]
2021-11-03T13:15:09.585 INFO:teuthology.orchestra.run.smithi191.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT]
2021-11-03T13:15:09.585 INFO:teuthology.orchestra.run.smithi191.stderr: [--retry RETRY] [--env ENV] [--no-container-init]
2021-11-03T13:15:09.585 INFO:teuthology.orchestra.run.smithi191.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deplo
y,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts}
2021-11-03T13:15:09.585 INFO:teuthology.orchestra.run.smithi191.stderr: ...
2021-11-03T13:15:09.585 INFO:teuthology.orchestra.run.smithi191.stderr:cephadm: error: unrecognized arguments: --keep-logs
2021-11-03T13:15:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 2
</pre>
<p><a class="external" href="https://pulpito.ceph.com/swagner-2021-11-03_11:47:26-orch:cephadm-wip-swagner-testing-2021-11-03-0958-distro-basic-smithi/6481219">https://pulpito.ceph.com/swagner-2021-11-03_11:47:26-orch:cephadm-wip-swagner-testing-2021-11-03-0958-distro-basic-smithi/6481219</a></p>
Orchestrator - Feature #52602 (Resolved): cephadm: Prometheus: 2.28: generic http based service d...
https://tracker.ceph.com/issues/52602
2021-09-14T10:53:51Z
Sebastian Wagner
<p><a class="external" href="https://promlabs.com/blog/2021/06/21/whats-new-in-prometheus-2-28#generic-http-based-service-discovery">https://promlabs.com/blog/2021/06/21/whats-new-in-prometheus-2-28#generic-http-based-service-discovery</a></p>
<p>Fewer reconfigs! Yay!</p>
Orchestrator - Bug #52109 (Won't Fix): test_cephadm.sh: Timeout('Port 8443 not free on 127.0.0.1.',)
https://tracker.ceph.com/issues/52109
2021-08-09T13:58:50Z
Sebastian Wagner
<p><a class="external" href="https://sentry.ceph.com/organizations/ceph/issues/1585/?project=2&query=is%3Aunresolved+%22workunit+test+cephadm%22&statsPeriod=14d">https://sentry.ceph.com/organizations/ceph/issues/1585/?project=2&query=is%3Aunresolved+%22workunit+test+cephadm%22&statsPeriod=14d</a></p>
<pre>
2021-08-08T08:51:36.788 INFO:tasks.workunit.client.0.smithi173.stdout:Aug 08 08:47:54 smithi173 ceph-mgr[45646]: Traceback (most recent call last):
2021-08-08T08:51:36.788 INFO:tasks.workunit.client.0.smithi173.stdout: File "/usr/share/ceph/mgr/dashboard/module.py", line 503, in serve
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: cherrypy.engine.start()
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 283, in start
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: raise e_info
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 268, in start
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: self.publish('start')
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout: File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 248, in publish
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout: raise exc
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout: cherrypy.process.wspbus.ChannelFailures: Timeout('Port 8443 not free on 127.0.0.1.',)
</pre>
<p><a class="external" href="http://qa-proxy.ceph.com/teuthology/kchai-2021-08-08_08:16:57-rados-wip-kefu-testing-2021-08-08-1031-distro-basic-smithi/6326974/teuthology.log">http://qa-proxy.ceph.com/teuthology/kchai-2021-08-08_08:16:57-rados-wip-kefu-testing-2021-08-08-1031-distro-basic-smithi/6326974/teuthology.log</a><br /><a class="external" href="https://pulpito.ceph.com/sage-2021-08-06_19:04:50-orch-master-distro-basic-smithi/">https://pulpito.ceph.com/sage-2021-08-06_19:04:50-orch-master-distro-basic-smithi/</a></p>
Orchestrator - Bug #51806 (Need More Info): cephadm: stopped contains end up in error state
https://tracker.ceph.com/issues/51806
2021-07-22T15:26:52Z
Sebastian Wagner
<pre>
[ceph: root@sebastians-laptop /]# ceph orch stop node-exporter
Scheduled to stop node-exporter.sebastians-laptop on host 'sebastians-laptop'
[ceph: root@sebastians-laptop /]# ceph orch ps --service-name node-exporter
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID
node-exporter.sebastians-laptop sebastians-laptop *:9100 error 3m ago 4m - - <unknown> <unknown>
</pre>
<pre>
➜ cephadm git:(cephadm-container-name-dashes) ✗ sudo systemctl status ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop | cat
● ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service - Ceph node-exporter.sebastians-laptop for b2f78482-ead5-11eb-9ac0-482ae35a5fbb
Loaded: loaded (/etc/systemd/system/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2021-07-22 16:04:16 CEST; 14s ago
Process: 2907757 ExecStartPre=/bin/rm -f /run/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service-pid /run/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service-cid (code=exited, status=0/SUCCESS)
Process: 2907758 ExecStart=/bin/bash /var/lib/ceph/b2f78482-ead5-11eb-9ac0-482ae35a5fbb/node-exporter.sebastians-laptop/unit.run (code=exited, status=0/SUCCESS)
Main PID: 2907955 (conmon)
Tasks: 8 (limit: 38293)
Memory: 2.7M
CPU: 562ms
CGroup: /system.slice/system-ceph\x2db2f78482\x2dead5\x2d11eb\x2d9ac0\x2d482ae35a5fbb.slice/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service
├─container
│ ├─2907958 /dev/init -- /bin/node_exporter --no-collector.timex --web.listen-address=:9100
│ └─2907960 /bin/node_exporter --no-collector.timex --web.listen-address=:9100
└─supervisor
└─2907955 /usr/bin/conmon --api-version 1 -c b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1 -u b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1/userdata -p /run/containers/storage/overlay-containers/b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1/userdata/pidfile -n ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb-node-exporter-sebastians-laptop --exit-dir /run/libpod/exits --socket-dir-path /run/libpod/socket -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1/userdata/oci-log --conmon-pidfile /run/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service-pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1
Jul 22 16:04:16 sebastians-laptop conmon[2907955]: time="2021-07-22T14:04:16Z" level=info msg=" - textfile" source="node_exporter.go:104"
Jul 22 16:04:16 sebastians-laptop conmon[2907955]: time="2021-07-22T14:04:16Z" level=info msg=" - time" source="node_exporter.go:104"
Jul 22 16:04:16 sebastians-laptop conmon[2907955]: time="2021-07-22T14:04:16Z" level=info msg=" - uname" source="node_exporter.go:104"
Jul 22 16:04:16 sebastians-laptop conmon[2907955]: time="2021-07-22T14:04:16Z" level=info msg=" - vmstat" source="node_exporter.go:104"
Jul 22 16:04:16 sebastians-laptop conmon[2907955]: time="2021-07-22T14:04:16Z" level=info msg=" - xfs" source="node_exporter.go:104"
Jul 22 16:04:16 sebastians-laptop conmon[2907955]: time="2021-07-22T14:04:16Z" level=info msg=" - zfs" source="node_exporter.go:104"
Jul 22 16:04:16 sebastians-laptop conmon[2907955]: time="2021-07-22T14:04:16Z" level=info msg="Listening on :9100" source="node_exporter.go:170"
Jul 22 16:04:16 sebastians-laptop podman[2907914]: 2021-07-22 16:04:16.490922626 +0200 CEST m=+0.269857979 container start b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb-node-exporter-sebastians-laptop, maintainer=The Prometheus Authors <prometheus-developers@googlegroups.com>)
Jul 22 16:04:16 sebastians-laptop bash[2907914]: b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1
Jul 22 16:04:16 sebastians-laptop systemd[1]: Started Ceph node-exporter.sebastians-laptop for b2f78482-ead5-11eb-9ac0-482ae35a5fbb.
➜ cephadm git:(cephadm-container-name-dashes) ✗ sudo systemctl status ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop | cat
● ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service - Ceph node-exporter.sebastians-laptop for b2f78482-ead5-11eb-9ac0-482ae35a5fbb
Loaded: loaded (/etc/systemd/system/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2021-07-22 16:05:04 CEST; 5s ago
Process: 2907757 ExecStartPre=/bin/rm -f /run/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service-pid /run/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service-cid (code=exited, status=0/SUCCESS)
Process: 2907758 ExecStart=/bin/bash /var/lib/ceph/b2f78482-ead5-11eb-9ac0-482ae35a5fbb/node-exporter.sebastians-laptop/unit.run (code=exited, status=0/SUCCESS)
Process: 2908848 ExecStop=/bin/bash -c /bin/podman stop ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb-node-exporter.sebastians-laptop ; bash /var/lib/ceph/b2f78482-ead5-11eb-9ac0-482ae35a5fbb/node-exporter.sebastians-laptop/unit.stop (code=exited, status=0/SUCCESS)
Process: 2909007 ExecStopPost=/bin/bash /var/lib/ceph/b2f78482-ead5-11eb-9ac0-482ae35a5fbb/node-exporter.sebastians-laptop/unit.poststop (code=exited, status=0/SUCCESS)
Process: 2909008 ExecStopPost=/bin/rm -f /run/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service-pid /run/ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service-cid (code=exited, status=0/SUCCESS)
Main PID: 2907955 (code=exited, status=143)
CPU: 1.030s
Jul 22 16:04:16 sebastians-laptop systemd[1]: Started Ceph node-exporter.sebastians-laptop for b2f78482-ead5-11eb-9ac0-482ae35a5fbb.
Jul 22 16:05:03 sebastians-laptop systemd[1]: Stopping Ceph node-exporter.sebastians-laptop for b2f78482-ead5-11eb-9ac0-482ae35a5fbb...
Jul 22 16:05:03 sebastians-laptop bash[2908849]: Error: no container with name or ID "ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb-node-exporter.sebastians-laptop" found: no such container
Jul 22 16:05:04 sebastians-laptop podman[2908926]: 2021-07-22 16:05:04.364723428 +0200 CEST m=+0.120139014 container remove b771c1403b134d57e8378aa979b297257a14880c249ce901263e1e771725c1d1 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb-node-exporter-sebastians-laptop, maintainer=The Prometheus Authors <prometheus-developers@googlegroups.com>)
Jul 22 16:05:04 sebastians-laptop bash[2908886]: ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb-node-exporter-sebastians-laptop
Jul 22 16:05:04 sebastians-laptop systemd[1]: ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service: Main process exited, code=exited, status=143/n/a
Jul 22 16:05:04 sebastians-laptop bash[2908968]: Error: no container with name or ID "ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb-node-exporter.sebastians-laptop" found: no such container
Jul 22 16:05:04 sebastians-laptop systemd[1]: ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service: Failed with result 'exit-code'.
Jul 22 16:05:04 sebastians-laptop systemd[1]: Stopped Ceph node-exporter.sebastians-laptop for b2f78482-ead5-11eb-9ac0-482ae35a5fbb.
Jul 22 16:05:04 sebastians-laptop systemd[1]: ceph-b2f78482-ead5-11eb-9ac0-482ae35a5fbb@node-exporter.sebastians-laptop.service: Consumed 1.030s CPU time.
➜ cephadm git:(cephadm-container-name-dashes) ✗
</pre>
<p>might be related to conmon is exiting with 143 instead of 0</p>
Orchestrator - Bug #51361 (New): KillMode=none is deprecated
https://tracker.ceph.com/issues/51361
2021-06-25T09:05:39Z
Sebastian Wagner
<p>We chaged systemd unit file killmode to none in <a class="external" href="https://github.com/ceph/ceph/pull/33162#issuecomment-584183316">https://github.com/ceph/ceph/pull/33162#issuecomment-584183316</a></p>
<p>Now we're getting a new warning:</p>
<pre>
Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.
</pre>
Orchestrator - Bug #49287 (New): podman: setting cgroup config for procHooks process caused: Unit...
https://tracker.ceph.com/issues/49287
2021-02-13T00:54:57Z
Sebastian Wagner
<pre>
2021-02-12T16:27:55.195 INFO:teuthology.orchestra.run.smithi014.stderr:Non-zero exit code 127 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:52fc503cf18cf3bb446b840ba00be073017b8373 -e NODE_NAME=smithi014 quay.ceph.io/ceph-ci/ceph:52fc503cf18cf3bb446b840ba0
0be073017b8373 -c %u %g /var/lib/ceph
2021-02-12T16:27:55.195 INFO:teuthology.orchestra.run.smithi014.stderr:stat: stderr Error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-056038e1126191fba41d8a037275136f2d7aeec9710b9ee
ff792c06d8544b983.scope not found.: OCI runtime error
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:Traceback (most recent call last):
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: File "/home/ubuntu/cephtest/cephadm", line 7697, in <module>
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: main()
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: File "/home/ubuntu/cephtest/cephadm", line 7686, in main
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: r = ctx.func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: File "/home/ubuntu/cephtest/cephadm", line 1566, in _infer_fsid
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: return func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: File "/home/ubuntu/cephtest/cephadm", line 1603, in _infer_config
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: return func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: File "/home/ubuntu/cephtest/cephadm", line 1650, in _infer_image
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr: return func(ctx)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr: File "/home/ubuntu/cephtest/cephadm", line 4128, in command_shell
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr: make_log_dir(ctx, ctx.fsid)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr: File "/home/ubuntu/cephtest/cephadm", line 1752, in make_log_dir
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr: uid, gid = extract_uid_gid(ctx)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr: File "/home/ubuntu/cephtest/cephadm", line 2428, in extract_uid_gid
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr: raise RuntimeError('uid/gid not found')
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:RuntimeError: uid/gid not found
</pre>
<p><a class="external" href="https://pulpito.ceph.com/swagner-2021-02-11_11:00:52-rados:cephadm-wip-swagner3-testing-2021-02-10-1322-distro-basic-smithi/5874630">https://pulpito.ceph.com/swagner-2021-02-11_11:00:52-rados:cephadm-wip-swagner3-testing-2021-02-10-1322-distro-basic-smithi/5874630</a></p>
CephFS - Bug #48873 (Triaged): test_cluster_set_reset_user_config: AssertionError: NFS Ganesha cl...
https://tracker.ceph.com/issues/48873
2021-01-14T11:09:59Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2021-01-13_11:19:08-rados:cephadm-wip-swagner3-testing-2021-01-12-1316-distro-basic-smithi/5783002/">https://pulpito.ceph.com/swagner-2021-01-13_11:19:08-rados:cephadm-wip-swagner3-testing-2021-01-12-1316-distro-basic-smithi/5783002/</a></p>
<pre>
2021-01-13T11:55:37.749 INFO:tasks.cephfs_test_runner:Starting test: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2021-01-13T11:55:37.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_nfs.TestNFS.test_cluster_set_reset_user_config'
2021-01-13T11:55:38.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl status nfs-server
2021-01-13T11:55:38.777 INFO:teuthology.orchestra.run.smithi043.stdout:* nfs-server.service - NFS server and services
2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout: Loaded: loaded (/lib/systemd/system/nfs-server.service; disabled; vendor preset: enabled)
2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout: Active: inactive (dead)
2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout:
2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:43:35 smithi006 systemd[1]: Starting NFS server and services...
2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:43:35 smithi006 systemd[1]: Started NFS server and services.
2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:55:10 smithi043 systemd[1]: Stopping NFS server and services...
2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:55:10 smithi043 systemd[1]: Stopped NFS server and services.
2021-01-13T11:55:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 3
2021-01-13T11:55:38.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph nfs cluster create cephfs test
2021-01-13T11:55:39.183 INFO:teuthology.orchestra.run.smithi043.stdout:NFS Cluster Created Successfully
2021-01-13T11:55:49.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:55:49.532 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:56:09.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:56:09.883 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:56:39.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:56:40.234 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:57:20.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:57:20.590 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:58:10.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:58:10.942 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:59:10.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:59:11.291 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:59:11.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_cluster_set_reset_user_config'
2021-01-13T11:59:12.048 INFO:tasks.cephfs_test_runner:test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) ... FAIL
2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner:
2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner:======================================================================
2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner:FAIL: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2021-01-13T11:59:12.050 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-01-13T11:59:12.050 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 462, in test_cluster_set_reset_user_config
2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner: self._test_create_cluster()
2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 123, in _test_create_cluster
2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner: self._check_nfs_cluster_status('running', 'NFS Ganesha cluster deployment failed')
2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 88, in _check_nfs_cluster_status
2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner: self.fail(fail_msg)
2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner:AssertionError: NFS Ganesha cluster deployment failed
</pre>
<p>Interestingly, previous calls to _test_create_cluster succeeded.</p>
Orchestrator - Feature #48340 (Resolved): cephadm/rgw: Add rgw_zonegroup to RGWSpec
https://tracker.ceph.com/issues/48340
2020-11-24T12:42:24Z
Sebastian Wagner
<p>adding a rgw_zonegroup in the yaml and args and allowing a creation of that, since we're hardcoding the zonegroup default everywhere, and that'd have consequences if you're planning to create a non "default" named realm/zone/zonegroup as the system would end up creating a new zone with the spec. under the default zonegroup and not the user specified one!</p>
teuthology - Bug #47441 (Closed): teuthology/task/install: verify_package_version: RuntimeError: ...
https://tracker.ceph.com/issues/47441
2020-09-14T14:25:51Z
Sebastian Wagner
<pre>
2020-09-14T13:32:56.135 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-5509.g7f41e68.el8
2020-09-14T13:32:56.136 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 31, in nested
vars.append(enter())
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 218, in install
install_packages(ctx, package_list, config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 87, in install_packages
verify_package_version(ctx, config, remote)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 61, in verify_package_version
pkg=pkg_to_check
RuntimeError: ceph version 16.0.0-5509.g7f41e68c8af was not installed, found 16.0.0-5509.g7f41e68.el8.
</pre>
<p>Looks like the builds were duplicated: See <a class="external" href="https://shaman.ceph.com/repos/ceph/wip-swagner-testing-2020-09-14-1230/7f41e68c8afa3f6a917ca548770374067fdb433f/">https://shaman.ceph.com/repos/ceph/wip-swagner-testing-2020-09-14-1230/7f41e68c8afa3f6a917ca548770374067fdb433f/</a></p>
rbd - Bug #46875 (New): TestLibRBD.TestPendingAio: test_librbd.cc:4539: Failure or SIGSEGV
https://tracker.ceph.com/issues/46875
2020-08-10T01:03:45Z
Sebastian Wagner
<pre>
[ RUN ] TestLibRBD.TestPendingAio
using new format!
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc:4539: Failure
Expected equality of these values:
1
rbd_aio_is_complete(comps[i])
Which is: 0
[ FAILED ] TestLibRBD.TestPendingAio (68 ms)
</pre>
<p><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/57209/consoleFull#-361705261e840cee4-f4a4-4183-81dd-42855615f2c1">https://jenkins.ceph.com/job/ceph-pull-requests/57209/consoleFull#-361705261e840cee4-f4a4-4183-81dd-42855615f2c1</a></p>
sepia - Bug #46299 (Closed): Trying to pull docker.io/prom/prometheus:v2.18.1: too many request t...
https://tracker.ceph.com/issues/46299
2020-07-01T10:35:53Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2020-07-01_09:26:21-rados:cephadm-wip-swagner-testing-2020-07-01-0956-distro-basic-smithi/5194228/">https://pulpito.ceph.com/swagner-2020-07-01_09:26:21-rados:cephadm-wip-swagner-testing-2020-07-01-0956-distro-basic-smithi/5194228/</a></p>
<pre>
2020-07-01T10:04:28.253 INFO:tasks.cephadm:Adding local image mirror vossi04.front.sepia.ceph.com:5000
2020-07-01T10:04:28.301 DEBUG:teuthology.orchestra.remote:smithi189:/etc/containers/registries.conf is 4KB
2020-07-01T10:04:28.340 INFO:teuthology.orchestra.run.smithi189:> sudo sh -c 'cat > /etc/containers/registries.conf'
2020-07-01T10:04:28.400 DEBUG:teuthology.orchestra.remote:smithi205:/etc/containers/registries.conf is 4KB
2020-07-01T10:04:28.447 INFO:teuthology.orchestra.run.smithi205:> sudo sh -c 'cat > /etc/containers/registries.conf'
...
2020-07-01T10:14:26.759 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: debug 2020-07-01T10:14:26.700+0000 7f9405ffb700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploy daemon prometheus.a ...
2020-07-01T10:14:26.759 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:Verifying port 9095 ...
2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=prom/prometheus:v2.18.1 -e NODE_NAME=smithi205 --entrypoint stat prom/prometheus:v2.18.1 -c %u %g /etc/prometheus
2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:v2.18.1...
2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr name unknown: Repo not found
2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Trying to pull registry.fedoraproject.org/prom/prometheus:v2.18.1...
2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr manifest unknown: manifest unknown
2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Trying to pull registry.centos.org/prom/prometheus:v2.18.1...
2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr manifest unknown: manifest unknown
2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:v2.18.1...
2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr time="2020-07-01T10:10:18Z" level=error msg="HEADER map[Cache-Control:[no-cache] Content-Type:[application/json] Retry-After:[60]]"
2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr time="2020-07-01T10:11:20Z" level=error msg="HEADER map[Cache-Control:[no-cache] Content-Type:[application/json] Retry-After:[60]]"
2020-07-01T10:14:26.762 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr time="2020-07-01T10:12:22Z" level=error msg="HEADER map[Cache-Control:[no-cache] Content-Type:[application/json] Retry-After:[60]]"
2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr time="2020-07-01T10:13:24Z" level=error msg="HEADER map[Cache-Control:[no-cache] Content-Type:[application/json] Retry-After:[60]]"
2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr too many request to registry
2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Error: unable to pull prom/prometheus:v2.18.1: 4 errors occurred:
2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr * Error initializing source docker://registry.access.redhat.com/prom/prometheus:v2.18.1: Error reading manifest v2.18.1 in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found
2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr * Error initializing source docker://registry.fedoraproject.org/prom/prometheus:v2.18.1: Error reading manifest v2.18.1 in registry.fedoraproject.org/prom/prometheus: manifest unknown: manifest unknown
2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr * Error initializing source docker://registry.centos.org/prom/prometheus:v2.18.1: Error reading manifest v2.18.1 in registry.centos.org/prom/prometheus: manifest unknown: manifest unknown
2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr * Error parsing image configuration: too many request to registry
2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr
2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: Traceback (most recent call last):
2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 4847, in <module>
2020-07-01T10:14:26.765 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 1187, in _default_image
2020-07-01T10:14:26.765 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 2886, in command_deploy
2020-07-01T10:14:26.765 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 2818, in extract_uid_gid_monitoring
2020-07-01T10:14:26.765 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 1803, in extract_uid_gid
2020-07-01T10:14:26.766 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 2280, in run
2020-07-01T10:14:26.766 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 866, in call_throws
2020-07-01T10:14:26.766 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: RuntimeError: Failed command: /bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=prom/prometheus:v2.18.1 -e NODE_NAME=smithi205 --entrypoint stat prom/prometheus:v2.18.1 -c %u %g /etc/prometheus
</pre>
<p>Changing the registry mirror to `docker-mirror.front.sepia.ceph.com:5000` should work. I just can't do that myself.</p>