Ceph : Issueshttps://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2022-01-26T11:02:45ZCeph
Redmine Orchestrator - Bug #54018 (Resolved): Suspicious behavior when deleting a cluster (by running cep...https://tracker.ceph.com/issues/540182022-01-26T11:02:45ZRedouane Kachach Elhichou
<p>It seems that new files are created once we have already ran cephadm rm-cluster in the node.</p>
<p>Steps to reproduce the issue:<br />1) Create a new cluster with few nodes (3 i.e)<br />2) Once the cluster is up & running, delete it by running from the first node:<br /><pre>
- ceph orch pause
- cephadm rm-cluster --force --zap-osds --fsid <your_cluster_fs_id>
</pre><br />3) Observe the files remaining after this operaction<br /><pre>
> find / | grep <your_cluster_fs_id> | grep -v cgroup
[root@ceph-node-00 ~]# find / | grep 36e3c242-7e88-11ec-b7c7-52540039ec3f | grep -v cgroup
/run/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f
/run/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f/ceph-osd.2.asok
/run/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f/ceph-mgr.ceph-node-00.puqqms.asok
/run/cephadm/36e3c242-7e88-11ec-b7c7-52540039ec3f.lock
/tmp/var/lib/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f
</pre></p>
<p>4) Wait for some minutes (3/4)<br /> > re-run the find command again and you will see that new files have appear in the node</p>
<pre>
[root@ceph-node-00 ~]# find / | grep 36e3c242-7e88-11ec-b7c7-52540039ec3f | grep -v cgroup
/run/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f
/run/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f/ceph-osd.2.asok
/run/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f/ceph-mgr.ceph-node-00.puqqms.asok
/run/cephadm/36e3c242-7e88-11ec-b7c7-52540039ec3f.lock
/tmp/var/lib/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f
/var/log/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f
/var/log/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f/ceph-volume.log
/var/lib/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f
/var/lib/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f/selinux
/var/lib/ceph/36e3c242-7e88-11ec-b7c7-52540039ec3f/cephadm.b8155e009332629135b14912b69ce375925c8b28ed28167233f4661dc1bf7b7f
</pre>
<p>These new dirs/files (/var/lib/ceph/<fsid> i.e) seem to have been created by the cephadm (ran from another node).</p> mgr - Bug #54003 (Resolved): cephadm/ssh.py: Failed run-tox-mgr test in Jenkins builds due to inc...https://tracker.ceph.com/issues/540032022-01-24T18:22:11ZLaura Flores
<p>Affects Jenkins 'make check' tests.</p>
<p><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/89073/">https://jenkins.ceph.com/job/ceph-pull-requests/89073/</a><br /><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/89074/">https://jenkins.ceph.com/job/ceph-pull-requests/89074/</a><br /><pre><code class="text syntaxhl"><span class="CodeRay">cephadm/module.py:45: note: In module imported here,
cephadm/__init__.py:1: note: ... from here:
cephadm/ssh.py:15: error: Incompatible types in assignment (expression has type "None", variable has type Module)
cephadm/ssh.py: note: In member "_remote_connection" of class "SSHManager":
cephadm/ssh.py:88: error: Incompatible types in assignment (expression has type "Optional[SSHClientConnection]", variable has type "SSHClientConnection")
cephadm/ssh.py: note: In member "_execute_command" of class "SSHManager":
cephadm/ssh.py:147: error: Item "None" of "Union[bytes, str, None]" has no attribute "rstrip"
cephadm/ssh.py:147: error: Argument 1 to "rstrip" of "bytes" has incompatible type "str"; expected "Optional[bytes]"
cephadm/ssh.py:148: error: Item "None" of "Union[bytes, str, None]" has no attribute "rstrip"
cephadm/ssh.py:148: error: Argument 1 to "rstrip" of "bytes" has incompatible type "str"; expected "Optional[bytes]"
cephadm/ssh.py:149: error: Incompatible return value type (got "Tuple[Union[bytes, str, Any], Union[bytes, str, Any], Optional[int]]", expected "Tuple[str, str, int]")
cephadm/__init__.py:1: note: In module imported here:
cephadm/module.py:66: error: Incompatible types in assignment (expression has type "None", variable has type Module)
Found 8 errors in 2 files (checked 31 source files)
ERROR: InvocationError for command /home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/.tox/mypy/bin/mypy --config-file=../../mypy.ini -m alerts -m balancer -m cephadm -m crash -m dashboard -m devicehealth -m diskprediction_local -m hello -m influx -m iostat -m localpool -m mds_autoscaler -m mgr_module -m mgr_util -m mirroring -m nfs -m orchestrator -m pg_autoscaler -m progress -m prometheus -m rbd_support -m rook -m snap_schedule -m selftest -m stats -m status -m telegraf -m telemetry -m test_orchestrator -m volumes -m zabbix (exited with code 1)
</span></code></pre></p> mgr - Bug #49205 (Resolved): orchestrator/01-hosts.e2e-spec.ts failed in test_dashboard_e2e.shhttps://tracker.ceph.com/issues/492052021-02-08T03:47:25ZKefu Chaitchaikov@gmail.com
<pre>
2021-02-08T02:41:42.641 INFO:tasks.workunit.client.0.smithi068.stderr:+ cypress_run orchestrator/01-hosts.e2e-spec.ts
2021-02-08T02:41:42.641 INFO:tasks.workunit.client.0.smithi068.stderr:+ local specs=orchestrator/01-hosts.e2e-spec.ts
2021-02-08T02:41:42.641 INFO:tasks.workunit.client.0.smithi068.stderr:+ local timeout=
2021-02-08T02:41:42.641 INFO:tasks.workunit.client.0.smithi068.stderr:+ local 'override_config=ignoreTestFiles=*.po.ts,retries=0,testFiles=orchestrator/01-hosts.e2e-spec.ts'
2021-02-08T02:41:42.641 INFO:tasks.workunit.client.0.smithi068.stderr:+ '[' x '!=' x ']'
2021-02-08T02:41:42.641 INFO:tasks.workunit.client.0.smithi068.stderr:+ npx cypress run --browser chrome --headless --config 'ignoreTestFiles=*.po.ts,retries=0,testFiles=orchestrator/01-hosts.e2e-spec.ts'
2021-02-08T02:41:46.338 INFO:tasks.workunit.client.0.smithi068.stderr:Couldn't determine Mocha version
2021-02-08T02:41:46.360 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:41:46.396 INFO:tasks.workunit.client.0.smithi068.stderr:tput: No value for $TERM and no -T specified
2021-02-08T02:41:46.396 INFO:tasks.workunit.client.0.smithi068.stdout:================================================================================
2021-02-08T02:41:46.396 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:41:46.396 INFO:tasks.workunit.client.0.smithi068.stdout: (Run Starting)
2021-02-08T02:41:46.396 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:41:46.402 INFO:tasks.workunit.client.0.smithi068.stdout: ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
2021-02-08T02:41:46.402 INFO:tasks.workunit.client.0.smithi068.stdout: │ Cypress: 5.3.0 │
2021-02-08T02:41:46.402 INFO:tasks.workunit.client.0.smithi068.stdout: │ Browser: Chrome 88 (headless) │
2021-02-08T02:41:46.402 INFO:tasks.workunit.client.0.smithi068.stdout: │ Specs: 1 found (orchestrator/01-hosts.e2e-spec.ts) │
2021-02-08T02:41:46.403 INFO:tasks.workunit.client.0.smithi068.stdout: └────────────────────────────────────────────────────────────────────────────────────────────────┘
2021-02-08T02:41:46.403 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:41:46.403 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:41:46.404 INFO:tasks.workunit.client.0.smithi068.stdout:────────────────────────────────────────────────────────────────────────────────────────────────────
2021-02-08T02:41:46.404 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:41:46.404 INFO:tasks.workunit.client.0.smithi068.stdout: Running: orchestrator/01-hosts.e2e-spec.ts (1 of 1)
2021-02-08T02:41:49.754 INFO:tasks.workunit.client.0.smithi068.stderr:Couldn't determine Mocha version
2021-02-08T02:41:49.760 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:41:49.761 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:41:49.769 INFO:tasks.workunit.client.0.smithi068.stdout: Hosts page
2021-02-08T02:41:49.769 INFO:tasks.workunit.client.0.smithi068.stdout: when Orchestrator is available
2021-02-08T02:41:51.289 INFO:tasks.workunit.client.0.smithi068.stdout: ✓ should not add an exsiting host (1501ms)
2021-02-08T02:41:58.769 INFO:tasks.workunit.client.0.smithi068.stdout: ✓ should delete a host and add it back (7399ms)
2021-02-08T02:41:59.675 INFO:tasks.workunit.client.0.smithi068.stdout: ✓ should display inventory (879ms)
2021-02-08T02:42:00.608 INFO:tasks.workunit.client.0.smithi068.stdout: ✓ should display daemons (904ms)
2021-02-08T02:44:02.977 INFO:tasks.workunit.client.0.smithi068.stdout: 1) should edit host labels
2021-02-08T02:44:03.141 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:44:03.142 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:44:03.142 INFO:tasks.workunit.client.0.smithi068.stdout: 4 passing (2m)
2021-02-08T02:44:03.142 INFO:tasks.workunit.client.0.smithi068.stdout: 1 failing
2021-02-08T02:44:03.142 INFO:tasks.workunit.client.0.smithi068.stdout:
2021-02-08T02:44:03.142 INFO:tasks.workunit.client.0.smithi068.stdout: 1) Hosts page
2021-02-08T02:44:03.142 INFO:tasks.workunit.client.0.smithi068.stdout: when Orchestrator is available
2021-02-08T02:44:03.142 INFO:tasks.workunit.client.0.smithi068.stdout: should edit host labels:
2021-02-08T02:44:03.142 INFO:tasks.workunit.client.0.smithi068.stdout: AssertionError: Timed out retrying: expected [ ' bar foo ' ] to include 'foo'
2021-02-08T02:44:03.143 INFO:tasks.workunit.client.0.smithi068.stdout: at Context.eval (https://smithi068:8443/__cypress/tests?p=cypress/integration/orchestrator/01-hosts.e2e-spec.ts:201:42)
</pre>
<p>in <a class="email" href="mailto:remote/ubuntu@smithi068.front.sepia.ceph.com">remote/ubuntu@smithi068.front.sepia.ceph.com</a>/log/ed1517d4-69b5-11eb-8fde-001a4aab830c/ceph-mgr.a.log</p>
<pre>
2021-02-08T02:42:00.913+0000 7fdfa5a54700 0 [dashboard ERROR exception] Internal Server Error
Traceback (most recent call last):
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler
return handler(*args, **kwargs)
File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/home.py", line 135, in __call__
return serve_file(full_path)
File "/lib/python3.6/site-packages/cherrypy/lib/static.py", line 77, in serve_file
cptools.validate_since()
File "/lib/python3.6/site-packages/cherrypy/lib/cptools.py", line 116, in validate_since
raise cherrypy.HTTPRedirect([], 304)
cherrypy._cperror.HTTPRedirect: ([], 304)
</pre>
<p>the e2e test was still green in</p>
<p>- <a class="external" href="https://pulpito.ceph.com/yuriw-2021-02-03_20:10:29-rados:cephadm-wip-yuri6-testing-2021-02-03-0803-distro-basic-smithi/5854552/">https://pulpito.ceph.com/yuriw-2021-02-03_20:10:29-rados:cephadm-wip-yuri6-testing-2021-02-03-0803-distro-basic-smithi/5854552/</a>. tested sha1: b70a63a91759ac9ae5c696580d78ed2543409e9b<br />- <a class="external" href="https://pulpito.ceph.com/kchai-2021-02-07_16:22:52-rados-master-distro-basic-smithi/5865046/">https://pulpito.ceph.com/kchai-2021-02-07_16:22:52-rados-master-distro-basic-smithi/5865046/</a>, tested sha1: 4308142b0929dc07a17f511ca9000197e53309da</p>
<p>/a/kchai-2021-02-08_02:14:21-rados-wip-kefu2-testing-2021-02-08-0023-distro-basic-smithi/5865428</p> Dashboard - Bug #45567 (New): mgr/dashboard: Services page shows incorrect/outdated datahttps://tracker.ceph.com/issues/455672020-05-15T12:46:10ZVolker Theile
<p>After deploying a service, e.g. grafana, to a host, delete it and redeploy it to another host the Dashboard still shows the service is running on the first installed host.</p>
<p><img src="screenshot2.mp4" alt="" /></p> Orchestrator - Bug #43561 (Resolved): mgr/cephadm: mgr is not named after hostnamehttps://tracker.ceph.com/issues/435612020-01-11T01:07:54ZJosh Durgin
<p>cephadm boostrap --help claims it will be name after the hostname, but it is randomly generated instead. The monitor is named correctly:</p>
<pre>
--mgr-id MGR_ID mgr id (default: local hostname)
</pre>
<pre>
services:
mon: 1 daemons, quorum mira045 (age 44m)
mgr: zbdiyk(active, since 43m)
</pre>
<p>This was setup on mira045 with:</p>
<pre>
cephadm bootstrap --mon-ip 172.21.5.124
</pre> Orchestrator - Bug #43560 (Resolved): mgr/cephadm: daemon logs are not saved in /var/log/ceph on ...https://tracker.ceph.com/issues/435602020-01-11T01:03:37ZJosh Durgin
<p>This means that container logs disappear if a container crashes - they are no longer accessible via 'cephadm log'.</p>
<p>None of mon, mgr, or osd logs appear in /var/log/ceph, other than logs from ceph-osd --mkfs.</p>
<p>Cluster log, audit log, and ceph-volume logs are present. This is with docker on ubuntu 18.04.</p>
<pre>
$ sudo ls -la /var/log/ceph/e0325e2c-3406-11ea-84c3-00259034c576
total 508
drwxrwx--- 2 167 167 4096 Jan 11 00:55 .
drwxr-x--- 3 root root 4096 Jan 11 00:11 ..
-rw------- 1 167 167 31887 Jan 11 00:56 ceph.audit.log
-rw------- 1 167 167 198004 Jan 11 01:01 ceph.log
-rw-r--r-- 1 167 167 103876 Jan 11 00:26 ceph-osd.0.log
-rw-r--r-- 1 167 167 103876 Jan 11 00:55 ceph-osd.1.log
-rw-r--r-- 1 root root 49680 Jan 11 00:55 ceph-volume.log
</pre> Orchestrator - Bug #43225 (Resolved): cephadm: Host misconfiguration result in unreadable error m...https://tracker.ceph.com/issues/432252019-12-10T11:53:16ZSebastian Wagner
<pre>
[ceph: root@host-1 /]# ceph orchestrator service ls
Error EINVAL: Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 1062, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
File "/usr/share/ceph/mgr/mgr_module.py", line 303, in call
return self.func(mgr, **kwargs)
File "/usr/share/ceph/mgr/orchestrator.py", line 131, in wrapper
return func(*args, **kwargs)
File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 275, in _list_services
orchestrator.raise_if_exception(completion)
File "/usr/share/ceph/mgr/orchestrator.py", line 630, in raise_if_exception
raise Exception(str(c.exception))
Exception: -F /tmp/ceph-mgr-ssh-conf-XxXxX -i /tmp/ceph-mgr-ssh-identity-YyYyY root@host-3
</pre>
<p>We'll need to provide better error messages here.</p>