Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2021-09-14T11:29:31Z
Ceph
Redmine
mgr - Bug #52603 (Duplicate): pacific ERROR: test_diskprediction_local (tasks.mgr.test_module_sel...
https://tracker.ceph.com/issues/52603
2021-09-14T11:29:31Z
Sebastian Wagner
<pre>
2021-09-12T15:40:17.598 INFO:teuthology.orchestra.run.smithi033.stderr:2021-09-12T15:40:17.603+0000 7f11dac3c700 20 mgrc start_command cmd: [{"prefix": "mgr self-test module", "module": "diskprediction_local", >
2021-09-12T15:40:17.598 INFO:teuthology.orchestra.run.smithi033.stderr:2021-09-12T15:40:17.603+0000 7f11dac3c700 1 -- 172.21.15.33:0/2949193497 --> [v2:172.21.15.200:6824/18725,v1:172.21.15.200:6825/18725] -- >
2021-09-12T15:40:17.599 INFO:tasks.ceph.mgr.z.smithi200.stderr:2021-09-12T15:40:17.607+0000 7f0443f9c700 -1 no module 'diskprediction_local'
2021-09-12T15:40:17.600 INFO:tasks.ceph.mgr.z.smithi200.stderr:2021-09-12T15:40:17.607+0000 7f0443f9c700 -1 mgr handle_command module 'selftest' command handler threw exception: Module not found
2021-09-12T15:40:17.601 INFO:tasks.ceph.mgr.z.smithi200.stderr:2021-09-12T15:40:17.607+0000 7f0443f9c700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
2021-09-12T15:40:17.601 INFO:tasks.ceph.mgr.z.smithi200.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 1353, in _handle_command
2021-09-12T15:40:17.602 INFO:tasks.ceph.mgr.z.smithi200.stderr: return self.handle_command(inbuf, cmd)
2021-09-12T15:40:17.602 INFO:tasks.ceph.mgr.z.smithi200.stderr: File "/usr/share/ceph/mgr/selftest/module.py", line 142, in handle_command
2021-09-12T15:40:17.602 INFO:tasks.ceph.mgr.z.smithi200.stderr: r = self.remote(command['module'], "self_test")
2021-09-12T15:40:17.602 INFO:tasks.ceph.mgr.z.smithi200.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 1738, in remote
2021-09-12T15:40:17.603 INFO:tasks.ceph.mgr.z.smithi200.stderr: return self._ceph_dispatch_remote(module_name, method_name,
2021-09-12T15:40:17.603 INFO:tasks.ceph.mgr.z.smithi200.stderr:ImportError: Module not found
2021-09-12T15:40:17.603 INFO:tasks.ceph.mgr.z.smithi200.stderr:
</pre>
<p><a class="external" href="https://pulpito.ceph.com/yuriw-2021-09-12_15:18:23-rados-pacific-distro-basic-smithi/6386597">https://pulpito.ceph.com/yuriw-2021-09-12_15:18:23-rados-pacific-distro-basic-smithi/6386597</a></p>
mgr - Bug #51564 (Duplicate): Dashboard URL address is incorrect after bootstrap with IPV6 address
https://tracker.ceph.com/issues/51564
2021-07-07T12:27:06Z
Sebastian Wagner
<p>Description of problem:<br />After cephadm bootstrap with IPV6 address as mon ip gives the dashboard url which is incorrect,</p>
<p>It got missed square brackets</p>
<pre>
[ceph: root@magna081 /]# ceph mgr services
{
"dashboard": "https://2620:52:0:880:225:90ff:fefc:2536:8443/",
"prometheus": "http://2620:52:0:880:225:90ff:fefc:2536:9283/"
}
[root@magna081 ~]# curl -k https://2620:52:0:880:225:90ff:fefc:2536:8443/
curl: (3) Port number ended with ':'
</pre>
<p>working URL</p>
<pre>
[root@magna081 ~]# curl -k https://[2620:52:0:880:225:90ff:fefc:2536]:8443/
<!doctype html>
<html lang="en-US">
<head>
<meta charset="utf-8">
<title>Red Hat Ceph Storage</title>
<script>
document.write('<base href="' + document.location+ '" />');
</script>
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="icon" type="image/x-icon" id="cdFavicon" href="assets/RedHat_favicon_0319.svg">
<link rel="stylesheet" href="styles.ba0e881c90a69f89b28e.css"></head>
<body>
<noscript>
<div class="noscript container"
ng-if="false">
<div class="jumbotron alert alert-danger">
<h2 i18n>JavaScript required!</h2>
<p i18n>A browser with JavaScript enabled is required in order to use this service.</p>
<p i18n>When using Internet Explorer, please check your security settings and add this address to your trusted sites.</p>
</div>
</div>
</noscript>
<cd-root></cd-root>
<script src="runtime.15da3e1803be577b1f00.js" defer></script><script src="polyfills.b66d1515aae6fe3887b1.js" defer></script><script src="scripts.6bda3fa7e09a87cd4228.js" defer></script><script src="main.b310ff35ff1005ba4a64.js" defer></script></body>
</html>
</pre>
<p>Steps to Reproduce:</p>
<ol>
<li>Bootstrap with IPV6 address as --mon-ip</li>
<li>Get dashboard URL using `ceph mgr services`</li>
<li>try to login to dashboard</li>
</ol>
Orchestrator - Feature #49249 (Duplicate): cephadm: Automatically create OSDs after reinstalling ...
https://tracker.ceph.com/issues/49249
2021-02-11T16:49:37Z
Sebastian Wagner
<p><a class="issue tracker-6 status-1 priority-4 priority-default" title="Documentation: Document manually deploment of OSDs (New)" href="https://tracker.ceph.com/issues/46691">#46691</a> provides the manual process of deploying cephadm OSDs.</p>
<p>we should probably provide an automated way to do this.</p>
Orchestrator - Bug #48463 (Duplicate): mon.c: Error: invalid config provided: CapAdd and privileg...
https://tracker.ceph.com/issues/48463
2020-12-04T11:17:07Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2020-12-04_10:02:29-rados:cephadm-wip-jmolmo-testing-2020-12-02-1452-distro-basic-smithi/5680473/">https://pulpito.ceph.com/swagner-2020-12-04_10:02:29-rados:cephadm-wip-jmolmo-testing-2020-12-02-1452-distro-basic-smithi/5680473/</a></p>
<pre>
['/bin/podman', 'run', '--rm', '--net=host', '-e', 'CONTAINER_IMAGE=docker.io/ceph/ceph:v15.2.0', '-e', 'NODE_NAME=smithi135', '-v', '/var/log/ceph/c4502caa-3619-11eb-980d-001a4
aab830c:/var/log/ceph:z', '-v', '/tmp/ceph-tmpr_67xdiq:/etc/ceph/ceph.client.admin.keyring:z', '-v', '/tmp/ceph-tmpl0cckyum:/etc/ceph/ceph.conf:z', '-v', '/var/lib/ceph/c4502caa-3619-11eb-980d-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z', '--entrypoint', '/usr
/bin/ceph', 'docker.io/ceph/ceph:v15.2.0', 'config', 'generate-minimal-conf', '-o', '/var/lib/ceph/mon/ceph-a/config']
</pre>
<p>Turns out, we're now installing podman 2 and then strting the upgrade from 15.2.0, which does not support podman 2.</p>
Orchestrator - Bug #48277 (Duplicate): cephadm infer image: <none>:<none>, despite --filter dangl...
https://tracker.ceph.com/issues/48277
2020-11-18T13:49:38Z
Sebastian Wagner
<pre>
master:~ # podman --version
podman version 2.0.6
master:~ # podman images --filter label=ceph=True --filter dangling=false --format '{{.Repository}} {{.Tag}}'
<none> <none>
registry.suse.com/ses/7/ceph/ceph latest
</pre>
Orchestrator - Bug #48275 (Duplicate): cephadm: get_last_local_ceph_image returns "<none>:<none>"
https://tracker.ceph.com/issues/48275
2020-11-18T13:17:23Z
Sebastian Wagner
<pre>
node2:~ # cephadm ceph-volume inventory
Inferring fsid 83c06a6e-298b-11eb-9c2a-525400501d50
Using recent ceph image <none>:<none>
Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --net=host --entrypoint stat -e CONTAINER_IMAGE=<none>:<none> -e NODE_NAME=node2 <none>:<none> -c %u %g /var/lib/ceph
stat:stderr Error: invalid reference format
Traceback (most recent call last):
File "/usr/sbin/cephadm", line 6043, in <module>
r = args.func()
File "/usr/sbin/cephadm", line 1299, in _infer_fsid
return func()
File "/usr/sbin/cephadm", line 1358, in _infer_image
return func()
File "/usr/sbin/cephadm", line 3558, in command_ceph_volume
make_log_dir(args.fsid)
File "/usr/sbin/cephadm", line 1453, in make_log_dir
uid, gid = extract_uid_gid()
File "/usr/sbin/cephadm", line 2061, in extract_uid_gid
raise RuntimeError('uid/gid not found')
RuntimeError: uid/gid not found
</pre>
Orchestrator - Bug #47340 (Duplicate): _list_devices: 'NoneType' object has no attribute 'get'
https://tracker.ceph.com/issues/47340
2020-09-07T16:01:49Z
Sebastian Wagner
<p><a class="external" href="https://pulpito.ceph.com/swagner-2020-09-07_12:17:11-rados:cephadm-wip-swagner2-testing-2020-09-07-1101-distro-basic-smithi/5415754/">https://pulpito.ceph.com/swagner-2020-09-07_12:17:11-rados:cephadm-wip-swagner2-testing-2020-09-07-1101-distro-basic-smithi/5415754/</a></p>
<pre>
2020-09-07T12:59:11.859 INFO:teuthology.orchestra.run.smithi150.stderr:Error EINVAL: Traceback (most recent call last):
2020-09-07T12:59:11.860 INFO:teuthology.orchestra.run.smithi150.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 1177, in _handle_command
2020-09-07T12:59:11.860 INFO:teuthology.orchestra.run.smithi150.stderr: return self.handle_command(inbuf, cmd)
2020-09-07T12:59:11.860 INFO:teuthology.orchestra.run.smithi150.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 141, in handle_command
2020-09-07T12:59:11.861 INFO:teuthology.orchestra.run.smithi150.stderr: return dispatch[cmd['prefix']].call(self, cmd, inbuf)
2020-09-07T12:59:11.861 INFO:teuthology.orchestra.run.smithi150.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 318, in call
2020-09-07T12:59:11.861 INFO:teuthology.orchestra.run.smithi150.stderr: return self.func(mgr, **kwargs)
2020-09-07T12:59:11.861 INFO:teuthology.orchestra.run.smithi150.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 103, in <lambda>
2020-09-07T12:59:11.861 INFO:teuthology.orchestra.run.smithi150.stderr: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
2020-09-07T12:59:11.862 INFO:teuthology.orchestra.run.smithi150.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 92, in wrapper
2020-09-07T12:59:11.862 INFO:teuthology.orchestra.run.smithi150.stderr: return func(*args, **kwargs)
2020-09-07T12:59:11.862 INFO:teuthology.orchestra.run.smithi150.stderr: File "/usr/share/ceph/mgr/orchestrator/module.py", line 421, in _list_devices
2020-09-07T12:59:11.862 INFO:teuthology.orchestra.run.smithi150.stderr: if d.lsm_data.get('ledSupport', None):
2020-09-07T12:59:11.862 INFO:teuthology.orchestra.run.smithi150.stderr:AttributeError: 'NoneType' object has no attribute 'get'
</pre>
Orchestrator - Feature #46265 (Duplicate): test cephadm MDS deployment
https://tracker.ceph.com/issues/46265
2020-06-29T18:40:07Z
Sebastian Wagner
<p>right now, the test is broken.</p>
<p>workaround is to apply it manuall: <a class="external" href="https://github.com/ceph/ceph/blob/cedf2bbd13daba6059c98046969f646b944042ae/qa/suites/rados/cephadm/workunits/task/test_orch_cli.yaml#L14">https://github.com/ceph/ceph/blob/cedf2bbd13daba6059c98046969f646b944042ae/qa/suites/rados/cephadm/workunits/task/test_orch_cli.yaml#L14</a></p>
Orchestrator - Feature #45905 (Duplicate): cephadm: errors in serve() should create a HEALTH warning
https://tracker.ceph.com/issues/45905
2020-06-05T09:50:18Z
Sebastian Wagner
<p>othwerwise users need to search the mgr log for hints manually.</p>
Orchestrator - Documentation #45564 (Duplicate): cephadm: document workaround for accessing the a...
https://tracker.ceph.com/issues/45564
2020-05-15T09:35:17Z
Sebastian Wagner
<pre>
$ ceph daemon mgr.ceph03 config show
admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
</pre>
<pre>
$ ceph --admin-daemon /var/run/ceph/mgr.ceph03 config show
No such file or directory
</pre>
<p>Probably, users have to run</p>
<pre>
cephadm enter mgr.ceph03
</pre>
<p>to access the admin socket.</p>
Orchestrator - Bug #45258 (Duplicate): cephadm: iSCSIServiceSpec: user/password should be mandato...
https://tracker.ceph.com/issues/45258
2020-04-24T11:59:15Z
Sebastian Wagner
<p>Some arguments in iSCSIServiceSpec should be mandatory, like user/password/port or at least cephadm should generate default value for them.</p>
Orchestrator - Bug #45197 (Duplicate): cephadm: rgw: failed to bind address 0.0.0.0:80
https://tracker.ceph.com/issues/45197
2020-04-23T09:01:45Z
Sebastian Wagner
<p>Despite running as root, RGW still cannot bind to port 80.</p>
<pre>
Apr 22 22:36:04 node02 systemd[1]: Starting Ceph rgw.myorg.us-east-1.node02.wzdozc for xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx...
Apr 22 22:36:04 node02 podman[3306]: Error: no container with name or ID ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx-rgw.myorg.us-east-1.node02.wzdozc found: no such container
Apr 22 22:36:04 node02 systemd[1]: Started Ceph rgw.myorg.us-east-1.node02.wzdozc for xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx.
Apr 22 22:36:04 node02 podman[3316]: 2020-04-22 22:36:04.900573616 +0300 +03 m=+0.195480600 container create 2df051c99b6a1a2b069576796b7a283d195da289ebace699674e60e5671beb52 (image=docker.io/ceph/ceph:v15, name=ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx-rgw.myorg.us-east-1.node02.wzdozc)
Apr 22 22:36:04 node02 systemd[1]: Started libpod-conmon-2df051c99b6a1a2b069576796b7a283d195da289ebace699674e60e5671beb52.scope.
Apr 22 22:36:05 node02 systemd[1]: Started libcontainer container 2df051c99b6a1a2b069576796b7a283d195da289ebace699674e60e5671beb52.
Apr 22 22:36:05 node02 bash[1571]: debug 2020-04-22T19:36:05.146+0000 7fee9ce70700 1 mon.node02@1(peon).osd e55 e55: 4 total, 4 up, 4 in
Apr 22 22:36:05 node02 bash[1576]: debug 2020-04-22T19:36:05.389+0000 7f135643a700 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.1/rpm/el8/BUILD/ceph-15.2.1/src/cls/queue/cls_queue_src.cc:54: ERROR: queue_read_head: failed to decode queue start: buffer::end_of_buffer
Apr 22 22:36:05 node02 bash[1561]: debug 2020-04-22T19:36:05.462+0000 7f10ef94a700 0 log_channel(cluster) log [DBG] : pgmap v616: 129 pgs: 1 creating+activating, 12 creating+peering, 7 unknown, 109 active+clean; 1.9 KiB data, 29 MiB used, 20 GiB / 24 GiB avail
Apr 22 22:36:05 node02 bash[1576]: debug 2020-04-22T19:36:05.521+0000 7f135643a700 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.1/rpm/el8/BUILD/ceph-15.2.1/src/cls/queue/cls_queue_src.cc:54: ERROR: queue_read_head: failed to decode queue start: buffer::end_of_buffer
Apr 22 22:36:05 node02 bash[1576]: debug 2020-04-22T19:36:05.541+0000 7f135643a700 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.1/rpm/el8/BUILD/ceph-15.2.1/src/cls/queue/cls_queue_src.cc:54: ERROR: queue_read_head: failed to decode queue start: buffer::end_of_buffer
Apr 22 22:36:05 node02 bash[1576]: debug 2020-04-22T19:36:05.626+0000 7f135643a700 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.1/rpm/el8/BUILD/ceph-15.2.1/src/cls/queue/cls_queue_src.cc:54: ERROR: queue_read_head: failed to decode queue start: buffer::end_of_buffer
[.....]
Apr 22 22:36:06 node02 bash[1571]: audit 2020-04-22T19:36:05.125075+0000 mon.node01 (mon.0) 82 : audit [INF] from='client.? 192.168.100.101:0/3331055036' entity='client.rgw.myorg.us-east-1.node01.dgfdkv' cmd='[{"prefix": "osd pool set", "pool": "us-east-1.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished
Apr 22 22:36:06 node02 bash[1571]: cluster 2020-04-22T19:36:05.125132+0000 mon.node01 (mon.0) 83 : cluster [DBG] osdmap e55: 4 total, 4 up, 4 in
Apr 22 22:36:06 node02 bash[1571]: cluster 2020-04-22T19:36:05.463711+0000 mgr.node02.qbhwjb (mgr.44101) 604 : cluster [DBG] pgmap v616: 129 pgs: 1 creating+activating, 12 creating+peering, 7 unknown, 109 active+clean; 1.9 KiB data, 29 MiB used, 20 GiB / 24 GiB avail
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.337+0000 7fb93538f240 0 set uid:gid to 167:167 (ceph:ceph)
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.337+0000 7fb93538f240 0 ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable), process radosgw, pid 1
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.337+0000 7fb93538f240 0 framework: beast
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.337+0000 7fb93538f240 0 framework conf key: port, val: 80
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.337+0000 7fb93538f240 1 radosgw_Main not setting numa affinity
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.819+0000 7fb93538f240 0 framework: beast
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.819+0000 7fb93538f240 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.819+0000 7fb93538f240 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key
Apr 22 22:36:06 node02 bash[3314]: debug 2020-04-22T19:36:06.819+0000 7fb93538f240 0 starting handler: beast
Apr 22 22:36:06 node02 podman[3316]: 2020-04-22 22:36:06.890791974 +0300 +03 m=+2.185699058 container died 2df051c99b6a1a2b069576796b7a283d195da289ebace699674e60e5671beb52 (image=docker.io/ceph/ceph:v15, name=ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx-rgw.myorg.us-east-1.node02.wzdozc)
Apr 22 22:36:06 node02 systemd[1]: libpod-2df051c99b6a1a2b069576796b7a283d195da289ebace699674e60e5671beb52.scope: Consumed 512ms CPU time
Apr 22 22:36:07 node02 podman[3316]: 2020-04-22 22:36:07.033069665 +0300 +03 m=+2.327976749 container remove 2df051c99b6a1a2b069576796b7a283d195da289ebace699674e60e5671beb52 (image=docker.io/ceph/ceph:v15, name=ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx-rgw.myorg.us-east-1.node02.wzdozc)
Apr 22 22:36:07 node02 systemd[1]: ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx@rgw.myorg.us-east-1.node02.wzdozc.service: Main process exited, code=exited, status=13/n/a
Apr 22 22:36:07 node02 systemd[1]: ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx@rgw.myorg.us-east-1.node02.wzdozc.service: Failed with result 'exit-code'.
Apr 22 22:36:07 node02 bash[1571]: debug 2020-04-22T19:36:07.341+0000 7fee9ce70700 0 mon.node02@1(peon) e3 handle_command mon_command({"prefix": "osd pool set", "pool": "us-east-1.rgw.meta", "var": "pg_num", "val": "8"} v 0) v1
Apr 22 22:36:07 node02 bash[1571]: debug 2020-04-22T19:36:07.341+0000 7fee9ce70700 0 log_channel(audit) log [INF] : from='mgr.44101 192.168.100.102:0/2789319733' entity='mgr.node02.qbhwjb' cmd=[{"prefix": "osd pool set", "pool": "us-east-1.rgw.meta", "var": "pg_num", "val": "8"}]: dispatch
Apr 22 22:36:07 node02 bash[1561]: debug 2020-04-22T19:36:07.464+0000 7f10ef94a700 0 log_channel(cluster) log [DBG] : pgmap v617: 129 pgs: 1 creating+activating, 12 creating+peering, 116 active+clean; 2.2 KiB data, 30 MiB used, 20 GiB / 24 GiB avail; 0 B/s rd, 381 B/s wr, 1 op/s
Apr 22 22:36:07 node02 firewalld[956]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be removed in a future release. Please consider disabling it now.
[....]
cmd=[{"prefix": "osd pool set", "pool": "us-east-1.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch
Apr 22 22:36:19 node02 bash[3695]: debug 2020-04-22T19:36:19.533+0000 7f7e36959240 0 framework: beast
Apr 22 22:36:19 node02 bash[3695]: debug 2020-04-22T19:36:19.533+0000 7f7e36959240 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt
Apr 22 22:36:19 node02 bash[3695]: debug 2020-04-22T19:36:19.533+0000 7f7e36959240 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key
Apr 22 22:36:19 node02 bash[3695]: debug 2020-04-22T19:36:19.533+0000 7f7e36959240 0 starting handler: beast
Apr 22 22:36:19 node02 bash[3695]: debug 2020-04-22T19:36:19.552+0000 7f7e36959240 -1 failed to bind address 0.0.0.0:80: Permission denied
Apr 22 22:36:19 node02 bash[3695]: debug 2020-04-22T19:36:19.553+0000 7f7e36959240 -1 ERROR: failed initializing frontend
Apr 22 22:36:19 node02 systemd[1]: libpod-473588aa4eb799f976544fdbee5ca1068346f7c69c802cc037b3d7e933cee1f9.scope: Consumed 522ms CPU time
Apr 22 22:36:19 node02 podman[3697]: 2020-04-22 22:36:19.61003749 +0300 +03 m=+2.235166237 container died 473588aa4eb799f976544fdbee5ca1068346f7c69c802cc037b3d7e933cee1f9 (image=docker.io/ceph/ceph:v15, name=ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx-rgw.myorg.us-east-1.node02.wzdozc)
Apr 22 22:36:19 node02 podman[3697]: 2020-04-22 22:36:19.841846867 +0300 +03 m=+2.466975714 container remove 473588aa4eb799f976544fdbee5ca1068346f7c69c802cc037b3d7e933cee1f9 (image=docker.io/ceph/ceph:v15, name=ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx-rgw.myorg.us-east-1.node02.wzdozc)
Apr 22 22:36:19 node02 systemd[1]: ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx@rgw.myorg.us-east-1.node02.wzdozc.service: Main process exited, code=exited, status=13/n/a
Apr 22 22:36:19 node02 systemd[1]: ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx@rgw.myorg.us-east-1.node02.wzdozc.service: Failed with result 'exit-code'.
</pre>
<p>Might relate to <a class="external" href="https://github.com/rook/rook/issues/5106">https://github.com/rook/rook/issues/5106</a></p>
Orchestrator - Bug #44830 (Duplicate): cpehadm bootstrap: improve error message, if `host add` fails
https://tracker.ceph.com/issues/44830
2020-03-31T09:42:18Z
Sebastian Wagner
<pre>
sudo cephadm bootstrap --mon-ip 51.38.177.40 --skip-mon-network
INFO:cephadm:Verifying podman|docker is present...
INFO:cephadm:Verifying lvm2 is present...
INFO:cephadm:Verifying time synchronization is in place...
INFO:cephadm:Unit ntp.service is enabled and running
INFO:cephadm:Repeating the final host check...
INFO:cephadm:podman|docker (/usr/bin/podman) is present
INFO:cephadm:systemctl is present
INFO:cephadm:lvcreate is present
INFO:cephadm:Unit ntp.service is enabled and running
INFO:cephadm:Host looks OK
INFO:root:Cluster fsid: 008d3602-7326-11ea-ba3d-fa163ee4dee1
INFO:cephadm:Verifying IP 51.38.177.40 port 3300 ...
INFO:cephadm:Verifying IP 51.38.177.40 port 6789 ...
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
INFO:cephadm:Extracting ceph user uid/gid from container image...
INFO:cephadm:Creating initial keys...
INFO:cephadm:Creating initial monmap...
INFO:cephadm:Creating mon...
INFO:cephadm:Waiting for mon to start...
INFO:cephadm:Waiting for mon...
INFO:cephadm:Assimilating anything we can from ceph.conf...
INFO:cephadm:Generating new minimal ceph.conf...
INFO:cephadm:Restarting the monitor...
INFO:cephadm:Creating mgr...
INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring
INFO:cephadm:Wrote config to /etc/ceph/ceph.conf
INFO:cephadm:Waiting for mgr to start...
INFO:cephadm:Waiting for mgr...
INFO:cephadm:mgr not available, waiting (1/10)...
INFO:cephadm:mgr not available, waiting (2/10)...
INFO:cephadm:mgr not available, waiting (3/10)...
INFO:cephadm:mgr not available, waiting (4/10)...
INFO:cephadm:mgr not available, waiting (5/10)...
INFO:cephadm:Enabling cephadm module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 5...
INFO:cephadm:Setting orchestrator backend to cephadm...
INFO:cephadm:Generating ssh key...
INFO:cephadm:Wrote public SSH key to to /etc/ceph/ceph.pub
INFO:cephadm:Adding key to root@localhost's authorized_keys...
INFO:cephadm:Adding host vps767688...
INFO:cephadm:Non-zero exit code 2 from /usr/bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=vps767688 -v /var/log/ceph/008d3602-7326-11ea-ba3d-fa163ee4dee1:/var/log/ceph:z -v /tmp/ceph-tmpwsrur6qb:/etc/ceph/ceph.client.admin.key
ring:z -v /tmp/ceph-tmpwfhyqpvq:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15 orch host add vps767688
INFO:cephadm:/usr/bin/ceph:stderr Error ENOENT: Failed to connect to vps767688 (vps767688). Check that the host is reachable and accepts connections using the cephadm SSH key
Traceback (most recent call last):
File "/usr/sbin/cephadm", line 4245, in <module>
r = args.func()
File "/usr/sbin/cephadm", line 966, in _default_image
return func()
File "/usr/sbin/cephadm", line 2378, in command_bootstrap
cli(['orch', 'host', 'add', host])
File "/usr/sbin/cephadm", line 2239, in cli
).run(timeout=timeout)
File "/usr/sbin/cephadm", line 1972, in run
self.run_cmd(), desc=self.entrypoint, timeout=timeout)
File "/usr/sbin/cephadm", line 694, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=vps767688 -v /var/log/ceph/008d3602-7326-11ea-ba3d-fa163ee4dee1:/var/log/ceph:z -v /tmp/ceph-tmpwsrur6qb:/etc/ceph/ceph.client.admin.keyring:z -v
/tmp/ceph-tmpwfhyqpvq:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15 orch host add vps767688
</pre>
<p>We should provide a better error message, if the user needs to setup a custom ssh config when calling bootstrap.</p>
<p>e.g. add `--skip-ssh` and then later run</p>
<pre>
cpeh orch set backend cephadm
cpeh cephadm set-ssh-config -i ...
cpeh orch add host $(hostname)
</pre>
Orchestrator - Bug #44170 (Duplicate): Teuthology is testing unrelated container images
https://tracker.ceph.com/issues/44170
2020-02-17T14:46:56Z
Sebastian Wagner
<p>shaman build: <a class="external" href="https://shaman.ceph.com/builds/ceph/wip-swagner-testing/290ad805b6b133320a894170a2157f1ffb45ed45/default/191192/">https://shaman.ceph.com/builds/ceph/wip-swagner-testing/290ad805b6b133320a894170a2157f1ffb45ed45/default/191192/</a></p>
<p>chacra: <a class="external" href="https://3.chacra.ceph.com/r/ceph/wip-swagner-testing/290ad805b6b133320a894170a2157f1ffb45ed45/centos/8/flavors/default/x86_64/">https://3.chacra.ceph.com/r/ceph/wip-swagner-testing/290ad805b6b133320a894170a2157f1ffb45ed45/centos/8/flavors/default/x86_64/</a></p>
<pre>
Index of /r/ceph/wip-swagner-testing/290ad805b6b133320a894170a2157f1ffb45ed45/centos/8/flavors/default/x86_64/
../
repodata/ 17-Feb-2020 11:56 -
ceph-mgr-15.1.0-816.g290ad80.el8.x86_64.rpm 17-Feb-2020 11:34 1625520
ceph-mgr-debuginfo-15.1.0-816.g290ad80.el8.x86_..> 17-Feb-2020 11:45 37146640
</pre>
<p>pulpito tests: <a class="external" href="http://pulpito.ceph.com/swagner-2020-02-17_12:27:04-rados-wip-swagner-testing-distro-basic-smithi/">http://pulpito.ceph.com/swagner-2020-02-17_12:27:04-rados-wip-swagner-testing-distro-basic-smithi/</a></p>
<p>Error message:</p>
<pre>
2020-02-17T12:54:29.252 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 32, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/cephadm.py", line 664, in ceph_prometheus
str(len(nodes))] + nodes
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/cephadm.py", line 44, in _shell
**kwargs
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi105 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph-ci/ceph:290ad805b6b133320a894170a2157f1ffb45ed45 shell -c /home/ubuntu/cephtest/ceph.conf -k /home/ubuntu/cephtest/ceph.keyring --f
</pre>
<pre>
podman run quay.io/ceph-ci/ceph:290ad805b6b133320a894170a2157f1ffb45ed45 -- /bin/bash
$ ls -l /usr/share/ceph/mgr/
total 232
drwxr-xr-x 2 root root 4096 Feb 17 11:20 alerts
drwxr-xr-x 2 root root 4096 Feb 17 11:20 balancer
drwxr-xr-x 3 root root 4096 Feb 17 11:20 cephadm
drwxr-xr-x 2 root root 4096 Feb 17 11:20 crash
drwxr-xr-x 8 root root 4096 Feb 17 11:20 dashboard
drwxr-xr-x 2 root root 4096 Feb 17 11:20 devicehealth
drwxr-xr-x 3 root root 4096 Feb 17 11:20 diskprediction_local
drwxr-xr-x 2 root root 4096 Feb 17 11:20 influx
drwxr-xr-x 3 root root 4096 Feb 17 11:20 insights
drwxr-xr-x 2 root root 4096 Feb 17 11:20 iostat
drwxr-xr-x 2 root root 4096 Feb 17 11:20 k8sevents
drwxr-xr-x 2 root root 4096 Feb 17 11:20 localpool
-rw-r--r-- 1 root root 51038 Feb 14 08:43 mgr_module.py
-rw-r--r-- 1 root root 9579 Feb 14 08:43 mgr_util.py
-rw-r--r-- 1 root root 58996 Feb 14 08:43 orchestrator.py
drwxr-xr-x 2 root root 4096 Feb 17 11:20 orchestrator_cli
drwxr-xr-x 2 root root 4096 Feb 14 10:26 osd_perf_query
drwxr-xr-x 3 root root 4096 Feb 17 11:20 pg_autoscaler
drwxr-xr-x 2 root root 4096 Feb 17 11:20 progress
drwxr-xr-x 2 root root 4096 Feb 17 11:20 prometheus
drwxr-xr-x 2 root root 4096 Feb 17 11:20 rbd_support
drwxr-xr-x 3 root root 4096 Feb 17 11:20 restful
drwxr-xr-x 3 root root 4096 Feb 17 11:20 rook
drwxr-xr-x 2 root root 4096 Feb 17 11:20 selftest
drwxr-xr-x 2 root root 4096 Feb 17 11:20 status
drwxr-xr-x 2 root root 4096 Feb 17 11:20 telegraf
drwxr-xr-x 2 root root 4096 Feb 17 11:20 telemetry
drwxr-xr-x 2 root root 4096 Feb 17 11:20 test_orchestrator
drwxr-xr-x 3 root root 4096 Feb 17 11:20 volumes
drwxr-xr-x 2 root root 4096 Feb 17 11:20 zabbix
</pre>
<p>which clearly doesn't contain the packages I just built in shaman</p>
Orchestrator - Feature #39057 (Duplicate): orchestrator_cli should check minimum device size
https://tracker.ceph.com/issues/39057
2019-04-01T08:07:47Z
Sebastian Wagner
<p>Otherwise we might get an error like this:</p>
<pre>
2019-04-01 08:01:11.465649 I | --> RuntimeError: Unable to use device 4.00 GB /dev/vdb, LVs would be smaller than 5GB
</pre>
<pre>
[root@kubic-1 /]# k get CephCluster -o yaml
apiVersion: v1
items:
- apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata: ...
spec:
cephVersion:
allowUnsupported: true
image: 192.168.122.1:443/ceph/ceph:latest
dashboard:
enabled: true
dataDirHostPath: /var/lib/rook
mon:
allowMultiplePerNode: true
count: 3
preferredCount: 0
network:
hostNetwork: false
rbdMirroring:
workers: 0
storage:
config:
databaseSizeMB: "1024"
journalSizeMB: "1024"
osdsPerDevice: "1"
directories:
- config: null
path: /var/lib/rook
nodes:
- config: null
devices:
- FullPath: ""
config: null
name: vdb
name: kubic-1
resources: {}
useAllDevices: false
status:
state: Updating
kind: List
metadata:
resourceVersion: ""
selfLink: ""
</pre>
<pre>
[root@kubic-1 /]# ceph orchestrator device ls
Host kubic-1:
Device Path Type Size Rotates Available Model
vdb hdd 5120M False False
vda hdd 24.0G False False
Host kubic-2:
Device Path Type Size Rotates Available Model
vdb hdd 5120M False False
vda hdd 24.0G False False
</pre>