Actions
Bug #51546
closedcephadm: remove iscsi service fails when the dashboard isn't deployed
% Done:
0%
Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
If the iscsi service is removed and the dashboard isn't deployed (dashboard mgr module not enabled) then the cluster status goes to ERR and the removal is stuck is deleting state.
Steps to reproduce:
1. bootstrap a cluster without dashboard : cephadm bootstrap --mon-ip x.x.x.x --skip-dashboard --skip-monitoring-stack
2. add some OSDs
3. deploy the iscsi service
4. remove iscsi with : ceph orch rm iscsi.iscsi
Results:
# ceph orch ls --service_type iscsi NAME PORTS RUNNING REFRESHED AGE PLACEMENT iscsi.iscsi 0/1 <deleting> 22m cephaio
# ceph health detail
[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: dashboard iscsi-gateway-rm failed: Module 'dashboard' is not enabled (required by command 'dashboard iscsi-gateway-rm'): use `ceph mgr module enable dashboard` to enable it retval: -95
Module 'cephadm' has failed: dashboard iscsi-gateway-rm failed: Module 'dashboard' is not enabled (required by command 'dashboard iscsi-gateway-rm'): use `ceph mgr module enable dashboard` to enable it retval: -95
This is probably coming from [1] which added the `ceph dashboard iscsi-gateway-rm xxx` command during the service removal but this doesn't check if the dashboard module is enabled first.
[1] https://github.com/ceph/ceph/commit/1b9e3edcfd1c1a3dd02d6eb14072494f57b086a8
Updated by Dimitri Savineau almost 3 years ago
# ceph crash info 2021-07-06T19:29:15.296973Z_332c5a68-0343-4d71-808e-fcf28c21be86
{
"backtrace": [
" File \"/usr/share/ceph/mgr/cephadm/module.py\", line 501, in serve\n serve.serve()",
" File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 92, in serve\n self._check_daemons()",
" File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 871, in _check_daemons\n self._remove_daemon(dd.name(), dd.hostname)",
" File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 1111, in _remove_daemon\n self.mgr.cephadm_services[daemon_type_to_service(daemon_type)].post_remove(daemon)",
" File \"/usr/share/ceph/mgr/cephadm/services/iscsi.py\", line 156, in post_remove\n 'name': daemon.hostname,",
" File \"/usr/share/ceph/mgr/mgr_module.py\", line 1475, in check_mon_command\n raise MonCommandFailed(f'{cmd_dict[\"prefix\"]} failed: {r.stderr} retval: {r.retval}')",
"mgr_module.MonCommandFailed: dashboard iscsi-gateway-rm failed: Module 'dashboard' is not enabled (required by command 'dashboard iscsi-gateway-rm'): use `ceph mgr module enable dashboard` to enable it retval: -95"
],
"ceph_version": "17.0.0-5816-g0b361349",
"crash_id": "2021-07-06T19:29:15.296973Z_332c5a68-0343-4d71-808e-fcf28c21be86",
"entity_name": "mgr.cephaio.wtfcic",
"mgr_module": "cephadm",
"mgr_module_caller": "PyModuleRunner::serve",
"mgr_python_exception": "MonCommandFailed",
"os_id": "centos",
"os_name": "CentOS Linux",
"os_version": "8",
"os_version_id": "8",
"process_name": "ceph-mgr",
"stack_sig": "5b5aea59f2dae2a8008d9f8699a829717e91e94d1963ae2c17b20253557caf19",
"timestamp": "2021-07-06T19:29:15.296973Z",
"utsname_hostname": "cephaio",
"utsname_machine": "x86_64",
"utsname_release": "4.18.0-305.7.1.el8_4.x86_64",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP Tue Jun 29 21:55:12 UTC 2021"
}
Updated by Sebastian Wagner almost 3 years ago
- Status changed from New to Fix Under Review
Updated by Adam King almost 3 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Sebastian Wagner over 2 years ago
- Status changed from Pending Backport to Resolved
Updated by Sebastian Wagner over 2 years ago
- Related to Bug #52692: Iscsi gateways are not showing "UP" in dashboard added
Updated by Sebastian Wagner over 2 years ago
- Related to Bug #53706: cephadm: Module 'cephadm' has failed: dashboard iscsi-gateway-rm failed: iSCSI gateway 'iscsi-gw' does not exist retval: -2 added
Actions