Project

General

Profile

Bug #53706

cephadm: Module 'cephadm' has failed: dashboard iscsi-gateway-rm failed: iSCSI gateway 'iscsi-gw' does not exist retval: -2

Added by Sebastian Wagner over 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Tags:
low-hanging-fruit
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

MGR_MODULE_ERROR: Module 'cephadm' has failed: dashboard iscsi-gateway-rm failed: iSCSI gateway 'iscsi-gw' does not exist retval: -2

This is a new one actually.

workaround 1

1. create a new iscsi gateway "iscsi-gw" in the dashboard
2. fail over the mgr (ceph mgr fail <currently active mgr>)

workaround 2

1. ceph config-key dump mgr/cephadm and look for the daemon iscsi-gw in the json output.
2. ceph config-key set <key> -i data.json and apply he json without the daemon. (just remove the daemon from the the json data)
3. fail over the mgr (ceph mgr fail <currently active mgr>)

workaround 3

Temporarily disable the dashboard during the removal operation.


Related issues

Related to Orchestrator - Bug #51546: cephadm: remove iscsi service fails when the dashboard isn't deployed Resolved

History

#1 Updated by Sebastian Wagner over 2 years ago

  • Related to Bug #51546: cephadm: remove iscsi service fails when the dashboard isn't deployed added

#2 Updated by Sebastian Wagner over 2 years ago

  • Description updated (diff)

#3 Updated by Melissa Li over 2 years ago

  • Assignee set to Melissa Li

#4 Updated by Melissa Li about 2 years ago

  • Status changed from New to In Progress
  • Pull request ID set to 44549

#5 Updated by Sebastian Wagner about 2 years ago

  • Status changed from In Progress to Fix Under Review

#6 Updated by Sebastian Wagner about 2 years ago

  • Description updated (diff)

#7 Updated by Manasa Gowri H S about 2 years ago

I am facing the same issue downstream and I tried the second workaround mentioned in this BZ. But it did not resolve the error for me. Now the cluster has no iscsi daemons and ceph orch ls looks like this. Could anyone please let me know if i tried the workaround properly or if i missed something?

[root@f25-h15-000-r730xd 40e96678-572d-49f8-bdb5-cf4b41d92615]# ceph orch ls
NAME RUNNING REFRESHED AGE PLACEMENT
alertmanager 1/1 14m ago 4h count:1;label:monitoring
crash 9/9 15m ago 4h label:ceph
grafana 1/1 14m ago 4h count:1;label:monitoring
iscsi.rbd 0/2 <deleting> 7m count:2;label:iscsigws
mds.cephfs 0/2 <deleting> 4h count:2;label:mdss
mgr 3/3 15m ago 4h count:3;label:mgrs
mon 5/5 15m ago 4h count:5;label:mons
node-exporter 9/9 15m ago 4h *
osd.unmanaged 80/80 14m ago - <unmanaged>
prometheus 1/1 14m ago 4h count:1;label:monitoring
rgw.f25-h21-000-r730xd 1/1 15m ago 4h f25-h21-000-r730xd.rdu2.scalelab.redhat.com;count-per-host:1
rgw.f25-h23-000-r730xd 1/1 14m ago 4h f25-h23-000-r730xd.rdu2.scalelab.redhat.com;count-per-host:1

[root@f25-h15-000-r730xd 40e96678-572d-49f8-bdb5-cf4b41d92615]# ceph orch ps --daemon_type iscsi
No daemons reported

[root@f25-h15-000-r730xd 40e96678-572d-49f8-bdb5-cf4b41d92615]# ceph -s
cluster:
id: 40e96678-572d-49f8-bdb5-cf4b41d92615
health: HEALTH_ERR
mons are allowing insecure global_id reclaim
2 failed cephadm daemon(s)
1 stray daemon(s) not managed by cephadm
1 filesystem is degraded
insufficient standby MDS daemons available
Module 'cephadm' has failed: dashboard iscsi-gateway-rm failed: iSCSI gateway 'f25-h23-000-r730xd.rdu2.scalelab.redhat.com' does not exist retval: -2
67 daemons have recently crashed

services:
mon: 5 daemons, quorum f25-h17-000-r730xd,f25-h15-000-r730xd,f25-h21-000-r730xd,f25-h19-000-r730xd,f25-h23-000-r730xd (age 2d)
mgr: f25-h15-000-r730xd(active, since 15m), standbys: f25-h17-000-r730xd, f25-h19-000-r730xd
mds: 1/1 daemons up
osd: 80 osds: 80 up (since 2d), 80 in (since 6w)
rgw: 2 daemons active (2 hosts, 1 zones)
data:
volumes: 0/1 healthy, 1 recovering
pools: 114 pools, 6665 pgs
objects: 11.77M objects, 118 GiB
usage: 2.4 TiB used, 5.4 TiB / 7.8 TiB avail
pgs: 6664 active+clean
1 active+clean+scrubbing+deep

#8 Updated by Adam King about 2 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF