Project

General

Profile

Bug #39586

dashboard: orchestrator mgr modules assert failure on iscsi service request

Added by Noah Watkins 11 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
High
Category:
dashboard/iscsi
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

https://github.com/rook/rook/issues/3106

The orchestrator manager modules (rook and deepsea) both assert that service types fall into a fixed set which excludes the `iscsi` type. The dashboard requests this type, and fails when it subsequently attempts to iterate over NoneType that results from the assertion failure.

I think the orchestrator modules should be returning something like an empty set, but presumably dashboard and orchestrator should coordinate on this change.

::ffff:100.192.13.24 - - [03/May/2019:11:10:03] "GET /api/health/minimal HTTP/1.1" 500 1646 "https://rook-ceph-dev-primary.k8s.conmpany.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" 
debug 2019-05-03 11:10:03.630 7f0474a6e700  0 log_channel(cluster) log [DBG] : pgmap v193: 200 pgs: 100 active+undersized, 100 active+clean; 0 B data, 278 MiB used, 3.8 TiB / 3.9 TiB avail
debug 2019-05-03 11:10:05.631 7f0474a6e700  0 log_channel(cluster) log [DBG] : pgmap v194: 200 pgs: 100 active+undersized, 100 active+clean; 0 B data, 278 MiB used, 3.8 TiB / 3.9 TiB avail
debug 2019-05-03 11:10:07.631 7f0474a6e700  0 log_channel(cluster) log [DBG] : pgmap v195: 200 pgs: 100 active+undersized, 100 active+clean; 0 B data, 278 MiB used, 3.8 TiB / 3.9 TiB avail
::ffff:100.192.13.24 - - [03/May/2019:11:10:07] "GET /api/summary HTTP/1.1" 200 257 "https://rook-ceph-dev-primary.k8s.conmpany.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" 
::ffff:100.192.13.24 - - [03/May/2019:11:10:07] "GET /api/feature_toggles HTTP/1.1" 200 66 "https://rook-ceph-dev-primary.k8s.conmpany.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" 
::ffff:100.192.13.24 - - [03/May/2019:11:10:07] "GET /api/prometheus/notifications?from=last HTTP/1.1" 200 22 "https://rook-ceph-dev-primary.k8s.conmpany.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" 
debug 2019-05-03 11:10:08.072 7f0438724700  0 mgr[rook] Completion <read op> threw an exception:
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/rook/module.py", line 191, in wait
    c.execute()
  File "/usr/share/ceph/mgr/rook/module.py", line 57, in execute
    self._result = self.cb()
  File "/usr/share/ceph/mgr/rook/module.py", line 132, in <lambda>
    return RookReadCompletion(lambda: f(*args, **kwargs))
  File "/usr/share/ceph/mgr/rook/module.py", line 348, in describe_service
    assert service_type in ("mds", "osd", "mgr", "mon", "nfs", None), service_type + " unsupported" 
AssertionError: iscsi unsupported
[03/May/2019:11:10:08] HTTP 
Request Headers:
  AUTHORIZATION: Bearer XXXXXX
  REFERER: https://rook-ceph-dev-primary.k8s.conmpany.com/
  X-FORWARDED-HOST: rook-ceph-dev-primary.k8s.conmpany.com
  Remote-Addr: ::ffff:100.192.13.24
  X-REAL-IP: 127.0.0.1
  USER-AGENT: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36
  CONNECTION: close
  COOKIE: _ga=GA1.2.1677006237.1530686008; session_id=d057bedfc9b0909a5da6956eb3d4b8650dce540e
  ACCEPT: application/json, text/plain, */*
  PRAGMA: no-cache
  X-FORWARDED-PROTO: https
  X-ORIGINAL-URI: /api/health/minimal
  HOST: rook-ceph-dev-primary.k8s.conmpany.com
  CACHE-CONTROL: no-cache
  X-SCHEME: https
  ACCEPT-LANGUAGE: en-US,en;q=0.9,de;q=0.8
  X-FORWARDED-FOR: 127.0.0.1
  X-FORWARDED-PORT: 443
  X-REQUEST-ID: b8ebc25bb44f0b89f890092e3b508d89
  ACCEPT-ENCODING: gzip, deflate, br
[03/May/2019:11:10:08] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 197, in minimal
    return self.health_minimal.all_health()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 62, in all_health
    result['iscsi_daemons'] = self.iscsi_daemons()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 126, in iscsi_daemons
    gateways = IscsiGatewaysConfig.get_gateways_config()['gateways']
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 93, in get_gateways_config
    for instance in instances:
TypeError: 'NoneType' object is not iterable

debug 2019-05-03 11:10:08.073 7f0438724700  0 mgr[dashboard] [03/May/2019:11:10:08] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 197, in minimal
    return self.health_minimal.all_health()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 62, in all_health
    result['iscsi_daemons'] = self.iscsi_daemons()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 126, in iscsi_daemons
    gateways = IscsiGatewaysConfig.get_gateways_config()['gateways']
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 93, in get_gateways_config
    for instance in instances:
TypeError: 'NoneType' object is not iterable

Related issues

Copied to mgr - Backport #40037: nautilus: dashboard: orchestrator mgr modules assert failure on iscsi service request Resolved

History

#1 Updated by Sebastian Wagner 11 months ago

  • Pull request ID set to 26341

#2 Updated by Sebastian Wagner 11 months ago

  • Project changed from Ceph to mgr
  • Category set to dashboard/iscsi
  • Backport set to nautilus

#3 Updated by Sebastian Wagner 11 months ago

  • Assignee set to Sebastian Wagner

#4 Updated by Sebastian Wagner 11 months ago

just of of curiosity: how does your iscsi setup look like within the k8s cluster?

#5 Updated by Maximilian Mack 11 months ago

Sebastian Wagner wrote:

just of of curiosity: how does your iscsi setup look like within the k8s cluster?

I'm the creator of the rook issue, and i have non iscsi setup at my k8s cluster at all..

#6 Updated by Sebastian Wagner 11 months ago

  • Status changed from New to Fix Under Review

#7 Updated by Kefu Chai 10 months ago

  • Status changed from Fix Under Review to Resolved

#8 Updated by Sebastian Wagner 10 months ago

  • Status changed from Resolved to Pending Backport

#9 Updated by Ricardo Marques 10 months ago

  • Copied to Backport #40037: nautilus: dashboard: orchestrator mgr modules assert failure on iscsi service request added

#10 Updated by Nathan Cutler 10 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF