Project

General

Profile

Actions

Bug #39586

closed

dashboard: orchestrator mgr modules assert failure on iscsi service request

Added by Noah Watkins almost 5 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
High
Category:
Component - iSCSI
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

https://github.com/rook/rook/issues/3106

The orchestrator manager modules (rook and deepsea) both assert that service types fall into a fixed set which excludes the `iscsi` type. The dashboard requests this type, and fails when it subsequently attempts to iterate over NoneType that results from the assertion failure.

I think the orchestrator modules should be returning something like an empty set, but presumably dashboard and orchestrator should coordinate on this change.

::ffff:100.192.13.24 - - [03/May/2019:11:10:03] "GET /api/health/minimal HTTP/1.1" 500 1646 "https://rook-ceph-dev-primary.k8s.conmpany.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" 
debug 2019-05-03 11:10:03.630 7f0474a6e700  0 log_channel(cluster) log [DBG] : pgmap v193: 200 pgs: 100 active+undersized, 100 active+clean; 0 B data, 278 MiB used, 3.8 TiB / 3.9 TiB avail
debug 2019-05-03 11:10:05.631 7f0474a6e700  0 log_channel(cluster) log [DBG] : pgmap v194: 200 pgs: 100 active+undersized, 100 active+clean; 0 B data, 278 MiB used, 3.8 TiB / 3.9 TiB avail
debug 2019-05-03 11:10:07.631 7f0474a6e700  0 log_channel(cluster) log [DBG] : pgmap v195: 200 pgs: 100 active+undersized, 100 active+clean; 0 B data, 278 MiB used, 3.8 TiB / 3.9 TiB avail
::ffff:100.192.13.24 - - [03/May/2019:11:10:07] "GET /api/summary HTTP/1.1" 200 257 "https://rook-ceph-dev-primary.k8s.conmpany.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" 
::ffff:100.192.13.24 - - [03/May/2019:11:10:07] "GET /api/feature_toggles HTTP/1.1" 200 66 "https://rook-ceph-dev-primary.k8s.conmpany.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" 
::ffff:100.192.13.24 - - [03/May/2019:11:10:07] "GET /api/prometheus/notifications?from=last HTTP/1.1" 200 22 "https://rook-ceph-dev-primary.k8s.conmpany.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" 
debug 2019-05-03 11:10:08.072 7f0438724700  0 mgr[rook] Completion <read op> threw an exception:
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/rook/module.py", line 191, in wait
    c.execute()
  File "/usr/share/ceph/mgr/rook/module.py", line 57, in execute
    self._result = self.cb()
  File "/usr/share/ceph/mgr/rook/module.py", line 132, in <lambda>
    return RookReadCompletion(lambda: f(*args, **kwargs))
  File "/usr/share/ceph/mgr/rook/module.py", line 348, in describe_service
    assert service_type in ("mds", "osd", "mgr", "mon", "nfs", None), service_type + " unsupported" 
AssertionError: iscsi unsupported
[03/May/2019:11:10:08] HTTP 
Request Headers:
  AUTHORIZATION: Bearer XXXXXX
  REFERER: https://rook-ceph-dev-primary.k8s.conmpany.com/
  X-FORWARDED-HOST: rook-ceph-dev-primary.k8s.conmpany.com
  Remote-Addr: ::ffff:100.192.13.24
  X-REAL-IP: 127.0.0.1
  USER-AGENT: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36
  CONNECTION: close
  COOKIE: _ga=GA1.2.1677006237.1530686008; session_id=d057bedfc9b0909a5da6956eb3d4b8650dce540e
  ACCEPT: application/json, text/plain, */*
  PRAGMA: no-cache
  X-FORWARDED-PROTO: https
  X-ORIGINAL-URI: /api/health/minimal
  HOST: rook-ceph-dev-primary.k8s.conmpany.com
  CACHE-CONTROL: no-cache
  X-SCHEME: https
  ACCEPT-LANGUAGE: en-US,en;q=0.9,de;q=0.8
  X-FORWARDED-FOR: 127.0.0.1
  X-FORWARDED-PORT: 443
  X-REQUEST-ID: b8ebc25bb44f0b89f890092e3b508d89
  ACCEPT-ENCODING: gzip, deflate, br
[03/May/2019:11:10:08] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 197, in minimal
    return self.health_minimal.all_health()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 62, in all_health
    result['iscsi_daemons'] = self.iscsi_daemons()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 126, in iscsi_daemons
    gateways = IscsiGatewaysConfig.get_gateways_config()['gateways']
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 93, in get_gateways_config
    for instance in instances:
TypeError: 'NoneType' object is not iterable

debug 2019-05-03 11:10:08.073 7f0438724700  0 mgr[dashboard] [03/May/2019:11:10:08] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 197, in minimal
    return self.health_minimal.all_health()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 62, in all_health
    result['iscsi_daemons'] = self.iscsi_daemons()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 126, in iscsi_daemons
    gateways = IscsiGatewaysConfig.get_gateways_config()['gateways']
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 93, in get_gateways_config
    for instance in instances:
TypeError: 'NoneType' object is not iterable

Related issues 1 (0 open1 closed)

Copied to mgr - Backport #40037: nautilus: dashboard: orchestrator mgr modules assert failure on iscsi service requestResolvedSebastian WagnerActions
Actions #1

Updated by Sebastian Wagner almost 5 years ago

  • Pull request ID set to 26341
Actions #2

Updated by Sebastian Wagner almost 5 years ago

  • Project changed from Ceph to mgr
  • Category set to 141
  • Backport set to nautilus
Actions #3

Updated by Sebastian Wagner almost 5 years ago

  • Assignee set to Sebastian Wagner
Actions #4

Updated by Sebastian Wagner almost 5 years ago

just of of curiosity: how does your iscsi setup look like within the k8s cluster?

Actions #5

Updated by Maximilian Mack almost 5 years ago

Sebastian Wagner wrote:

just of of curiosity: how does your iscsi setup look like within the k8s cluster?

I'm the creator of the rook issue, and i have non iscsi setup at my k8s cluster at all..

Actions #6

Updated by Sebastian Wagner almost 5 years ago

  • Status changed from New to Fix Under Review
Actions #7

Updated by Kefu Chai almost 5 years ago

  • Status changed from Fix Under Review to Resolved
Actions #8

Updated by Sebastian Wagner almost 5 years ago

  • Status changed from Resolved to Pending Backport
Actions #9

Updated by Ricardo Marques almost 5 years ago

  • Copied to Backport #40037: nautilus: dashboard: orchestrator mgr modules assert failure on iscsi service request added
Actions #10

Updated by Nathan Cutler almost 5 years ago

  • Status changed from Pending Backport to Resolved
Actions #11

Updated by Ernesto Puerta about 3 years ago

  • Project changed from mgr to Dashboard
  • Category changed from 141 to Component - iSCSI
Actions

Also available in: Atom PDF