Project

General

Profile

Bug #58210

Rook orchestrator reading from wrong namespace

Added by Juan Miguel Olmo Martínez over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Category:
mgr/rook
Target version:
% Done:

100%

Source:
Community (user)
Tags:
Backport:
quincy
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Deviation from expected behavior:
Some dashboards do not load. Rook orchestrator fetching CephNFS/CephFS CR from wrong namespace.

Expected behavior:
Dashboards do load.

Seems it can be fix by using the correct namespace variable (example), although there are other hardcoded CR name.

How to reproduce it (minimal and precise):
create a cluster in non rook-ceph namespace
enabled rook orchestrator
create CephNFS
go to NFS dashboard

Related to:
https://github.com/rook/rook/issues/11386

debug 2022-12-06T03:56:39.722+0000 7f7bae8e0700  0 [rook ERROR orchestrator._interface] (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '87b51331-3262-4374-bdef-b5645d48e052', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '28374a6d-236b-4e03-a764-388189ef6555', 'X-Kubernetes-Pf-Prioritylevel-Uid': '72f430bf-7eb3-49a2-8725-a36c539ac7d7', 'Date': 'Tue, 06 Dec 2022 03:56:39 GMT', 'Content-Length': '363'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"cephnfses.ceph.rook.io is forbidden: User \"system:serviceaccount:openshift-storage:rook-ceph-mgr\" cannot list resource \"cephnfses\" in API group \"ceph.rook.io\" in the namespace \"rook-ceph\"","reason":"Forbidden","details":{"group":"ceph.rook.io","kind":"cephnfses"},"code":403}

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/rook/module.py", line 351, in describe_service
    all_nfs = self.rook_cluster.get_resource("cephnfses")
  File "/usr/share/ceph/mgr/rook/rook_cluster.py", line 1079, in get_resource
    return custom_objects.items
  File "/usr/share/ceph/mgr/rook/rook_cluster.py", line 260, in items
    resource_version = self._fetch()
  File "/usr/share/ceph/mgr/rook/rook_cluster.py", line 319, in _fetch
    response = self.api_func(**self.kwargs)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/api/custom_objects_api.py", line 1489, in list_namespaced_custom_object
    (data) = self.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, **kwargs)  # noqa: E501
  File "/usr/lib/python3.6/site-packages/kubernetes/client/api/custom_objects_api.py", line 1609, in list_namespaced_custom_object_with_http_info
    collection_formats=collection_formats)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 345, in call_api
    _preload_content, _request_timeout)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 366, in request
    headers=headers)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '87b51331-3262-4374-bdef-b5645d48e052', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '28374a6d-236b-4e03-a764-388189ef6555', 'X-Kubernetes-Pf-Prioritylevel-Uid': '72f430bf-7eb3-49a2-8725-a36c539ac7d7', 'Date': 'Tue, 06 Dec 2022 03:56:39 GMT', 'Content-Length': '363'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"cephnfses.ceph.rook.io is forbidden: User \"system:serviceaccount:openshift-storage:rook-ceph-mgr\" cannot list resource \"cephnfses\" in API group \"ceph.rook.io\" in the namespace \"rook-ceph\"","reason":"Forbidden","details":{"group":"ceph.rook.io","kind":"cephnfses"},"code":403}

debug 2022-12-06T03:56:39.724+0000 7f7bae8e0700 -1 Remote method threw exception: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/nfs/module.py", line 154, in cluster_ls
    return available_clusters(self)
  File "/usr/share/ceph/mgr/nfs/utils.py", line 39, in available_clusters
    orchestrator.raise_if_exception(completion)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 228, in raise_if_exception
    raise e
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '87b51331-3262-4374-bdef-b5645d48e052', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '28374a6d-236b-4e03-a764-388189ef6555', 'X-Kubernetes-Pf-Prioritylevel-Uid': '72f430bf-7eb3-49a2-8725-a36c539ac7d7', 'Date': 'Tue, 06 Dec 2022 03:56:39 GMT', 'Content-Length': '363'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"cephnfses.ceph.rook.io is forbidden: User \"system:serviceaccount:openshift-storage:rook-ceph-mgr\" cannot list resource \"cephnfses\" in API group \"ceph.rook.io\" in the namespace \"rook-ceph\"","reason":"Forbidden","details":{"group":"ceph.rook.io","kind":"cephnfses"},"code":403}

History

#1 Updated by Juan Miguel Olmo Martínez over 1 year ago

  • Pull request ID set to 49432

#2 Updated by Juan Miguel Olmo Martínez over 1 year ago

  • % Done changed from 0 to 80

#3 Updated by Juan Miguel Olmo Martínez about 1 year ago

  • Status changed from New to Resolved
  • % Done changed from 80 to 100

Also available in: Atom PDF