Ceph » mgr

On a freshly installed non-cephadm Centos 9 Stream / Reef 18.2.2 cluster, each time a MGR is started it logs a crash due to a missing python3 module. It relates to the k8events dashboard module which is not enabled.

This does not prevent the enabled modules working. It merely creates nuisance crashes which put the cluster into a WARN state until they are archived.

Crash info:

# ceph crash info 2024-03-17T13:06:31.005076Z_9ee3dc3b-d3df-4a31-9392-2d95b800b091
{
    "archived": "2024-03-17 13:17:20.255706",
    "backtrace": [
        "  File \"/usr/share/ceph/mgr/k8sevents/__init__.py\", line 1, in <module>\n    from .module import Module",
        "  File \"/usr/share/ceph/mgr/k8sevents/module.py\", line 70, in <module>\n    from kubernetes.client.models.v1_event import V1Event",
        "ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event'" 
    ],
    "ceph_version": "18.2.2",
    "crash_id": "2024-03-17T13:06:31.005076Z_9ee3dc3b-d3df-4a31-9392-2d95b800b091",
    "entity_name": "mgr.xxxx1",
    "mgr_module": "k8sevents",
    "mgr_module_caller": "PyModule::load_subclass_of",
    "mgr_python_exception": "ModuleNotFoundError",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "9",
    "os_version_id": "9",
    "process_name": "ceph-mgr",
    "stack_sig": "3e6f3bd4c0c94e39be24c1a477ad1480cd0475cf18d128fe0503516405c48f88",
    "timestamp": "2024-03-17T13:06:31.005076Z",
    "utsname_hostname": "xxxx1.yyyyyy",
    "utsname_machine": "x86_64",
    "utsname_release": "6.1.82-1.el9.elrepo.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Fri Mar 15 18:18:05 EDT 2024" 
}

MGR module list:

# ceph mgr module ls
MODULE                              
balancer              on (always on)
crash                 on (always on)
devicehealth          on (always on)
orchestrator          on (always on)
pg_autoscaler         on (always on)
progress              on (always on)
rbd_support           on (always on)
status                on (always on)
telemetry             on (always on)
volumes               on (always on)
alerts                on            
dashboard             on            
iostat                on            
restful               on            
snap_schedule         on            
cephadm               -             
diskprediction_local  -             
influx                -             
insights              -             
k8sevents             -             
localpool             -             
mds_autoscaler        -             
mirroring             -             
nfs                   -             
osd_perf_query        -             
osd_support           -             
prometheus            -             
rgw                   -             
rook                  -             
selftest              -             
stats                 -             
telegraf              -             
test_orchestrator     -             
zabbix                -             

This looks similar to an issue from 3 years ago: https://tracker.ceph.com/issues/53044

The relevant package installed is:

# dnf list python3-kubernetes
Last metadata expiration check: 2:00:33 ago on Tue 19 Mar 2024 11:06:36 GMT.
Installed Packages
python3-kubernetes.noarch       1:26.1.0-2.el9       @epel