Actions
Bug #64374
openError ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'mgr_module' (pass --force to force enablement)
Added by Prashant D 3 months ago. Updated 25 days ago.
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Tests are failing due to failed to import internal mgr_module :
2024-02-09T00:51:05.636 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 2 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -e NODE_NAME=smithi002 -v /var/log/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp20jrlflo:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv7483kqa:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 mgr module enable cephadm 2024-02-09T00:51:05.636 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'mgr_module' (pass --force to force enablement) 2024-02-09T00:51:05.636 INFO:teuthology.orchestra.run.smithi002.stderr:RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -e NODE_NAME=smithi002 -v /var/log/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp20jrlflo:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpv7483kqa:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'mgr_module' (pass --force to force enablement) 2024-02-09T00:51:05.636 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: *************** 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically. 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: broken installation, users must use the following command to completely delete the broken cluster: 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: > cephadm rm-cluster --force --zap-osds --fsid <fsid> 2024-02-09T00:51:05.637 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-09T00:51:05.638 INFO:teuthology.orchestra.run.smithi002.stdout: for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster 2024-02-09T00:51:05.638 INFO:teuthology.orchestra.run.smithi002.stdout: ***************
Updated by Prashant D 3 months ago
The mgr modules path seems to be not appended to PYTHONPATH :
[user@testnode ~]$ ssh user@smithi002.front.sepia.ceph.com
The authenticity of host 'smithi002.front.sepia.ceph.com (172.21.15.2)' can't be established.
ECDSA key fingerprint is SHA256:Pt5zs9W9FXd4V2S3CgJQO/bCPS+i2m/6nzun1513cXE.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'smithi002.front.sepia.ceph.com' (ECDSA) to the list of known hosts.
sudo su -
[user@smithi002 ~]$ sudo su -
[root@smithi002 ~]# curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri6-testing-2024-02-07-1000/eb583889924ee88759f8e8d8bd48edeb78bfd3c0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm
-rw-r--r--. 1 root root 766949 Feb 9 01:54 /home/ubuntu/cephtest/cephadm
[root@smithi002 ~]# chmod +x /home/ubuntu/cephtest/cephadm
[root@smithi002 ~]# sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 pull
Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0...
{
"ceph_version": "ceph version 19.0.0-1298-geb583889 (eb583889924ee88759f8e8d8bd48edeb78bfd3c0) squid (dev)",
"image_id": "38c4039b0e98b53ff89657a16fec4c63d01c188f936580ce26e5e09a61ad1a2d",
"repo_digests": [
"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:14bf4dbdc3d1ee48c9713ffb480ed1b59d64ddbe0851bf44ef27b7db227241b5"
]
}
[root@smithi002 ~]# vim /home/ubuntu/cephtest/seed.ceph.conf
[root@smithi002 ~]#
[root@smithi002 ~]#
[root@smithi002 ~]# sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -v bootstrap --fsid 1c7ef528-c6e5-11ee-95b6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring
--------------------------------------------------------------------------------
cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0', '-v', 'bootstrap', '--fsid', '1c7ef528-c6e5-11ee-95b6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.2', '--skip-admin-label']
/bin/podman: stdout 4.8.1
Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts.
...
Waiting for mon...
/usr/bin/ceph: stdout cluster:
/usr/bin/ceph: stdout id: 1c7ef528-c6e5-11ee-95b6-87774f69a715
/usr/bin/ceph: stdout health: HEALTH_OK
/usr/bin/ceph: stdout
/usr/bin/ceph: stdout services:
/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi002 (age 0.566787s)
/usr/bin/ceph: stdout mgr: no daemons active
/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in
/usr/bin/ceph: stdout
/usr/bin/ceph: stdout data:
/usr/bin/ceph: stdout pools: 0 pools, 0 pgs
/usr/bin/ceph: stdout objects: 0 objects, 0 B
/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail
/usr/bin/ceph: stdout pgs:
/usr/bin/ceph: stdout
mon is available
Assimilating anything we can from ceph.conf...
/usr/bin/ceph: stdout
/usr/bin/ceph: stdout [global]
/usr/bin/ceph: stdout fsid = 1c7ef528-c6e5-11ee-95b6-87774f69a715
/usr/bin/ceph: stdout mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789]
/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true
/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true
/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false
/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0
/usr/bin/ceph: stdout
/usr/bin/ceph: stdout [mgr]
/usr/bin/ceph: stdout mgr/telemetry/nag = false
/usr/bin/ceph: stdout
/usr/bin/ceph: stdout [osd]
/usr/bin/ceph: stdout osd_map_max_advance = 10
/usr/bin/ceph: stdout osd_sloppy_crc = true
Generating new minimal ceph.conf...
Restarting the monitor...
Setting public_network to 172.21.0.0/20 in mon config section
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 0.0.0.0:9283 ...
Verifying port 0.0.0.0:8765 ...
Verifying port 0.0.0.0:8443 ...
...
Enabling cephadm module...
Non-zero exit code 2 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -e NODE_NAME=smithi002 -v /var/log/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0x139bqo:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy3f55sb8:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 mgr module enable cephadm
/usr/bin/ceph: stderr Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'mgr_module' (pass --force to force enablement)
RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -e NODE_NAME=smithi002 -v /var/log/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0x139bqo:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy3f55sb8:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'mgr_module' (pass --force to force enablement)
***************
Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically.
To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous
broken installation, users must use the following command to completely delete the broken cluster:
> cephadm rm-cluster --force --zap-osds --fsid <fsid>
for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster
***************
Deleting cluster with fsid: 1c7ef528-c6e5-11ee-95b6-87774f69a715
/bin/podman: stdout e681ca382ed0,17.92MB / 33.27GB
/bin/podman: stdout 29e7368314f2,100.3MB / 33.27GB
/bin/podman: stdout e681ca382ed0,1.25%
/bin/podman: stdout 29e7368314f2,12.31%
systemctl: stderr Removed "/etc/systemd/system/ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715.target.wants/ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715@mon.smithi002.service".
Non-zero exit code 5 from systemctl stop ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mon.smithi002.service
systemctl: stderr Failed to stop ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mon.smithi002.service: Unit ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mon.smithi002.service not loaded.
Non-zero exit code 1 from systemctl reset-failed ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mon.smithi002.service
systemctl: stderr Failed to reset failed state of unit ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mon.smithi002.service: Unit ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mon.smithi002.service not loaded.
Non-zero exit code 1 from systemctl disable ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mon.smithi002.service
systemctl: stderr Failed to disable unit: Unit file ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mon.smithi002.service does not exist.
systemctl: stderr Removed "/etc/systemd/system/ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715.target.wants/ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715@mgr.smithi002.ahpaml.service".
Non-zero exit code 5 from systemctl stop ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mgr.smithi002.ahpaml.service
systemctl: stderr Failed to stop ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mgr.smithi002.ahpaml.service: Unit ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mgr.smithi002.ahpaml.service not loaded.
Non-zero exit code 1 from systemctl reset-failed ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mgr.smithi002.ahpaml.service
systemctl: stderr Failed to reset failed state of unit ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mgr.smithi002.ahpaml.service: Unit ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mgr.smithi002.ahpaml.service not loaded.
Non-zero exit code 1 from systemctl disable ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mgr.smithi002.ahpaml.service
systemctl: stderr Failed to disable unit: Unit file ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-init@mgr.smithi002.ahpaml.service does not exist.
systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715.target".
systemctl: stderr Removed "/etc/systemd/system/ceph.target.wants/ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715.target".
systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target".
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/tmp/tmpk3h8peah.cephadm.build/app/__main__.py", line 5520, in <module>
File "/tmp/tmpk3h8peah.cephadm.build/app/__main__.py", line 5508, in main
File "/tmp/tmpk3h8peah.cephadm.build/app/__main__.py", line 2635, in _rollback
File "/tmp/tmpk3h8peah.cephadm.build/app/__main__.py", line 443, in _default_image
File "/tmp/tmpk3h8peah.cephadm.build/app/__main__.py", line 2867, in command_bootstrap
File "/tmp/tmpk3h8peah.cephadm.build/app/__main__.py", line 2329, in enable_cephadm_mgr_module
File "/tmp/tmpk3h8peah.cephadm.build/app/__main__.py", line 2796, in cli
File "/tmp/tmpk3h8peah.cephadm.build/app/cephadmlib/container_types.py", line 433, in run
File "/tmp/tmpk3h8peah.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws
RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -e NODE_NAME=smithi002 -v /var/log/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0x139bqo:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy3f55sb8:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: No module named 'mgr_module' (pass --force to force enablement)
[root@smithi002 ~]# vim /var/lib/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/mgr.smithi002.hlzqyo/unit.run
[root@smithi002 ~]# cat /var/lib/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/mgr.smithi002.hlzqyo/unit.run
set -e
/bin/install -d -m0770 -o 167 -g 167 /var/run/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715
# mgr.smithi002.hlzqyo
! /bin/podman rm -f ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mgr.smithi002.hlzqyo 2> /dev/null
! /bin/podman rm -f ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mgr-smithi002-hlzqyo 2> /dev/null
! /bin/podman rm -f --storage ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mgr-smithi002-hlzqyo 2> /dev/null
! /bin/podman rm -f --storage ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mgr.smithi002.hlzqyo 2> /dev/null
/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-mgr --init --name ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mgr-smithi002-hlzqyo --pids-limit=-1 -d --log-driver journald --conmon-pidfile /run/ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715@mgr.smithi002.hlzqyo.service-pid --cidfile /run/ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715@mgr.smithi002.hlzqyo.service-cid --cgroups=split -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -e NODE_NAME=smithi002 -e TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728 -v /var/run/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715:/var/run/ceph:z -v /var/log/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /var/lib/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/mgr.smithi002.hlzqyo:/var/lib/ceph/mgr/ceph-smithi002.hlzqyo:z -v /var/lib/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/mgr.smithi002.hlzqyo/config:/etc/ceph/ceph.conf:z -v /etc/hosts:/etc/hosts:ro quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -n mgr.smithi002.hlzqyo -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false
[root@smithi002 ~]# systemctl restart ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715@mgr.smithi002.hlzqyo.service
[root@smithi002 ~]# sudo /home/ubuntu/cephtest/cephadm shell
Inferring fsid 1c7ef528-c6e5-11ee-95b6-87774f69a715
Inferring config /var/lib/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/mon.smithi002/config
Using ceph image with id '339a52f5ffac' and tag '742e4ab9212fcc3e520b729693dcac293c946fc0' created on 2024-02-02 08:14:02 +0000 UTC
quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36a27b2fc44154ed789d5bc8725dbf47f3eeeccf13f781b04610840bd69ea6eb
[ceph: root@smithi002 /]# ceph -s
cluster:
id: 1c7ef528-c6e5-11ee-95b6-87774f69a715
health: HEALTH_WARN
16 mgr modules have failed dependencies
services:
mon: 1 daemons, quorum smithi002 (age 2m)
mgr: smithi002.hlzqyo(active, since 27s)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@smithi002 /]# ceph health detail
HEALTH_WARN 16 mgr modules have failed dependencies
[WRN] MGR_MODULE_DEPENDENCY: 16 mgr modules have failed dependencies
Module 'balancer' has failed dependency: No module named 'mgr_module'
Module 'cephadm' has failed dependency: No module named 'mgr_module'
Module 'crash' has failed dependency: No module named 'mgr_module'
Module 'dashboard' has failed dependency: No module named 'mgr_module'
Module 'devicehealth' has failed dependency: No module named 'mgr_module'
Module 'iostat' has failed dependency: No module named 'mgr_module'
Module 'nfs' has failed dependency: No module named 'mgr_module'
Module 'orchestrator' has failed dependency: No module named 'mgr_module'
Module 'pg_autoscaler' has failed dependency: No module named 'mgr_module'
Module 'progress' has failed dependency: No module named 'mgr_module'
Module 'prometheus' has failed dependency: No module named 'mgr_module'
Module 'rbd_support' has failed dependency: No module named 'mgr_module'
Module 'restful' has failed dependency: No module named 'mgr_module'
Module 'status' has failed dependency: No module named 'mgr_module'
Module 'telemetry' has failed dependency: No module named 'mgr_module'
Module 'volumes' has failed dependency: No module named 'mgr_module'
[ceph: root@smithi002 /]# exit
vi[root@smithi002 ~]# vim /var/log/ceph/
1c7ef528-c6e5-11ee-95b6-87774f69a715/ cephadm.log
[root@smithi002 ~]# vim /var/log/ceph/
1c7ef528-c6e5-11ee-95b6-87774f69a715/ cephadm.log
[root@smithi002 ~]# vim /var/log/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/ceph-mgr.smithi002.hlzqyo.log
[root@smithi002 ~]# cp /var/log/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/ceph-mgr.smithi002.hlzqyo.log /tmp/
[root@smithi002 ~]# vim /tmp/ceph-mgr.smithi002.hlzqyo.log
[root@smithi002 ~]# sudo /home/ubuntu/cephtest/cephadm shell
Inferring fsid 1c7ef528-c6e5-11ee-95b6-87774f69a715
Inferring config /var/lib/ceph/1c7ef528-c6e5-11ee-95b6-87774f69a715/mon.smithi002/config
Using ceph image with id '339a52f5ffac' and tag '742e4ab9212fcc3e520b729693dcac293c946fc0' created on 2024-02-02 08:14:02 +0000 UTC
quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36a27b2fc44154ed789d5bc8725dbf47f3eeeccf13f781b04610840bd69ea6eb
[ceph: root@smithi002 /]# ceph config get mgr mgr_module_path
/usr/share/ceph/mgr
[ceph: root@smithi002 /]# exit
[root@smithi002 ~]# podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee74b6f9d25d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:742e4ab9212fcc3e520b729693dcac293c946fc0 -n mon.smithi002 ... 41 minutes ago Up 41 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mon-smithi002
084550bab5df quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36a27b2fc44154ed789d5bc8725dbf47f3eeeccf13f781b04610840bd69ea6eb -n client.ceph-ex... 40 minutes ago Up 40 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-ceph-exporter-smithi002
09155cce7e3e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36a27b2fc44154ed789d5bc8725dbf47f3eeeccf13f781b04610840bd69ea6eb -n client.crash.s... 40 minutes ago Up 40 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-crash-smithi002
015d64d6c3b1 quay.io/prometheus/node-exporter:v1.5.0 --no-collector.ti... 40 minutes ago Up 40 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-node-exporter-smithi002
fb5a54387a47 quay.io/ceph/ceph-grafana:9.4.12 /bin/bash 39 minutes ago Up 39 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-grafana-smithi002
025ce38873f7 quay.io/prometheus/prometheus:v2.43.0 --config.file=/et... 39 minutes ago Up 39 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-prometheus-smithi002
5566e6c58712 quay.io/prometheus/alertmanager:v0.25.0 --cluster.listen-... 39 minutes ago Up 39 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-alertmanager-smithi002
4aa984fb1f79 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -n mgr.smithi002.... 35 minutes ago Up 35 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mgr-smithi002-hlzqyo
[root@smithi002 ~]# podman exec -it ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mgr-smithi002-hlzqyo /bin/bash
[root@smithi002 /]# env
CEPH_VERSION=main
LANG=en_US.UTF-8
CEPH_POINT_RELEASE=
CEPH_DEVEL=true
which_declare=declare -f
container=podman
CEPH_REF=wip-yuri6-testing-2024-02-07-1000
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728
PWD=/
HOME=/root
NODE_NAME=smithi002
CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0
TERM=xterm
I_AM_IN_A_CONTAINER=1
SHLVL=1
OSD_FLAVOR=default
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LESSOPEN=||/usr/bin/lesspipe.sh %s
BASH_FUNC_which%%=() { ( alias;
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
_=/usr/bin/env
[root@smithi002 /]# exit
exit
[root@smithi002 ~]# podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee74b6f9d25d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:742e4ab9212fcc3e520b729693dcac293c946fc0 -n mon.smithi002 ... 42 minutes ago Up 42 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mon-smithi002
084550bab5df quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36a27b2fc44154ed789d5bc8725dbf47f3eeeccf13f781b04610840bd69ea6eb -n client.ceph-ex... 41 minutes ago Up 41 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-ceph-exporter-smithi002
09155cce7e3e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36a27b2fc44154ed789d5bc8725dbf47f3eeeccf13f781b04610840bd69ea6eb -n client.crash.s... 41 minutes ago Up 41 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-crash-smithi002
015d64d6c3b1 quay.io/prometheus/node-exporter:v1.5.0 --no-collector.ti... 41 minutes ago Up 41 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-node-exporter-smithi002
fb5a54387a47 quay.io/ceph/ceph-grafana:9.4.12 /bin/bash 41 minutes ago Up 41 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-grafana-smithi002
025ce38873f7 quay.io/prometheus/prometheus:v2.43.0 --config.file=/et... 40 minutes ago Up 40 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-prometheus-smithi002
5566e6c58712 quay.io/prometheus/alertmanager:v0.25.0 --cluster.listen-... 40 minutes ago Up 40 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-alertmanager-smithi002
4aa984fb1f79 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0 -n mgr.smithi002.... 36 minutes ago Up 36 minutes ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715-mgr-smithi002-hlzqyo
[root@smithi002 ~]# ps -aef|grep ceph-mgr
root 46185 46183 0 02:03 ? 00:00:00 /run/podman-init -- /usr/bin/ceph-mgr -n mgr.smithi002.hlzqyo -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false
167 46187 46185 0 02:03 ? 00:00:03 /usr/bin/ceph-mgr -n mgr.smithi002.hlzqyo -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false
root 46613 26390 0 02:40 pts/0 00:00:00 grep --color=auto ceph-mgr
[root@smithi002 ~]# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph eb583889924ee88759f8e8d8bd48edeb78bfd3c0 38c4039b0e98 30 hours ago 1.32 GB
quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph 742e4ab9212fcc3e520b729693dcac293c946fc0 339a52f5ffac 6 days ago 1.32 GB
quay.io/ceph/ceph-grafana 9.4.12 a0ea5e3c46d9 2 months ago 648 MB
quay.io/prometheus/prometheus v2.43.0 a07b618ecd1d 10 months ago 235 MB
quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 13 months ago 66.5 MB
quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 14 months ago 23.9 MB
[root@smithi002 ~]#
[root@smithi002 ~]#
[root@smithi002 ~]# systemctl stop ceph-1c7ef528-c6e5-11ee-95b6-87774f69a715@mgr.smithi002.hlzqyo.service
[root@smithi002 ~]# podman run -it --privileged --rm quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb583889924ee88759f8e8d8bd48edeb78bfd3c0
[root@a63daccde388 /]# env
CEPH_VERSION=main
LANG=en_US.UTF-8
HOSTNAME=a63daccde388
CEPH_POINT_RELEASE=
CEPH_DEVEL=true
which_declare=declare -f
container=podman
CEPH_REF=wip-yuri6-testing-2024-02-07-1000
PWD=/
HOME=/root
TERM=xterm
I_AM_IN_A_CONTAINER=1
SHLVL=1
OSD_FLAVOR=default
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LESSOPEN=||/usr/bin/lesspipe.sh %s
BASH_FUNC_which%%=() { ( alias;
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
_=/usr/bin/env
[root@a63daccde388 /]# find /usr -iname mgr_module.py
/usr/share/ceph/mgr/mgr_module.py
After specifying the mgr modules path in PYTHONPATH, modules were imported successfully :
[root@a63daccde388 /]# export PYTHONPATH=/usr/share/ceph/mgr:$PYTHONPATH
[root@a63daccde388 /]# vi /etc/ceph/ceph.conf
[root@a63daccde388 /]# vi /etc/ceph/mgr.keyring
[root@a63daccde388 /]# ceph-mgr -n mgr.smithi002.hlzqyo -f --setuser ceph --setgroup ceph --keyring /etc/ceph/mgr.keyring
2024-02-09T03:00:15.350+0000 7fe632aeb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member
2024-02-09T03:00:15.445+0000 7fe632aeb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member
2024-02-09T03:00:16.103+0000 7fe632aeb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member
2024-02-09T03:00:16.197+0000 7fe632aeb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member
2024-02-09T03:00:16.340+0000 7fe632aeb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member
2024-02-09T03:00:16.437+0000 7fe632aeb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member
2024-02-09T03:00:16.521+0000 7fe632aeb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
2024-02-09T03:00:16.787+0000 7fe632aeb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member
2024-02-09T03:00:16.899+0000 7fe632aeb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member
2024-02-09T03:00:16.987+0000 7fe632aeb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
2024-02-09T03:00:17.083+0000 7fe632aeb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member
2024-02-09T03:00:17.259+0000 7fe632aeb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member
2024-02-09T03:00:18.258+0000 7fe632aeb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member
2024-02-09T03:00:18.390+0000 7fe632aeb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member
2024-02-09T03:00:18.726+0000 7fe632aeb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member
2024-02-09T03:00:18.817+0000 7fe632aeb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member
2024-02-09T03:00:19.016+0000 7fe632aeb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member
2024-02-09T03:00:19.204+0000 7fe632aeb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member
2024-02-09T03:00:19.491+0000 7fe632aeb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member
2024-02-09T03:00:19.751+0000 7fe632aeb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member
2024-02-09T03:00:19.991+0000 7fe632aeb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member
2024-02-09T03:00:20.812+0000 7fe632aeb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member
2024-02-09T03:00:21.387+0000 7fe632aeb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member
2024-02-09T03:00:21.486+0000 7fe632aeb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member
2024-02-09T03:00:21.706+0000 7fe632aeb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member
[09/Feb/2024:03:00:21] ENGINE Bus STARTING
CherryPy Checker:
The Application mounted at '' has an empty config.
[09/Feb/2024:03:00:21] ENGINE Serving on http://:::9283
[09/Feb/2024:03:00:21] ENGINE Bus STARTED
[09/Feb/2024:03:00:43] ENGINE Bus STOPPING
[09/Feb/2024:03:00:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down
[09/Feb/2024:03:00:43] ENGINE Bus STOPPED
[09/Feb/2024:03:00:43] ENGINE Bus STARTING
[09/Feb/2024:03:00:43] ENGINE Serving on http://:::9283
[09/Feb/2024:03:00:43] ENGINE Bus STARTED
[09/Feb/2024:03:00:43] ENGINE Bus STOPPING
[09/Feb/2024:03:00:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down
[09/Feb/2024:03:00:43] ENGINE Bus STOPPED
[09/Feb/2024:03:00:43] ENGINE Bus STARTING
[09/Feb/2024:03:00:43] ENGINE Serving on http://:::9283
[09/Feb/2024:03:00:43] ENGINE Bus STARTED
Updated by Radoslaw Zarzynski 3 months ago
note from a scrub: it seems the problem comes from an unmerged PR; if so, we should close this ticket.
Updated by Laura Flores about 1 month ago
/a/yuriw-2024-04-01_20:57:46-rados-wip-yuri3-testing-2024-04-01-0837-squid-distro-default-smithi/7634416
Updated by Laura Flores about 1 month ago
Possible fix: https://github.com/ceph/ceph/pull/56617
Although this reverted a commit that was only merged a few days ago. So the root cause for the original report is likely a different root cause.
Updated by Laura Flores about 1 month ago
- Project changed from RADOS to Orchestrator
Updated by Laura Flores 25 days ago
/a/lflores-2024-04-01_18:07:25-rados-wip-yuri8-testing-2024-03-25-1419-distro-default-smithi/7634080
Actions