Project

General

Profile

Actions
Copy link

Bug #45596

closed

qa/tasks/cephadm: No cephadm module detected

Added by Varsha Rao almost 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Varsha Rao
Category:
teuthology
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
35198
Crash signature (v1):
Crash signature (v2):

Description

Something weird is happening here.
First mgr fails and cephadm is disabled.

2020-05-08T07:53:17.716 INFO:ceph.mgr.a.smithi003.stdout:May 08 07:53:17 smithi003 podman[39947]: 2020-05-08 07:53:17.404452424 +0000 UTC m=+0.254745655 container died 164e714f7b5017223ced2fe56efb54e8b0076d495c028be9bab85bda91f22c97 (image=quay.io/ceph-ci/ceph:5d96f0c9612029b065cea7c34cd161b174878f8c, name=ceph-8f4e11de-9100-11ea-a068-001a4aab830c-mgr.a)
2020-05-08T07:53:17.717 INFO:ceph.mgr.a.smithi003.stdout:May 08 07:53:17 smithi003 podman[39947]: 2020-05-08 07:53:17.467673383 +0000 UTC m=+0.317966394 container stop 164e714f7b5017223ced2fe56efb54e8b0076d495c028be9bab85bda91f22c97 (image=quay.io/ceph-ci/ceph:5d96f0c9612029b065cea7c34cd161b174878f8c, name=ceph-8f4e11de-9100-11ea-a068-001a4aab830c-mgr.a)
2020-05-08T07:53:17.717 INFO:ceph.mgr.a.smithi003.stdout:May 08 07:53:17 smithi003 podman[39947]: 164e714f7b5017223ced2fe56efb54e8b0076d495c028be9bab85bda91f22c97
2020-05-08T07:53:17.997 INFO:teuthology.orchestra.run.smithi003:> true
2020-05-08T07:53:18.013 INFO:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8f4e11de-9100-11ea-a068-001a4aab830c@mgr.a.service'
2020-05-08T07:53:18.083 INFO:ceph.mgr.a.smithi003.stdout:May 08 07:53:17 smithi003 podman[26049]: 2020-05-08 07:53:17.983398302 +0000 UTC m=+134.198961367 container remove 164e714f7b5017223ced2fe56efb54e8b0076d495c028be9bab85bda91f22c97 (image=quay.io/ceph-ci/ceph:5d96f0c9612029b065cea7c34cd161b174878f8c, name=ceph-8f4e11de-9100-11ea-a068-001a4aab830c-mgr.a)
2020-05-08T07:53:18.083 INFO:ceph.mgr.a.smithi003.stdout:May 08 07:53:17 smithi003 bash[26046]: debug
2020-05-08T07:53:18.083 INFO:ceph.mgr.a.smithi003.stdout:May 08 07:53:17 smithi003 systemd[1]: Stopped Ceph mgr.a for 8f4e11de-9100-11ea-a068-001a4aab830c.
2020-05-08T07:53:18.203 DEBUG:teuthology.orchestra.run:got remote process result: None
2020-05-08T07:53:18.204 INFO:tasks.cephadm.mgr.a:Stopped mgr.a
2020-05-08T07:53:18.205 INFO:teuthology.orchestra.run.smithi003:> true
2020-05-08T07:53:18.251 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mgr fail a
2020-05-08T07:53:18.713 INFO:teuthology.orchestra.run.smithi003:> true

Then mgr is restarted and fails as cephadm is not detected.

2020-05-08T07:53:19.453 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mgr module disable cephadm
2020-05-08T07:53:19.964 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:19 smithi003 bash[25807]: audit 2020-05-08T07:53:18.700715+0000 mon.a (mon.0) 256 : audit [INF] from='client.? 172.21.15.3:0/3482683540' entity='client.admin' cmd='[{"prefix": "mgr fail", "who": "a"}]': finished
2020-05-08T07:53:19.965 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:19 smithi003 bash[25807]: cluster 2020-05-08T07:53:18.700760+0000 mon.a (mon.0) 257 : cluster [DBG] mgrmap e14: no daemons active (since 0.0557753s)
2020-05-08T07:53:19.965 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:19 smithi003 bash[25807]: audit 2020-05-08T07:53:19.085697+0000 mon.a (mon.0) 258 : audit [DBG] from='client.? 172.21.15.3:0/2781800809' entity='client.admin' cmd=[{"prefix": "mgr module ls"}]: dispatch
2020-05-08T07:53:20.727 INFO:teuthology.orchestra.run.smithi003:> true
2020-05-08T07:53:20.744 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mgr module disable iostat
2020-05-08T07:53:20.964 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:20 smithi003 bash[25807]: audit 2020-05-08T07:53:19.810781+0000 mon.a (mon.0) 259 : audit [INF] from='client.? 172.21.15.3:0/3588794767' entity='client.admin' cmd=[{"prefix": "mgr module disable", "module": "cephadm"}]: dispatch
2020-05-08T07:53:21.736 INFO:teuthology.orchestra.run.smithi003:> true
2020-05-08T07:53:21.753 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mgr module disable dashboard
2020-05-08T07:53:21.964 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:21 smithi003 bash[25807]: audit 2020-05-08T07:53:20.710746+0000 mon.a (mon.0) 260 : audit [INF] from='client.? 172.21.15.3:0/3588794767' entity='client.admin' cmd='[{"prefix": "mgr module disable", "module": "cephadm"}]': finished
2020-05-08T07:53:21.964 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:21 smithi003 bash[25807]: cluster 2020-05-08T07:53:20.710801+0000 mon.a (mon.0) 261 : cluster [DBG] mgrmap e15: no daemons active (since 2s)
2020-05-08T07:53:21.965 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:21 smithi003 bash[25807]: audit 2020-05-08T07:53:21.110849+0000 mon.a (mon.0) 262 : audit [INF] from='client.? 172.21.15.3:0/1093881759' entity='client.admin' cmd=[{"prefix": "mgr module disable", "module": "iostat"}]: dispatch
2020-05-08T07:53:22.731 INFO:tasks.cephadm.mgr.a:Restarting mgr.a (starting--it wasn't running)...


2020-05-08T07:53:35.900 INFO:ceph.mgr.a.smithi003.stdout:May 08 07:53:35 smithi003 bash[40619]: debug 2020-05-08T07:53:35.522+0000 7f1e08ae5700 -1 no module 'cephadm'
2020-05-08T07:53:35.901 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:35 smithi003 bash[25807]: audit 2020-05-08T07:53:33.861586+0000 mgr.a (mgr.14247) 5 : audit [DBG] from='client.14262 -' entity='client.admin' cmd=[{"prefix": "nfs clust
er create", "type": "cephfs", "clusterid": "test", "target": ["mon-mgr", ""]}]: dispatch
2020-05-08T07:53:35.901 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:35 smithi003 bash[25807]: audit 2020-05-08T07:53:34.470014+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14247 172.21.15.3:0/57223522' entity='mgr.a' cmd='[{"prefix": 
"osd pool create", "pool": "nfs-ganesha"}]': finished
2020-05-08T07:53:35.901 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:35 smithi003 bash[25807]: cluster 2020-05-08T07:53:34.470056+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e24: 3 total, 3 up, 3 in
2020-05-08T07:53:35.901 INFO:ceph.mon.a.smithi003.stdout:May 08 07:53:35 smithi003 bash[25807]: audit 2020-05-08T07:53:34.473610+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14247 172.21.15.3:0/57223522' entity='mgr.a' cmd=[{"prefix": " 
osd pool application enable", "pool": "nfs-ganesha", "app": "nfs"}]: dispatch
2020-05-08T07:53:35.902 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENOENT: Module not found

http://qa-proxy.ceph.com/teuthology/varsha-2020-05-08_07:35:51-rados-wip-varsha-testing-distro-basic-smithi/5034105/

Actions
Copy link

Also available in: Atom PDF