Project

General

Profile

Actions
Copy link

Bug #40782

closed

"test_telemetry (tasks.mgr.test_module_selftest.TestModuleSelftest) ... ERROR" in rados/mimic

Added by Yuri Weinstein almost 5 years ago. Updated almost 4 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
Neha Ojha
Category:
telemetry module
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
nautilus, mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/yuriw-2019-07-12_15:10:31-rados-wip-yuri2-testing-2019-07-11-1516-mimic-distro-basic-smithi/
Jobs: 4114604, 4114605
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2019-07-12_15:10:31-rados-wip-yuri2-testing-2019-07-11-1516-mimic-distro-basic-smithi/4114604/teuthology.log

2019-07-12T15:29:05.850 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.853 7f88d93d2700 20 mgrc start_command cmd: [{"prefix": "telemetry self-test", "target": ["mgr", ""]}]
2019-07-12T15:29:05.850 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.853 7f88d93d2700  1 -- 172.21.15.33:0/2231163965 --> 172.21.15.16:6809/13391 -- command(tid 0: {"prefix": "telemetry self-test", "target": ["mgr", ""]}) v1 -- 0x7f88b00028f0 con 0
2019-07-12T15:29:05.855 INFO:tasks.ceph.mgr.y.smithi016.stderr:2019-07-12 15:29:05.857 7f9fee6cd700 -1 no module 'crash'
2019-07-12T15:29:05.856 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.857 7f88d0a64700  1 -- 172.21.15.33:0/2231163965 <== mgr.5918 172.21.15.16:6809/13391 1 ==== command_reply(tid 0: -22 Traceback (most recent call last):
2019-07-12T15:29:05.857 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 328, in handle_command
2019-07-12T15:29:05.857 INFO:teuthology.orchestra.run.smithi033.stderr:    self.self_test()
2019-07-12T15:29:05.857 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 384, in self_test
2019-07-12T15:29:05.857 INFO:teuthology.orchestra.run.smithi033.stderr:    self.compile_report()
2019-07-12T15:29:05.857 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 291, in compile_report
2019-07-12T15:29:05.857 INFO:teuthology.orchestra.run.smithi033.stderr:    report['crashes'] = self.gather_crashinfo()
2019-07-12T15:29:05.858 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 214, in gather_crashinfo
2019-07-12T15:29:05.858 INFO:teuthology.orchestra.run.smithi033.stderr:    errno, crashids, err = self.remote('crash', 'do_ls', '', '')
2019-07-12T15:29:05.858 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/mgr_module.py", line 845, in remote
2019-07-12T15:29:05.858 INFO:teuthology.orchestra.run.smithi033.stderr:    args, kwargs)
2019-07-12T15:29:05.858 INFO:teuthology.orchestra.run.smithi033.stderr:ImportError: Module not found
2019-07-12T15:29:05.858 INFO:teuthology.orchestra.run.smithi033.stderr:) v1 ==== 614+0+0 (2804853677 0 0) 0x7f88d4115530 con 0x7f88b40072f0
2019-07-12T15:29:05.858 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.857 7f88d0a64700 20 mgrc handle_command_reply command_reply(tid 0: -22 Traceback (most recent call last):
2019-07-12T15:29:05.859 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 328, in handle_command
2019-07-12T15:29:05.859 INFO:teuthology.orchestra.run.smithi033.stderr:    self.self_test()
2019-07-12T15:29:05.859 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 384, in self_test
2019-07-12T15:29:05.859 INFO:teuthology.orchestra.run.smithi033.stderr:    self.compile_report()
2019-07-12T15:29:05.859 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 291, in compile_report
2019-07-12T15:29:05.859 INFO:teuthology.orchestra.run.smithi033.stderr:    report['crashes'] = self.gather_crashinfo()
2019-07-12T15:29:05.859 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 214, in gather_crashinfo
2019-07-12T15:29:05.860 INFO:teuthology.orchestra.run.smithi033.stderr:    errno, crashids, err = self.remote('crash', 'do_ls', '', '')
2019-07-12T15:29:05.860 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/mgr_module.py", line 845, in remote
2019-07-12T15:29:05.860 INFO:teuthology.orchestra.run.smithi033.stderr:    args, kwargs)
2019-07-12T15:29:05.860 INFO:teuthology.orchestra.run.smithi033.stderr:ImportError: Module not found
2019-07-12T15:29:05.860 INFO:teuthology.orchestra.run.smithi033.stderr:) v1
2019-07-12T15:29:05.861 INFO:teuthology.orchestra.run.smithi033.stderr:Error EINVAL: Traceback (most recent call last):
2019-07-12T15:29:05.862 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 328, in handle_command
2019-07-12T15:29:05.862 INFO:teuthology.orchestra.run.smithi033.stderr:    self.self_test()
2019-07-12T15:29:05.862 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 384, in self_test
2019-07-12T15:29:05.862 INFO:teuthology.orchestra.run.smithi033.stderr:    self.compile_report()
2019-07-12T15:29:05.862 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 291, in compile_report
2019-07-12T15:29:05.862 INFO:teuthology.orchestra.run.smithi033.stderr:    report['crashes'] = self.gather_crashinfo()
2019-07-12T15:29:05.863 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/telemetry/module.py", line 214, in gather_crashinfo
2019-07-12T15:29:05.863 INFO:teuthology.orchestra.run.smithi033.stderr:    errno, crashids, err = self.remote('crash', 'do_ls', '', '')
2019-07-12T15:29:05.863 INFO:teuthology.orchestra.run.smithi033.stderr:  File "/usr/lib/ceph/mgr/mgr_module.py", line 845, in remote
2019-07-12T15:29:05.863 INFO:teuthology.orchestra.run.smithi033.stderr:    args, kwargs)
2019-07-12T15:29:05.863 INFO:teuthology.orchestra.run.smithi033.stderr:ImportError: Module not found
2019-07-12T15:29:05.863 INFO:teuthology.orchestra.run.smithi033.stderr:
2019-07-12T15:29:05.863 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.861 7f88d93d2700 10 mgrc shutdown
2019-07-12T15:29:05.864 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.861 7f88d93d2700  1 -- 172.21.15.33:0/2231163965 >> 172.21.15.16:6809/13391 conn(0x7f88b40072f0 :-1 s=STATE_OPEN pgs=14 cs=1 l=1).mark_down
2019-07-12T15:29:05.864 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.861 7f88d93d2700  1 -- 172.21.15.33:0/2231163965 >> 172.21.15.33:6789/0 conn(0x7f88d40ad2e0 :-1 s=STATE_OPEN pgs=624 cs=1 l=1).mark_down
2019-07-12T15:29:05.864 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.861 7f88d93d2700  1 -- 172.21.15.33:0/2231163965 shutdown_connections
2019-07-12T15:29:05.864 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.861 7f88d93d2700  1 -- 172.21.15.33:0/2231163965 shutdown_connections
2019-07-12T15:29:05.864 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.861 7f88d93d2700  1 -- 172.21.15.33:0/2231163965 wait complete.
2019-07-12T15:29:05.864 INFO:teuthology.orchestra.run.smithi033.stderr:2019-07-12 15:29:05.861 7f88d93d2700  1 -- 172.21.15.33:0/2231163965 >> 172.21.15.33:0/2231163965 conn(0x7f88d40a1c80 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down
2019-07-12T15:29:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 22
2019-07-12T15:29:05.883 INFO:tasks.cephfs_test_runner:test_telemetry (tasks.mgr.test_module_selftest.TestModuleSelftest) ... ERROR
2019-07-12T15:29:05.884 INFO:teuthology.orchestra.run.smithi033:Running:
2019-07-12T15:29:05.884 INFO:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.mgr.test_module_selftest.TestModuleSelftest.test_telemetry'

Related issues 2 (1 open — 1 closed)

Related to mgr - Bug #42216: Test failure: setUpClass (tasks.mgr.test_module_selftest.TestModuleSelftest)New10/07/2019

Actions
Has duplicate mgr - Bug #41168: mimic: mgr/telemetry: ImportError when running ceph mgr showCan't reproduceKefu Chai08/08/2019

Actions
Actions
Copy link
 #1

Updated by Greg Farnum almost 5 years ago

  • Project changed from Ceph to mgr
Actions
Copy link
 #2

Updated by Neha Ojha almost 5 years ago

  • Status changed from New to 12
  • Priority changed from Normal to High

/a/yuriw-2019-07-15_19:24:27-rados-wip-yuri4-testing-2019-07-15-1517-mimic-distro-basic-smithi/4121490/

Actions
Copy link
 #3

Updated by Blake Klynsma over 4 years ago

I encountered this same thing on our Ceph cluster, ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) on Ubuntu 18.04.3 LTS, when I enabled telemetry and performed a self test using the `ceph telemetry self-test` command. When checking our ceph-mgr.ceph-admin.log I found the following line at the same time.

2019-08-12 06:30:54.877 7fe378119700  1 mgr.server reply handle_command (95) Operation not supported Module 'crash' is not enabled (required by command 'crash post'): use `ceph mgr module enable crash` to enable it

So run the `ceph mgr module enable crash` command then the `ceph telemetry self-test` again and found that everything passed without error. I can see in the output above that this exact module is missing. `7f9fee6cd700 -1 no module 'crash'`.

I understand that this error does need to be handled appropriately. But it would also make sense to update the documentation at the same time that end-users need to enable the crash module as well when enabling telemetry.

Actions
Copy link
 #4

Updated by Sebastian Wagner over 4 years ago

  • Has duplicate Bug #41168: mimic: mgr/telemetry: ImportError when running ceph mgr show added
Actions
Copy link
 #5

Updated by Sebastian Wagner over 4 years ago

  • Backport set to nautilus, mimic
Actions
Copy link
 #6

Updated by Sebastian Wagner over 4 years ago

  • Category set to telemetry module
Actions
Copy link
 #7

Updated by Neha Ojha over 4 years ago

  • Assignee set to Neha Ojha
Actions
Copy link
 #8

Updated by David Zafman over 4 years ago

  • Related to Bug #42216: Test failure: setUpClass (tasks.mgr.test_module_selftest.TestModuleSelftest) added
Actions
Copy link
 #9

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions
Copy link
 #10

Updated by Neha Ojha over 4 years ago

  • Priority changed from High to Normal

Not seen this in a while.

Actions
Copy link
 #11

Updated by Neha Ojha almost 4 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link

Also available in: Atom PDF