Feature #19885
closed`ceph tell mgr help` failed on Kraken
100%
Description
My ceph version is Kraken V11.2.0, and I follow this document1, but there is not `ceph tell mgr help` command. any wrong here?
[root@TENCENT64 /data/seal/deploy]# ceph tell mgr help 2017-05-09 15:12:58.976859 7f717ca46700 -1 WARNING: the following dangerous and experimental features are enabled: rocksdb 2017-05-09 15:12:59.000314 7f717ca46700 -1 WARNING: the following dangerous and experimental features are enabled: rocksdb no valid command found; 10 closest matches: Error EINVAL: invalid command [root@TENCENT64 /data/seal/deploy]# ceph -s 2017-05-09 15:13:08.333228 7f49f820f700 -1 WARNING: the following dangerous and experimental features are enabled: rocksdb 2017-05-09 15:13:08.353465 7f49f820f700 -1 WARNING: the following dangerous and experimental features are enabled: rocksdb cluster e5e1d39d-0e5c-41d5-be01-e55e1578725b health HEALTH_OK monmap e2: 3 mons at {node41=10.53.178.10:6789/0,node42=10.53.184.208:6789/0,node43=10.53.178.78:6789/0} election epoch 6, quorum 0,1,2 node41,node43,node42 mgr active: foo osdmap e1960: 443 osds: 443 up, 443 in flags sortbitwise,require_jewel_osds,require_kraken_osds pgmap v73311: 120 pgs, 8 pools, 2937 bytes data, 205 objects 77948 MB used, 1606 TB / 1606 TB avail 120 active+clean
Updated by Chang Liu almost 7 years ago
sorry, i missed document link: http://docs.ceph.com/docs/kraken/mgr/administrator/
Updated by Chang Liu almost 7 years ago
Well, I try to read relevant code. here is different between `ceph tell` and `daemon(admin_socket)`, which is just my understand, and may be mistake.
- `ceph tell` is a MMonCommand. for example, if we use `ceph tell osd.12 bench`, MON will send osd.12 a MMonCommand to OSD.12. see `void OSD::handle_command(MMonCommand *m)`
- `daemon(adin_socket)` is based on admin_socket(unix_socket). We must use this feature in corresponding machine.
Now, what should we do ?
1) update documentation?
2) add new MMonCommand to MGR?
Updated by Kefu Chai almost 7 years ago
MgrPyModules reports all supported commands via "get_commands", we could report the supported commands by handling MMonCommand.
do you want to take over this ticket? or i can take care of it.
Updated by Chang Liu almost 7 years ago
- Tracker changed from Bug to Feature
- Assignee changed from Kefu Chai to Chang Liu
well, I found the command "get_command_descriptions" in source that used in OSD/MON module. I will take over this ticket. thanks Kefu.
BTW, That we get via "get_command_descriptions" is not easy to understand, there are a lot of self-defined concepts in this1 file. I also want the output of the command to better understand.
https://github.com/ceph/ceph/blob/master/src/mon/MonCommands.h#L79
Updated by Tim Serong almost 7 years ago
This problem doesn't occur in master. I believe it was fixed by https://github.com/ceph/ceph/commit/9988a564d8
Updated by Chang Liu almost 7 years ago
Tim Serong wrote:
This problem doesn't occur in master. I believe it was fixed by https://github.com/ceph/ceph/commit/9988a564d8
Thanks, Tim. I tried it with master branch. it seems that it was not fixed, but it's ok, I will fix it as soon as possible.
➜ build git:(wip-19382) ✗ ./bin/ceph tell osd.1 get_command_descriptions *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-05-15 15:17:33.809898 7f054c4fc700 -1 WARNING: all dangerous and experimental features are enabled. 2017-05-15 15:17:33.818258 7f054c4fc700 -1 WARNING: all dangerous and experimental features are enabled. {"cmd000":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"query","type":"CephChoices"}],"help":"show details of a specific pg","module":"osd","perm":"r","avail":"cli","flags":0},"cmd001":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"mark_unfound_lost","type":"CephChoices"},{"name":"mulcmd","strings":"revert|delete","type":"CephChoices"}],"help":"mark all unfound objects in this pg as lost, either removing or reverting to a prior version if one is available","module":"osd","perm":"rw","avail":"cli","flags":0},"cmd002":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"list_missing","type":"CephChoices"},{"name":"offset","req":"false","type":"CephString"}],"help":"list missing objects on this pg, perhaps starting at an offset given in JSON","module":"osd","perm":"r","avail":"cli","flags":0},"cmd003":{"sig":["query"],"help":"show details of a specific pg","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd004":{"sig":["mark_unfound_lost",{"name":"mulcmd","strings":"revert|delete","type":"CephChoices"}],"help":"mark all unfound objects in this pg as lost, either removing or reverting to a prior version if one is available","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd005":{"sig":["list_missing",{"name":"offset","req":"false","type":"CephString"}],"help":"list missing objects on this pg, perhaps starting at an offset given in JSON","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd006":{"sig":["version"],"help":"report version of OSD","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd007":{"sig":["get_command_descriptions"],"help":"list commands descriptions","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd008":{"sig":["injectargs",{"n":"N","name":"injected_args","type":"CephString"}],"help":"inject configuration arguments into running OSD","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd009":{"sig":["cluster_log",{"debug":"\u0001","info":"\u0001","name":"level","strings":"error","type":"CephChoices","warning":"\u0001"},{"n":"N","name":"message","type":"CephString"}],"help":"log a message to the cluster log","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd010":{"sig":["bench",{"name":"count","req":"false","type":"CephInt"},{"name":"size","req":"false","type":"CephInt"},{"name":"object_size","req":"false","type":"CephInt"},{"name":"object_num","req":"false","type":"CephInt"}],"help":"OSD benchmark: write <count> <size>-byte objects, (default 1G size 4MB). Results in log.","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd011":{"sig":["flush_pg_stats"],"help":"flush pg stats","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd012":{"sig":["heap",{"name":"heapcmd","strings":"dump|start_profiler|stop_profiler|release|stats","type":"CephChoices"}],"help":"show heap usage info (available only if compiled with tcmalloc)","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd013":{"sig":["debug","dump_missing",{"name":"filename","type":"CephFilepath"}],"help":"dump missing objects to a named file","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd014":{"sig":["debug","kick_recovery_wq",{"name":"delay","range":"0","type":"CephInt"}],"help":"set osd_recovery_delay_start to <val>","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd015":{"sig":["cpu_profiler",{"name":"arg","strings":"status|flush","type":"CephChoices"}],"help":"run cpu profiling on daemon","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd016":{"sig":["dump_pg_recovery_stats"],"help":"dump pg recovery statistics","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd017":{"sig":["reset_pg_recovery_stats"],"help":"reset pg recovery statistics","module":"osd","perm":"rw","avail":"cli,rest","flags":0}} ➜ build git:(wip-19382) ✗ ./bin/ceph tell mgr get_command_descriptions *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-05-15 15:17:40.749573 7ff261fa3700 -1 WARNING: all dangerous and experimental features are enabled. 2017-05-15 15:17:40.755452 7ff261fa3700 -1 WARNING: all dangerous and experimental features are enabled. no valid command found; 10 closest matches: fs status {<fs>} osd perf {<bucket>} Error EINVAL: invalid command ➜ build git:(wip-19382) ✗ ceph -s zsh: command not found: ceph ➜ build git:(wip-19382) ✗ ./bin/ceph -s *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-05-15 15:17:48.836006 7faa1ff61700 -1 WARNING: all dangerous and experimental features are enabled. 2017-05-15 15:17:48.838762 7faa1ff61700 -1 WARNING: all dangerous and experimental features are enabled. cluster 1d312f4b-b356-42d7-89d7-9493dd133bc7 health HEALTH_OK monmap e2: 3 mons at {a=127.0.0.1:40000/0,b=127.0.0.1:40001/0,c=127.0.0.1:40002/0} election epoch 10, quorum 0,1,2 a,b,c fsmap e7: 1/1/1 up {0=b=up:active}, 2 up:standby mgr active: x osdmap e19: 3 osds: 3 up, 3 in pgmap v54: 56 pgs, 7 pools, 5268 bytes data, 203 objects 158 GB used, 254 GB / 413 GB avail 56 active+clean ➜ build git:(wip-19382) ✗ ./bin/ceph tell mgr.x get_command_descriptions *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-05-15 15:18:00.659132 7f9f982e9700 -1 WARNING: all dangerous and experimental features are enabled. 2017-05-15 15:18:00.695208 7f9f982e9700 -1 WARNING: all dangerous and experimental features are enabled. no valid command found; 10 closest matches: fs status {<fs>} osd perf {<bucket>} Error EINVAL: invalid command ➜ build git:(wip-19382) ✗ ./bin/ceph tell mgr.x osd perf *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-05-15 15:22:06.333824 7fa1df958700 -1 WARNING: all dangerous and experimental features are enabled. 2017-05-15 15:22:06.352858 7fa1df958700 -1 WARNING: all dangerous and experimental features are enabled. osd commit_latency(ms) apply_latency(ms) 2 4 25 1 4 22 0 5 20
Updated by Tim Serong almost 7 years ago
Sorry, my mistake. I was confusing this for a problem we had at one point where invoking commands defined in the python modules didn't work (e.g. "bin/ceph tell mgr.x fs status" would always return EINVAL, even when the fsstatus module was loaded).
Updated by Chang Liu almost 7 years ago
Tim Serong wrote:
Sorry, my mistake. I was confusing this for a problem we had at one point where invoking commands defined in the python modules didn't work (e.g. "bin/ceph tell mgr.x fs status" would always return EINVAL, even when the fsstatus module was loaded).
Hi, Tim
I tried with your command, and it works. maybe you should check mgr log to find the problem.
./bin/ceph tell mgr.x fs status *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-05-15 19:26:02.497881 7fcbbc5db700 -1 WARNING: all dangerous and experimental features are enabled. 2017-05-15 19:26:02.541064 7fcbbc5db700 -1 WARNING: all dangerous and experimental features are enabled. cephfs_a - 0 clients ======== +------+--------+-----+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+-----+---------------+-------+-------+ | 0 | active | a | Reqs: 0 /s | 10 | 2 | +------+--------+-----+---------------+-------+-------+ +-------------------+----------+-------+-------+ | Pool | type | used | avail | +-------------------+----------+-------+-------+ | cephfs_metadata_a | metadata | 2246 | 87.6G | | cephfs_data_a | data | 0 | 87.6G | +-------------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ | c | | b | +-------------+ MDS version: ceph version 12.0.1-2360-g38b6dcd (38b6dcd38e11c42124637331b59e8e3a6c747337)
Updated by Tim Serong almost 7 years ago
Sorry, I'm not explaining myself clearly. What I meant was;
- At some time in the past, commands in python modules (e.g.: "bin/ceph tell mgr.x fs status") didn't work; I mistakenly thought this problem was the issue you were referring to in this bug.
- The problem with commands in python modules not working has since been fixed in master and works fine now.
You should probably just ignore all my comments on this bug :-)
Updated by Chang Liu almost 7 years ago
Tim Serong wrote:
Sorry, I'm not explaining myself clearly. What I meant was;
- At some time in the past, commands in python modules (e.g.: "bin/ceph tell mgr.x fs status") didn't work; I mistakenly thought this problem was the issue you were referring to in this bug.
- The problem with commands in python modules not working has since been fixed in master and works fine now.
You should probably just ignore all my comments on this bug :-)
thank you for your comments. :-)
Updated by Chang Liu almost 7 years ago
- Status changed from New to Resolved
- % Done changed from 0 to 100