Project

General

Profile

Feature #19885

`ceph tell mgr help` failed on Kraken

Added by Chang Liu 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
05/09/2017
Due date:
% Done:

100%

Source:
Tags:
Backport:
Reviewed:
User Impact:
Affected Versions:
Release:
kraken
Needs Doc:
No

Description

My ceph version is Kraken V11.2.0, and I follow this document1, but there is not `ceph tell mgr help` command. any wrong here?

[root@TENCENT64 /data/seal/deploy]# ceph tell mgr help
2017-05-09 15:12:58.976859 7f717ca46700 -1 WARNING: the following dangerous and experimental features are enabled: rocksdb
2017-05-09 15:12:59.000314 7f717ca46700 -1 WARNING: the following dangerous and experimental features are enabled: rocksdb
no valid command found; 10 closest matches:
Error EINVAL: invalid command
[root@TENCENT64 /data/seal/deploy]# ceph -s
2017-05-09 15:13:08.333228 7f49f820f700 -1 WARNING: the following dangerous and experimental features are enabled: rocksdb
2017-05-09 15:13:08.353465 7f49f820f700 -1 WARNING: the following dangerous and experimental features are enabled: rocksdb
    cluster e5e1d39d-0e5c-41d5-be01-e55e1578725b
     health HEALTH_OK
     monmap e2: 3 mons at {node41=10.53.178.10:6789/0,node42=10.53.184.208:6789/0,node43=10.53.178.78:6789/0}
            election epoch 6, quorum 0,1,2 node41,node43,node42
        mgr active: foo 
     osdmap e1960: 443 osds: 443 up, 443 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v73311: 120 pgs, 8 pools, 2937 bytes data, 205 objects
            77948 MB used, 1606 TB / 1606 TB avail
                 120 active+clean

History

#1 Updated by Chang Liu 3 months ago

#2 Updated by Chang Liu 3 months ago

  • Assignee set to Kefu Chai

@Kefu, please take a look

#3 Updated by Chang Liu 3 months ago

Well, I try to read relevant code. here is different between `ceph tell` and `daemon(admin_socket)`, which is just my understand, and may be mistake.

  1. `ceph tell` is a MMonCommand. for example, if we use `ceph tell osd.12 bench`, MON will send osd.12 a MMonCommand to OSD.12. see `void OSD::handle_command(MMonCommand *m)`
  2. `daemon(adin_socket)` is based on admin_socket(unix_socket). We must use this feature in corresponding machine.

Now, what should we do ?

1) update documentation?
2) add new MMonCommand to MGR?

#4 Updated by Kefu Chai 3 months ago

MgrPyModules reports all supported commands via "get_commands", we could report the supported commands by handling MMonCommand.

do you want to take over this ticket? or i can take care of it.

#5 Updated by Chang Liu 2 months ago

  • Tracker changed from Bug to Feature
  • Assignee changed from Kefu Chai to Chang Liu

well, I found the command "get_command_descriptions" in source that used in OSD/MON module. I will take over this ticket. thanks Kefu.

BTW, That we get via "get_command_descriptions" is not easy to understand, there are a lot of self-defined concepts in this1 file. I also want the output of the command to better understand.

https://github.com/ceph/ceph/blob/master/src/mon/MonCommands.h#L79

#6 Updated by Tim Serong 2 months ago

This problem doesn't occur in master. I believe it was fixed by https://github.com/ceph/ceph/commit/9988a564d8

#7 Updated by Chang Liu 2 months ago

Tim Serong wrote:

This problem doesn't occur in master. I believe it was fixed by https://github.com/ceph/ceph/commit/9988a564d8

Thanks, Tim. I tried it with master branch. it seems that it was not fixed, but it's ok, I will fix it as soon as possible.

➜  build git:(wip-19382) ✗ ./bin/ceph tell osd.1 get_command_descriptions
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:17:33.809898 7f054c4fc700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:17:33.818258 7f054c4fc700 -1 WARNING: all dangerous and experimental features are enabled.
{"cmd000":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"query","type":"CephChoices"}],"help":"show details of a specific pg","module":"osd","perm":"r","avail":"cli","flags":0},"cmd001":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"mark_unfound_lost","type":"CephChoices"},{"name":"mulcmd","strings":"revert|delete","type":"CephChoices"}],"help":"mark all unfound objects in this pg as lost, either removing or reverting to a prior version if one is available","module":"osd","perm":"rw","avail":"cli","flags":0},"cmd002":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"list_missing","type":"CephChoices"},{"name":"offset","req":"false","type":"CephString"}],"help":"list missing objects on this pg, perhaps starting at an offset given in JSON","module":"osd","perm":"r","avail":"cli","flags":0},"cmd003":{"sig":["query"],"help":"show details of a specific pg","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd004":{"sig":["mark_unfound_lost",{"name":"mulcmd","strings":"revert|delete","type":"CephChoices"}],"help":"mark all unfound objects in this pg as lost, either removing or reverting to a prior version if one is available","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd005":{"sig":["list_missing",{"name":"offset","req":"false","type":"CephString"}],"help":"list missing objects on this pg, perhaps starting at an offset given in JSON","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd006":{"sig":["version"],"help":"report version of OSD","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd007":{"sig":["get_command_descriptions"],"help":"list commands descriptions","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd008":{"sig":["injectargs",{"n":"N","name":"injected_args","type":"CephString"}],"help":"inject configuration arguments into running OSD","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd009":{"sig":["cluster_log",{"debug":"\u0001","info":"\u0001","name":"level","strings":"error","type":"CephChoices","warning":"\u0001"},{"n":"N","name":"message","type":"CephString"}],"help":"log a message to the cluster log","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd010":{"sig":["bench",{"name":"count","req":"false","type":"CephInt"},{"name":"size","req":"false","type":"CephInt"},{"name":"object_size","req":"false","type":"CephInt"},{"name":"object_num","req":"false","type":"CephInt"}],"help":"OSD benchmark: write <count> <size>-byte objects, (default 1G size 4MB). Results in log.","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd011":{"sig":["flush_pg_stats"],"help":"flush pg stats","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd012":{"sig":["heap",{"name":"heapcmd","strings":"dump|start_profiler|stop_profiler|release|stats","type":"CephChoices"}],"help":"show heap usage info (available only if compiled with tcmalloc)","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd013":{"sig":["debug","dump_missing",{"name":"filename","type":"CephFilepath"}],"help":"dump missing objects to a named file","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd014":{"sig":["debug","kick_recovery_wq",{"name":"delay","range":"0","type":"CephInt"}],"help":"set osd_recovery_delay_start to <val>","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd015":{"sig":["cpu_profiler",{"name":"arg","strings":"status|flush","type":"CephChoices"}],"help":"run cpu profiling on daemon","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd016":{"sig":["dump_pg_recovery_stats"],"help":"dump pg recovery statistics","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd017":{"sig":["reset_pg_recovery_stats"],"help":"reset pg recovery statistics","module":"osd","perm":"rw","avail":"cli,rest","flags":0}}
➜  build git:(wip-19382) ✗ ./bin/ceph tell mgr get_command_descriptions
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:17:40.749573 7ff261fa3700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:17:40.755452 7ff261fa3700 -1 WARNING: all dangerous and experimental features are enabled.
no valid command found; 10 closest matches:
fs status {<fs>}
osd perf {<bucket>}
Error EINVAL: invalid command
➜  build git:(wip-19382) ✗ ceph -s
zsh: command not found: ceph
➜  build git:(wip-19382) ✗ ./bin/ceph -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:17:48.836006 7faa1ff61700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:17:48.838762 7faa1ff61700 -1 WARNING: all dangerous and experimental features are enabled.
    cluster 1d312f4b-b356-42d7-89d7-9493dd133bc7
     health HEALTH_OK
     monmap e2: 3 mons at {a=127.0.0.1:40000/0,b=127.0.0.1:40001/0,c=127.0.0.1:40002/0}
            election epoch 10, quorum 0,1,2 a,b,c
      fsmap e7: 1/1/1 up {0=b=up:active}, 2 up:standby
        mgr active: x 
     osdmap e19: 3 osds: 3 up, 3 in
      pgmap v54: 56 pgs, 7 pools, 5268 bytes data, 203 objects
            158 GB used, 254 GB / 413 GB avail
                  56 active+clean
➜  build git:(wip-19382) ✗ ./bin/ceph tell mgr.x get_command_descriptions
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:18:00.659132 7f9f982e9700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:18:00.695208 7f9f982e9700 -1 WARNING: all dangerous and experimental features are enabled.
no valid command found; 10 closest matches:
fs status {<fs>}
osd perf {<bucket>}
Error EINVAL: invalid command

➜  build git:(wip-19382) ✗ ./bin/ceph tell mgr.x osd perf
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:22:06.333824 7fa1df958700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:22:06.352858 7fa1df958700 -1 WARNING: all dangerous and experimental features are enabled.
osd commit_latency(ms) apply_latency(ms) 
  2                  4                25 
  1                  4                22 
  0                  5                20 

#8 Updated by Tim Serong 2 months ago

Sorry, my mistake. I was confusing this for a problem we had at one point where invoking commands defined in the python modules didn't work (e.g. "bin/ceph tell mgr.x fs status" would always return EINVAL, even when the fsstatus module was loaded).

#9 Updated by Chang Liu 2 months ago

Tim Serong wrote:

Sorry, my mistake. I was confusing this for a problem we had at one point where invoking commands defined in the python modules didn't work (e.g. "bin/ceph tell mgr.x fs status" would always return EINVAL, even when the fsstatus module was loaded).

Hi, Tim

I tried with your command, and it works. maybe you should check mgr log to find the problem.

./bin/ceph tell mgr.x fs status
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 19:26:02.497881 7fcbbc5db700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 19:26:02.541064 7fcbbc5db700 -1 WARNING: all dangerous and experimental features are enabled.
cephfs_a - 0 clients
========
+------+--------+-----+---------------+-------+-------+
| Rank | State  | MDS |    Activity   |  dns  |  inos |
+------+--------+-----+---------------+-------+-------+
|  0   | active |  a  | Reqs:    0 /s |   10  |    2  |
+------+--------+-----+---------------+-------+-------+
+-------------------+----------+-------+-------+
|        Pool       |   type   |  used | avail |
+-------------------+----------+-------+-------+
| cephfs_metadata_a | metadata | 2246  | 87.6G |
|   cephfs_data_a   |   data   |    0  | 87.6G |
+-------------------+----------+-------+-------+

+-------------+
| Standby MDS |
+-------------+
|      c      |
|      b      |
+-------------+
MDS version: ceph version 12.0.1-2360-g38b6dcd (38b6dcd38e11c42124637331b59e8e3a6c747337)

#10 Updated by Tim Serong 2 months ago

Sorry, I'm not explaining myself clearly. What I meant was;

- At some time in the past, commands in python modules (e.g.: "bin/ceph tell mgr.x fs status") didn't work; I mistakenly thought this problem was the issue you were referring to in this bug.

- The problem with commands in python modules not working has since been fixed in master and works fine now.

You should probably just ignore all my comments on this bug :-)

#11 Updated by Chang Liu 2 months ago

Tim Serong wrote:

Sorry, I'm not explaining myself clearly. What I meant was;

- At some time in the past, commands in python modules (e.g.: "bin/ceph tell mgr.x fs status") didn't work; I mistakenly thought this problem was the issue you were referring to in this bug.

- The problem with commands in python modules not working has since been fixed in master and works fine now.

You should probably just ignore all my comments on this bug :-)

thank you for your comments. :-)

#12 Updated by Chang Liu 2 months ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF