Ceph » mgr

sorry, i missed document link: http://docs.ceph.com/docs/kraken/mgr/administrator/

Well, I try to read relevant code. here is different between `ceph tell` and `daemon(admin_socket)`, which is just my understand, and may be mistake.

1. `ceph tell` is a MMonCommand. for example, if we use `ceph tell osd.12 bench`, MON will send osd.12 a MMonCommand to OSD.12. see `void OSD::handle_command(MMonCommand *m)`
2. `daemon(adin_socket)` is based on admin_socket(unix_socket). We must use this feature in corresponding machine.

Now, what should we do ?

1) update documentation?
2) add new MMonCommand to MGR?

MgrPyModules reports all supported commands via "get_commands", we could report the supported commands by handling MMonCommand.

do you want to take over this ticket? or i can take care of it.

This problem doesn't occur in master. I believe it was fixed by https://github.com/ceph/ceph/commit/9988a564d8

Tim Serong wrote:

This problem doesn't occur in master. I believe it was fixed by https://github.com/ceph/ceph/commit/9988a564d8

Thanks, Tim. I tried it with master branch. it seems that it was not fixed, but it's ok, I will fix it as soon as possible.

➜  build git:(wip-19382) ✗ ./bin/ceph tell osd.1 get_command_descriptions
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:17:33.809898 7f054c4fc700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:17:33.818258 7f054c4fc700 -1 WARNING: all dangerous and experimental features are enabled.
{"cmd000":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"query","type":"CephChoices"}],"help":"show details of a specific pg","module":"osd","perm":"r","avail":"cli","flags":0},"cmd001":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"mark_unfound_lost","type":"CephChoices"},{"name":"mulcmd","strings":"revert|delete","type":"CephChoices"}],"help":"mark all unfound objects in this pg as lost, either removing or reverting to a prior version if one is available","module":"osd","perm":"rw","avail":"cli","flags":0},"cmd002":{"sig":["pg",{"name":"pgid","type":"CephPgid"},{"name":"cmd","strings":"list_missing","type":"CephChoices"},{"name":"offset","req":"false","type":"CephString"}],"help":"list missing objects on this pg, perhaps starting at an offset given in JSON","module":"osd","perm":"r","avail":"cli","flags":0},"cmd003":{"sig":["query"],"help":"show details of a specific pg","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd004":{"sig":["mark_unfound_lost",{"name":"mulcmd","strings":"revert|delete","type":"CephChoices"}],"help":"mark all unfound objects in this pg as lost, either removing or reverting to a prior version if one is available","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd005":{"sig":["list_missing",{"name":"offset","req":"false","type":"CephString"}],"help":"list missing objects on this pg, perhaps starting at an offset given in JSON","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd006":{"sig":["version"],"help":"report version of OSD","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd007":{"sig":["get_command_descriptions"],"help":"list commands descriptions","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd008":{"sig":["injectargs",{"n":"N","name":"injected_args","type":"CephString"}],"help":"inject configuration arguments into running OSD","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd009":{"sig":["cluster_log",{"debug":"\u0001","info":"\u0001","name":"level","strings":"error","type":"CephChoices","warning":"\u0001"},{"n":"N","name":"message","type":"CephString"}],"help":"log a message to the cluster log","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd010":{"sig":["bench",{"name":"count","req":"false","type":"CephInt"},{"name":"size","req":"false","type":"CephInt"},{"name":"object_size","req":"false","type":"CephInt"},{"name":"object_num","req":"false","type":"CephInt"}],"help":"OSD benchmark: write <count> <size>-byte objects, (default 1G size 4MB). Results in log.","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd011":{"sig":["flush_pg_stats"],"help":"flush pg stats","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd012":{"sig":["heap",{"name":"heapcmd","strings":"dump|start_profiler|stop_profiler|release|stats","type":"CephChoices"}],"help":"show heap usage info (available only if compiled with tcmalloc)","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd013":{"sig":["debug","dump_missing",{"name":"filename","type":"CephFilepath"}],"help":"dump missing objects to a named file","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd014":{"sig":["debug","kick_recovery_wq",{"name":"delay","range":"0","type":"CephInt"}],"help":"set osd_recovery_delay_start to <val>","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd015":{"sig":["cpu_profiler",{"name":"arg","strings":"status|flush","type":"CephChoices"}],"help":"run cpu profiling on daemon","module":"osd","perm":"rw","avail":"cli,rest","flags":0},"cmd016":{"sig":["dump_pg_recovery_stats"],"help":"dump pg recovery statistics","module":"osd","perm":"r","avail":"cli,rest","flags":0},"cmd017":{"sig":["reset_pg_recovery_stats"],"help":"reset pg recovery statistics","module":"osd","perm":"rw","avail":"cli,rest","flags":0}}
➜  build git:(wip-19382) ✗ ./bin/ceph tell mgr get_command_descriptions
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:17:40.749573 7ff261fa3700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:17:40.755452 7ff261fa3700 -1 WARNING: all dangerous and experimental features are enabled.
no valid command found; 10 closest matches:
fs status {<fs>}
osd perf {<bucket>}
Error EINVAL: invalid command
➜  build git:(wip-19382) ✗ ceph -s
zsh: command not found: ceph
➜  build git:(wip-19382) ✗ ./bin/ceph -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:17:48.836006 7faa1ff61700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:17:48.838762 7faa1ff61700 -1 WARNING: all dangerous and experimental features are enabled.
    cluster 1d312f4b-b356-42d7-89d7-9493dd133bc7
     health HEALTH_OK
     monmap e2: 3 mons at {a=127.0.0.1:40000/0,b=127.0.0.1:40001/0,c=127.0.0.1:40002/0}
            election epoch 10, quorum 0,1,2 a,b,c
      fsmap e7: 1/1/1 up {0=b=up:active}, 2 up:standby
        mgr active: x 
     osdmap e19: 3 osds: 3 up, 3 in
      pgmap v54: 56 pgs, 7 pools, 5268 bytes data, 203 objects
            158 GB used, 254 GB / 413 GB avail
                  56 active+clean
➜  build git:(wip-19382) ✗ ./bin/ceph tell mgr.x get_command_descriptions
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:18:00.659132 7f9f982e9700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:18:00.695208 7f9f982e9700 -1 WARNING: all dangerous and experimental features are enabled.
no valid command found; 10 closest matches:
fs status {<fs>}
osd perf {<bucket>}
Error EINVAL: invalid command

➜  build git:(wip-19382) ✗ ./bin/ceph tell mgr.x osd perf
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 15:22:06.333824 7fa1df958700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 15:22:06.352858 7fa1df958700 -1 WARNING: all dangerous and experimental features are enabled.
osd commit_latency(ms) apply_latency(ms) 
  2                  4                25 
  1                  4                22 
  0                  5                20 

Sorry, my mistake. I was confusing this for a problem we had at one point where invoking commands defined in the python modules didn't work (e.g. "bin/ceph tell mgr.x fs status" would always return EINVAL, even when the fsstatus module was loaded).

Tim Serong wrote:

Sorry, my mistake. I was confusing this for a problem we had at one point where invoking commands defined in the python modules didn't work (e.g. "bin/ceph tell mgr.x fs status" would always return EINVAL, even when the fsstatus module was loaded).

Hi, Tim

I tried with your command, and it works. maybe you should check mgr log to find the problem.

./bin/ceph tell mgr.x fs status
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-05-15 19:26:02.497881 7fcbbc5db700 -1 WARNING: all dangerous and experimental features are enabled.
2017-05-15 19:26:02.541064 7fcbbc5db700 -1 WARNING: all dangerous and experimental features are enabled.
cephfs_a - 0 clients
========
+------+--------+-----+---------------+-------+-------+
| Rank | State  | MDS |    Activity   |  dns  |  inos |
+------+--------+-----+---------------+-------+-------+
|  0   | active |  a  | Reqs:    0 /s |   10  |    2  |
+------+--------+-----+---------------+-------+-------+
+-------------------+----------+-------+-------+
|        Pool       |   type   |  used | avail |
+-------------------+----------+-------+-------+
| cephfs_metadata_a | metadata | 2246  | 87.6G |
|   cephfs_data_a   |   data   |    0  | 87.6G |
+-------------------+----------+-------+-------+

+-------------+
| Standby MDS |
+-------------+
|      c      |
|      b      |
+-------------+
MDS version: ceph version 12.0.1-2360-g38b6dcd (38b6dcd38e11c42124637331b59e8e3a6c747337)

Sorry, I'm not explaining myself clearly. What I meant was;

- At some time in the past, commands in python modules (e.g.: "bin/ceph tell mgr.x fs status") didn't work; I mistakenly thought this problem was the issue you were referring to in this bug.

- The problem with commands in python modules not working has since been fixed in master and works fine now.

You should probably just ignore all my comments on this bug :-)

Tim Serong wrote:

Sorry, I'm not explaining myself clearly. What I meant was;

- At some time in the past, commands in python modules (e.g.: "bin/ceph tell mgr.x fs status") didn't work; I mistakenly thought this problem was the issue you were referring to in this bug.

- The problem with commands in python modules not working has since been fixed in master and works fine now.

You should probably just ignore all my comments on this bug :-)

thank you for your comments. :-)