Project

General

Profile

Actions
Copy link

Bug #21192

closed

ceph.in: ceph tell gives EINVAL errors

Added by Patrick Donnelly over 6 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Chang Liu
Category:
ceph cli
Target version:
v12.2.0
% Done:

0%

Source:
Development
Tags:
Backport:
luminous
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

On a vstart cluster following http://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#runtime-changes

pdonnell@senta02 ~/ceph/build$ bin/ceph tell mds.\* injectargs '--mds_cache_size = 10001'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-08-30 18:53:36.108413 7fcf5f010700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:53:36.120089 7fcf5f010700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:53:36.158967 7fcf6d7f3700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:53:36.159262 7fcf6d7f3700 -1 asok(0x1e3caa0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.F0eyT1/client.admin.2320.asok': (17) File exists
2017-08-30 18:53:36.174302 7fcf09ffb700  0 client.4297 ms_handle_reset on 127.0.0.1:6846/2527346815
2017-08-30 18:53:36.174327 7fcf09ffb700  0 client.4297 ms_handle_reset on 127.0.0.1:6845/299228287
2017-08-30 18:53:36.174386 7fcf09ffb700  0 client.4297 ms_handle_reset on 127.0.0.1:6847/2179721628
2017-08-30 18:53:36.202308 7fcf6d7f3700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:53:36.202524 7fcf6d7f3700 -1 asok(0x1e3caa0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.F0eyT1/client.admin.2320.asok': (17) File exists
2017-08-30 18:53:36.213893 7fcf097fa700  0 client.4259 ms_handle_reset on 127.0.0.1:6846/2527346815
2017-08-30 18:53:36.213906 7fcf097fa700  0 client.4259 ms_handle_reset on 127.0.0.1:6845/299228287
2017-08-30 18:53:36.214007 7fcf097fa700  0 client.4259 ms_handle_reset on 127.0.0.1:6847/2179721628
Error EINVAL: 
pdonnell@senta02 ~/ceph/build$ bin/ceph tell mds.a injectargs '--mds_cache_size = 10001'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-08-30 18:53:42.054454 7f0eed5b6700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:53:42.067568 7f0eed5b6700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:53:42.104772 7f0efbd99700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:53:42.105157 7f0efbd99700 -1 asok(0x22d2aa0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.F0eyT1/client.admin.2373.asok': (17) File exists
2017-08-30 18:53:42.116647 7f0e977fe700  0 client.4262 ms_handle_reset on 127.0.0.1:6845/299228287
2017-08-30 18:53:42.145215 7f0efbd99700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:53:42.145515 7f0efbd99700 -1 asok(0x23c8430) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.F0eyT1/client.admin.2373.asok': (17) File exists
2017-08-30 18:53:42.159517 7f0e977fe700  0 client.4265 ms_handle_reset on 127.0.0.1:6845/299228287
Error EINVAL: 

$ bin/ceph mds tell mon.a injectargs --mds_cache_size 10001
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-08-30 18:50:07.587730 7f2addfab700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-30 18:50:07.599689 7f2addfab700 -1 WARNING: all dangerous and experimental features are enabled.
Invalid command:  saw 0 of args(<string>) [<string>...], expected at least 1
mds tell <who> <args> [<args>...] :  send command to particular mds
Error EINVAL: invalid command

Related issues 1 (0 open — 1 closed)

Related to CephFS - Bug #21191: ceph: tell mds.* results in warningResolved08/30/2017

Actions
Actions
Copy link
 #1

Updated by Patrick Donnelly over 6 years ago

  • Related to Bug #21191: ceph: tell mds.* results in warning added
Actions
Copy link
 #2

Updated by Patrick Donnelly over 6 years ago

Okay, the right thing to do is:

$ bin/ceph tell mds.\* injectargs --mds_cache_size 1001
mds_cache_size = '1001' (not observed, change may require restart)

The standbys are not updated but that's a separate problem.

It's surprising that this works even without the fix to #21191. The validation logic for injectargs is a different code-path.

As for `ceph mds tell`, that still appears to be broken?

Actions
Copy link
 #3

Updated by Chang Liu over 6 years ago

  • Assignee set to Chang Liu
Actions
Copy link
 #4

Updated by Chang Liu over 6 years ago 

➜  build git:(master) ✗ bin/ceph tell mds.\* injectargs '--mds_cache_size=10001'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-08-31 17:21:46.778070 7f09b5e6c700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-31 17:21:46.791325 7f09b5e6c700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-31 17:21:46.810790 7f09c47e4700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-31 17:21:46.811112 7f09c47e4700 -1 asok(0x1d3aee0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.hEeEEp/client.admin.26610.asok': (17) File exists
2017-08-31 17:21:46.817195 7f096b7fe700  0 client.4144 ms_handle_reset on 127.0.0.1:6813/653633072
2017-08-31 17:21:46.817201 7f096b7fe700  0 client.4144 ms_handle_reset on 127.0.0.1:6814/2779152210
2017-08-31 17:21:46.817203 7f096b7fe700  0 client.4144 ms_handle_reset on 127.0.0.1:6815/3565352982
2017-08-31 17:21:46.832064 7f09c47e4700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-31 17:21:46.832224 7f09c47e4700 -1 asok(0x1d3aee0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.hEeEEp/client.admin.26610.asok': (17) File exists
2017-08-31 17:21:46.869087 7f096b7fe700  0 client.4179 ms_handle_reset on 127.0.0.1:6813/653633072
2017-08-31 17:21:46.869101 7f096b7fe700  0 client.4179 ms_handle_reset on 127.0.0.1:6815/3565352982
2017-08-31 17:21:46.869106 7f096b7fe700  0 client.4179 ms_handle_reset on 127.0.0.1:6814/2779152210
mds_cache_size = '10001' (not observed, change may require restart) 
➜  build git:(master) ✗ bin/ceph tell osd.\* injectargs '--mds_cache_size=10001'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-08-31 17:22:02.772591 7f7a1f739700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-31 17:22:02.779543 7f7a1f739700 -1 WARNING: all dangerous and experimental features are enabled.
osd.0: mds_cache_size = '10001' (not observed, change may require restart) osd_objectstore = 'filestore' (not observed, change may require restart) 
osd.1: mds_cache_size = '10001' (not observed, change may require restart) osd_objectstore = 'filestore' (not observed, change may require restart) 
osd.2: mds_cache_size = '10001' (not observed, change may require restart) osd_objectstore = 'filestore' (not observed, change may require restart) 
➜  build git:(master) ✗ bin/ceph tell mon.\* injectargs '--mds_cache_size=10001'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2017-08-31 17:22:35.606214 7fd10b5bb700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-31 17:22:35.613034 7fd10b5bb700 -1 WARNING: all dangerous and experimental features are enabled.
mon.a: injectargs:mds_cache_size = '10001' (not observed, change may require restart) 
mon.b: injectargs:mds_cache_size = '10001' (not observed, change may require restart) 
mon.c: injectargs:mds_cache_size = '10001' (not observed, change may require restart)

CLI will send "injectargs" command to active MDS only when target is "mds.*". it's bug. both "mon.*" and "osd.*" works well.

Actions
Copy link
 #5

Updated by Chang Liu over 6 years ago

  • Status changed from New to Closed

close this issue. I create another issue "the standbys are not updated via 'ceph tell mds.* command'" , see http://tracker.ceph.com/issues/21230

Actions
Copy link

Also available in: Atom PDF