Actions
Bug #21192
closedceph.in: ceph tell gives EINVAL errors
Source:
Development
Tags:
Backport:
luminous
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
On a vstart cluster following http://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#runtime-changes
pdonnell@senta02 ~/ceph/build$ bin/ceph tell mds.\* injectargs '--mds_cache_size = 10001' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-08-30 18:53:36.108413 7fcf5f010700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:53:36.120089 7fcf5f010700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:53:36.158967 7fcf6d7f3700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:53:36.159262 7fcf6d7f3700 -1 asok(0x1e3caa0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.F0eyT1/client.admin.2320.asok': (17) File exists 2017-08-30 18:53:36.174302 7fcf09ffb700 0 client.4297 ms_handle_reset on 127.0.0.1:6846/2527346815 2017-08-30 18:53:36.174327 7fcf09ffb700 0 client.4297 ms_handle_reset on 127.0.0.1:6845/299228287 2017-08-30 18:53:36.174386 7fcf09ffb700 0 client.4297 ms_handle_reset on 127.0.0.1:6847/2179721628 2017-08-30 18:53:36.202308 7fcf6d7f3700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:53:36.202524 7fcf6d7f3700 -1 asok(0x1e3caa0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.F0eyT1/client.admin.2320.asok': (17) File exists 2017-08-30 18:53:36.213893 7fcf097fa700 0 client.4259 ms_handle_reset on 127.0.0.1:6846/2527346815 2017-08-30 18:53:36.213906 7fcf097fa700 0 client.4259 ms_handle_reset on 127.0.0.1:6845/299228287 2017-08-30 18:53:36.214007 7fcf097fa700 0 client.4259 ms_handle_reset on 127.0.0.1:6847/2179721628 Error EINVAL: pdonnell@senta02 ~/ceph/build$ bin/ceph tell mds.a injectargs '--mds_cache_size = 10001' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-08-30 18:53:42.054454 7f0eed5b6700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:53:42.067568 7f0eed5b6700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:53:42.104772 7f0efbd99700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:53:42.105157 7f0efbd99700 -1 asok(0x22d2aa0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.F0eyT1/client.admin.2373.asok': (17) File exists 2017-08-30 18:53:42.116647 7f0e977fe700 0 client.4262 ms_handle_reset on 127.0.0.1:6845/299228287 2017-08-30 18:53:42.145215 7f0efbd99700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:53:42.145515 7f0efbd99700 -1 asok(0x23c8430) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.F0eyT1/client.admin.2373.asok': (17) File exists 2017-08-30 18:53:42.159517 7f0e977fe700 0 client.4265 ms_handle_reset on 127.0.0.1:6845/299228287 Error EINVAL:
$ bin/ceph mds tell mon.a injectargs --mds_cache_size 10001 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-08-30 18:50:07.587730 7f2addfab700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-30 18:50:07.599689 7f2addfab700 -1 WARNING: all dangerous and experimental features are enabled. Invalid command: saw 0 of args(<string>) [<string>...], expected at least 1 mds tell <who> <args> [<args>...] : send command to particular mds Error EINVAL: invalid command
Updated by Patrick Donnelly over 6 years ago
- Related to Bug #21191: ceph: tell mds.* results in warning added
Updated by Patrick Donnelly over 6 years ago
Okay, the right thing to do is:
$ bin/ceph tell mds.\* injectargs --mds_cache_size 1001 mds_cache_size = '1001' (not observed, change may require restart)
The standbys are not updated but that's a separate problem.
It's surprising that this works even without the fix to #21191. The validation logic for injectargs is a different code-path.
As for `ceph mds tell`, that still appears to be broken?
Updated by Chang Liu over 6 years ago
➜ build git:(master) ✗ bin/ceph tell mds.\* injectargs '--mds_cache_size=10001' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-08-31 17:21:46.778070 7f09b5e6c700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-31 17:21:46.791325 7f09b5e6c700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-31 17:21:46.810790 7f09c47e4700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-31 17:21:46.811112 7f09c47e4700 -1 asok(0x1d3aee0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.hEeEEp/client.admin.26610.asok': (17) File exists 2017-08-31 17:21:46.817195 7f096b7fe700 0 client.4144 ms_handle_reset on 127.0.0.1:6813/653633072 2017-08-31 17:21:46.817201 7f096b7fe700 0 client.4144 ms_handle_reset on 127.0.0.1:6814/2779152210 2017-08-31 17:21:46.817203 7f096b7fe700 0 client.4144 ms_handle_reset on 127.0.0.1:6815/3565352982 2017-08-31 17:21:46.832064 7f09c47e4700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-31 17:21:46.832224 7f09c47e4700 -1 asok(0x1d3aee0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.hEeEEp/client.admin.26610.asok': (17) File exists 2017-08-31 17:21:46.869087 7f096b7fe700 0 client.4179 ms_handle_reset on 127.0.0.1:6813/653633072 2017-08-31 17:21:46.869101 7f096b7fe700 0 client.4179 ms_handle_reset on 127.0.0.1:6815/3565352982 2017-08-31 17:21:46.869106 7f096b7fe700 0 client.4179 ms_handle_reset on 127.0.0.1:6814/2779152210 mds_cache_size = '10001' (not observed, change may require restart) ➜ build git:(master) ✗ bin/ceph tell osd.\* injectargs '--mds_cache_size=10001' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-08-31 17:22:02.772591 7f7a1f739700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-31 17:22:02.779543 7f7a1f739700 -1 WARNING: all dangerous and experimental features are enabled. osd.0: mds_cache_size = '10001' (not observed, change may require restart) osd_objectstore = 'filestore' (not observed, change may require restart) osd.1: mds_cache_size = '10001' (not observed, change may require restart) osd_objectstore = 'filestore' (not observed, change may require restart) osd.2: mds_cache_size = '10001' (not observed, change may require restart) osd_objectstore = 'filestore' (not observed, change may require restart) ➜ build git:(master) ✗ bin/ceph tell mon.\* injectargs '--mds_cache_size=10001' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2017-08-31 17:22:35.606214 7fd10b5bb700 -1 WARNING: all dangerous and experimental features are enabled. 2017-08-31 17:22:35.613034 7fd10b5bb700 -1 WARNING: all dangerous and experimental features are enabled. mon.a: injectargs:mds_cache_size = '10001' (not observed, change may require restart) mon.b: injectargs:mds_cache_size = '10001' (not observed, change may require restart) mon.c: injectargs:mds_cache_size = '10001' (not observed, change may require restart)
CLI will send "injectargs" command to active MDS only when target is "mds.*". it's bug. both "mon.*" and "osd.*" works well.
Updated by Chang Liu over 6 years ago
- Status changed from New to Closed
close this issue. I create another issue "the standbys are not updated via 'ceph tell mds.* command'" , see http://tracker.ceph.com/issues/21230
Actions