Ceph » CephFS

Setting up a cluster I typoed a allow rwx mds instead of allow rwx, like this:

ceph auth get-or-create mds.node1 osd 'allow rwx mds' mds 'allow' mon 'allow profile mds'
ceph auth get-or-create mds.node1 osd 'allow rwx' mds 'allow' mon 'allow profile mds'

This manifests as Operation not permitted, which makes sense, but there is no further info on where that error comes from. I found it out of luck reading over what I had typed again. It would be nice if the error messages pointed out a more direct path to indicate where the error is.

This is the mds log:

2018-03-01 15:27:11.471196 7fe2c270e2c0  0 set uid:gid to 1001:499 (ceph:ceph)
2018-03-01 15:27:11.471209 7fe2c270e2c0  0 ceph version 12.2.3 (2dab17a455c09584f2a85e6b10888337d1ec8949) luminous (stable), process (unknown), pid 2548
2018-03-01 15:27:11.482600 7fe2c270e2c0  0 pidfile_write: ignore empty --pid-file
2018-03-01 15:27:16.121155 7fe2b7eb9700  1 mds.test-node-1 handle_mds_map standby
2018-03-01 15:27:16.147790 7fe2b7eb9700  1 mds.0.4 handle_mds_map i am now mds.0.4
2018-03-01 15:27:16.147791 7fe2b7eb9700  1 mds.0.4 handle_mds_map state change up:boot --> up:creating
2018-03-01 15:27:16.147860 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x1
2018-03-01 15:27:16.148269 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x100
2018-03-01 15:27:16.148278 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x600
2018-03-01 15:27:16.148313 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x601
2018-03-01 15:27:16.148342 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x602
2018-03-01 15:27:16.148369 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x603
2018-03-01 15:27:16.148394 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x604
2018-03-01 15:27:16.148417 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x605
2018-03-01 15:27:16.148444 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x606
2018-03-01 15:27:16.148469 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x607
2018-03-01 15:27:16.148503 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x608
2018-03-01 15:27:16.148524 7fe2b7eb9700  0 mds.0.cache creating system inode with ino:0x609
2018-03-01 15:27:29.874508 7fe2b1ead700  1 mds.0.cache.dir(0x605) commit error -1 v 1
2018-03-01 15:27:29.874516 7fe2b1ead700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x605 object, errno -1
2018-03-01 15:27:29.874519 7fe2b1ead700 -1 mds.0.4 unhandled write error (1) Operation not permitted, force readonly...
2018-03-01 15:27:29.874521 7fe2b1ead700  1 mds.0.cache force file system read-only
2018-03-01 15:27:29.874522 7fe2b1ead700  0 log_channel(cluster) log [WRN] : force file system read-only
2018-03-01 15:27:29.874540 7fe2b1ead700  1 mds.0.cache.ino(0x605) store backtrace error -1 v 0
2018-03-01 15:27:29.874542 7fe2b1ead700 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x605 object, pool 2, errno -1
2018-03-01 15:27:29.874543 7fe2b1ead700 -1 mds.0.4 unhandled write error (1) Operation not permitted, force readonly...
2018-03-01 15:27:30.656603 7fe2b1ead700  1 mds.0.cache.dir(0x606) commit error -1 v 1
2018-03-01 15:27:30.656612 7fe2b1ead700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x606 object, errno -1
2018-03-01 15:27:30.656615 7fe2b1ead700 -1 mds.0.4 unhandled write error (1) Operation not permitted, force readonly...
2018-03-01 15:27:30.656913 7fe2b1ead700  1 mds.0.cache.ino(0x606) store backtrace error -1 v 0
2018-03-01 15:27:30.656918 7fe2b1ead700 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x606 object, pool 2, errno -1
2018-03-01 15:27:30.656920 7fe2b1ead700 -1 mds.0.4 unhandled write error (1) Operation not permitted, force readonly...
2018-03-01 15:27:30.656924 7fe2b1ead700 -1 mds.0.journaler.mdlog(rw) _finish_write_head got (1) Operation not permitted
2018-03-01 15:27:30.656925 7fe2b1ead700 -1 mds.0.journaler.mdlog(rw) handle_write_error (1) Operation not permitted
2018-03-01 15:27:30.656928 7fe2b1ead700  1 mds.0.cache.ino(0x1) store error -1 v 1 on [inode 0x1 [...2,head] / auth v1 snaprealm=0x5602a16af400 f() n(v0 1=0+1) (iversion lock) | dirfrag=1 0x5602a1820000]
2018-03-01 15:27:30.656934 7fe2b1ead700 -1 log_channel(cluster) log [ERR] : failed to store inode 0x1 object: (1) Operation not permitted
2018-03-01 15:27:30.656935 7fe2b1ead700 -1 mds.0.4 unhandled write error (1) Operation not permitted, force readonly...
2018-03-01 15:27:30.656938 7fe2b1ead700 -1 mds.0.journaler.mdlog(rw) _prezeroed got (1) Operation not permitted
2018-03-01 15:27:30.656939 7fe2b1ead700 -1 mds.0.journaler.mdlog(rw) handle_write_error (1) Operation not permitted
2018-03-01 15:27:30.656940 7fe2b1ead700 -1 mds.0.journaler.mdlog(rw) handle_write_error: multiple write errors, handler already called
2018-03-01 15:27:30.656941 7fe2b1ead700 -1 mds.0.log unhandled error (1) Operation not permitted, shutting down...
2018-03-01 15:27:30.656943 7fe2b1ead700 -1 log_channel(cluster) log [ERR] : Unhandled journal write error on MDS rank 0: (1) Operation not permitted, shutting down.
2018-03-01 15:27:30.705870 7fef7f0402c0  0 ceph version 12.2.3 (2dab17a455c09584f2a85e6b10888337d1ec8949) luminous (stable), process (unknown), pid 2548
2018-03-01 15:27:30.707086 7fef7f0402c0  0 pidfile_write: ignore empty --pid-file
2018-03-01 15:27:35.062993 7fef747eb700  1 mds.test-node-1 handle_mds_map standby

Thanks!