Ceph » CephFS

The command ceph fs authorize b client.air / rw root_squash /volumes rw fails, when it is run for a client that already contains caps for another FS.

Adding caps for a FS to a client is fine. Adding caps for 2nd caps is also fine -

$ ./bin/ceph fs volume create b
$ ./bin/ceph fs authorize a client.x / rw
$ ./bin/ceph fs authorize b client.x / rw
[client.x]
        key = AQCDj69lh8JqNRAAkaZfZ64v27d5NZUVB7F8Pg==
        caps mds = "allow rw fsname=a, allow rw fsname=b" 
        caps mon = "allow r fsname=a, allow r fsname=b" 
        caps osd = "allow rw tag cephfs data=a, allow rw tag cephfs data=b" 
updated caps for client.x

But when following command is run, command hangs -

$ ./bin/ceph fs authorize b client.x / rw root_squash /volumes rw
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2024-01-23T14:45:25.827+0530 7ffa7937d6c0 -1 WARNING: all dangerous and experimental features are enabled.
2024-01-23T14:45:25.836+0530 7ffa7937d6c0 -1 WARNING: all dangerous and experimental features are enabled.

After this ceph status also hangs because MON daemons have crashed.

Traceback from MON logs -

 ceph version 19.0.0-772-gd7e2a322a54 (d7e2a322a5490d1fc9395471e25c3a98499975ea) squid (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x125) [0x7f2392c79a3e]
 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2392c79c60]
 3: (MDSAuthCaps::merge(MDSAuthCaps)+0x78) [0x560a9b8b057a]
 4: (AuthMonitor::caps_update AuthMonitor::_merge_caps<MDSAuthCaps>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::ostream&)+0x106) [0x560a9b86abba]
 5: (AuthMonitor::_gen_wanted_caps(EntityAuth&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::ostream&)+0xde) [0x560a9b857344]
 6: (AuthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x238e) [0x560a9b861c96]
 7: (AuthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x8e) [0x560a9b8630ba]
 8: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x853) [0x560a9b9184df]
 9: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x20df) [0x560a9b7e8b9b]
 10: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x85d) [0x560a9b7ed203]
 11: (Monitor::_ms_dispatch(Message*)+0xada) [0x560a9b7eeeea]
 12: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x5d1) [0x560a9b7ef773]
 13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1045) [0x560a9b7ed9eb]
 14: (Monitor::_ms_dispatch(Message*)+0xada) [0x560a9b7eeeea]
 15: (Monitor::ms_dispatch(Message*)+0x32) [0x560a9b826932]
 16: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x2a) [0x560a9b7f3386]
 17: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xa4) [0x7f2392dac4b4]
 18: (DispatchQueue::entry()+0x43d) [0x7f2392da8893]
 19: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2392e5061b]
 20: (Thread::entry_wrapper()+0x3f) [0x7f2392c5489f]
 21: (Thread::_entry_func(void*)+0x9) [0x7f2392c548b7]
 22: /lib64/libc.so.6(+0x8c947) [0x7f23914ae947]
 23: /lib64/libc.so.6(+0x112860) [0x7f2391534860]

Following are few log entries before traceback -

    -5> 2024-01-23T15:36:45.715+0530 7f2389bf46c0 10 mon.a@0(leader).paxosservice(auth 1..13) dispatch 0x560aa40a5200 mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885 con 0x560aa404e1e0
    -4> 2024-01-23T15:36:45.716+0530 7f2389bf46c0  5 mon.a@0(leader).paxos(paxos active c 1..229) is_readable = 1 - now=2024-01-23T15:36:45.717565+0530 lease_expire=2024-01-23T15:36:50.530042+0530 has v0 lc 229
    -3> 2024-01-23T15:36:45.716+0530 7f2389bf46c0 10 mon.a@0(leader).auth v13 preprocess_query mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885
    -2> 2024-01-23T15:36:45.716+0530 7f2389bf46c0 10 mon.a@0(leader).auth v13 prepare_update mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885
    -1> 2024-01-23T15:36:45.725+0530 7f2389bf46c0 -1 /home/rishabh/repos/ceph/test2/src/mds/MDSAuthCaps.cc: In function 'bool MDSAuthCaps::merge(MDSAuthCaps)' thread 7f2389bf46c0 time 2024-01-23T15:36:45.717766+0530
/home/rishabh/repos/ceph/test2/src/mds/MDSAuthCaps.cc: 387: FAILED ceph_assert(newcap.grants.size() == 1)