Bug #64127
Updated by Rishabh Dave 4 months ago
The command @ceph fs authorize b client.air / rw root_squash /volumes rw@ fails, when it is run for a client that already contains caps for another FS.
Adding caps for a FS to a client is fine. Adding caps for 2nd caps is also fine -
<pre>
$ ./bin/ceph fs volume create b
$ ./bin/ceph fs authorize a client.x / rw
$ ./bin/ceph fs authorize b client.x / rw
[client.x]
key = AQCDj69lh8JqNRAAkaZfZ64v27d5NZUVB7F8Pg==
caps mds = "allow rw fsname=a, allow rw fsname=b"
caps mon = "allow r fsname=a, allow r fsname=b"
caps osd = "allow rw tag cephfs data=a, allow rw tag cephfs data=b"
updated caps for client.x
</pre>
But when following command is run, command hangs -
<pre>
$ ./bin/ceph fs authorize b client.x / rw root_squash /volumes rw
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2024-01-23T14:45:25.827+0530 7ffa7937d6c0 -1 WARNING: all dangerous and experimental features are enabled.
2024-01-23T14:45:25.836+0530 7ffa7937d6c0 -1 WARNING: all dangerous and experimental features are enabled.
</pre>
After this @ceph status@ also hangs because MON daemons have crashed.
Traceback from MON logs -
<pre>
ceph version 19.0.0-772-gd7e2a322a54 (d7e2a322a5490d1fc9395471e25c3a98499975ea) squid (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x125) [0x7f2392c79a3e]
2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2392c79c60]
3: (MDSAuthCaps::merge(MDSAuthCaps)+0x78) [0x560a9b8b057a]
4: (AuthMonitor::caps_update AuthMonitor::_merge_caps<MDSAuthCaps>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::ostream&)+0x106) [0x560a9b86abba]
5: (AuthMonitor::_gen_wanted_caps(EntityAuth&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::ostream&)+0xde) [0x560a9b857344]
6: (AuthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x238e) [0x560a9b861c96]
7: (AuthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x8e) [0x560a9b8630ba]
8: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x853) [0x560a9b9184df]
9: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x20df) [0x560a9b7e8b9b]
10: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x85d) [0x560a9b7ed203]
11: (Monitor::_ms_dispatch(Message*)+0xada) [0x560a9b7eeeea]
12: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x5d1) [0x560a9b7ef773]
13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1045) [0x560a9b7ed9eb]
14: (Monitor::_ms_dispatch(Message*)+0xada) [0x560a9b7eeeea]
15: (Monitor::ms_dispatch(Message*)+0x32) [0x560a9b826932]
16: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x2a) [0x560a9b7f3386]
17: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xa4) [0x7f2392dac4b4]
18: (DispatchQueue::entry()+0x43d) [0x7f2392da8893]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2392e5061b]
20: (Thread::entry_wrapper()+0x3f) [0x7f2392c5489f]
21: (Thread::_entry_func(void*)+0x9) [0x7f2392c548b7]
22: /lib64/libc.so.6(+0x8c947) [0x7f23914ae947]
23: /lib64/libc.so.6(+0x112860) [0x7f2391534860]
</pre>
Following are few log entries before traceback -
<pre>
-5> 2024-01-23T15:36:45.715+0530 7f2389bf46c0 10 mon.a@0(leader).paxosservice(auth 1..13) dispatch 0x560aa40a5200 mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885 con 0x560aa404e1e0
-4> 2024-01-23T15:36:45.716+0530 7f2389bf46c0 5 mon.a@0(leader).paxos(paxos active c 1..229) is_readable = 1 - now=2024-01-23T15:36:45.717565+0530 lease_expire=2024-01-23T15:36:50.530042+0530 has v0 lc 229
-3> 2024-01-23T15:36:45.716+0530 7f2389bf46c0 10 mon.a@0(leader).auth v13 preprocess_query mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885
-2> 2024-01-23T15:36:45.716+0530 7f2389bf46c0 10 mon.a@0(leader).auth v13 prepare_update mon_command({"prefix": "fs authorize", "filesystem": "b", "entity": "client.x", "caps": ["/", "rw", "root_squash", "/volumes", "rw"]} v 0) v1 from client.? 192.168.29.219:0/842725885
-1> 2024-01-23T15:36:45.725+0530 7f2389bf46c0 -1 /home/rishabh/repos/ceph/test2/src/mds/MDSAuthCaps.cc: In function 'bool MDSAuthCaps::merge(MDSAuthCaps)' thread 7f2389bf46c0 time 2024-01-23T15:36:45.717766+0530
/home/rishabh/repos/ceph/test2/src/mds/MDSAuthCaps.cc: 387: FAILED ceph_assert(newcap.grants.size() == 1)
</pre>