Bug #63381
Updated by Xiubo Li 7 months ago
This is initially from https://github.com/ceph/ceph/pull/53981#issuecomment-1760916771. And I added some debug logs and reproduced it again: Jenkins test failed : https://jenkins.ceph.com/job/ceph-api/64028/console <pre> -24> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/dashboard/x/ssl_server_port = 7789 (unrecognized option) -23> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/dashboard/y/ssl_server_port = 7790 (unrecognized option) -22> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/dashboard/z/ssl_server_port = 7791 (unrecognized option) -21> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/prometheus/x/server_port = 9283 (unrecognized option) -20> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/prometheus/y/server_port = 10283 (unrecognized option) -19> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/prometheus/z/server_port = 11283 (unrecognized option) -18> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/restful/x/server_port = 42803 (unrecognized option) -17> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/restful/y/server_port = 44803 (unrecognized option) -16> 2023-11-01T06:36:01.866+0000 7f25a8d41640 10 set_mon_vals mgr/restful/z/server_port = 46803 (unrecognized option) -15> 2023-11-01T06:36:01.870+0000 7f25a4538640 5 client.6679 collect_and_send_global_metrics MDS rank 0 is not ready yet -- not sending metric -14> 2023-11-01T06:36:01.870+0000 7f25a5d3b640 1 client.6679 handle_mds_map epoch 10 -13> 2023-11-01T06:36:01.870+0000 7f25ebf1c640 10 monclient: get_auth_request con 0x5604f3f6b800 auth_method 0 -12> 2023-11-01T06:36:01.882+0000 7f25b1051640 5 client.6679 xattr_permission 0x5604f3538a00 = 0 -11> 2023-11-01T06:36:01.882+0000 7f25b1051640 5 client.6679 xattr_permission 0x5604f3538a00 = 0 -10> 2023-11-01T06:36:01.886+0000 7f25b1051640 2 client.6679 unmounting -9> 2023-11-01T06:36:01.886+0000 7f25b1051640 0 client.6679 _unmount:6618 -8> 2023-11-01T06:36:01.886+0000 7f25b1051640 0 client.6679 _unmount:6634 -7> 2023-11-01T06:36:01.886+0000 7f25b1051640 0 client.6679 _unmount:6646 -6> 2023-11-01T06:36:01.886+0000 7f25b1051640 0 client.6679 _unmount:6655 -5> 2023-11-01T06:36:01.886+0000 7f25b1051640 0 client.6679 _unmount:6664 -4> 2023-11-01T06:36:01.886+0000 7f25b1051640 0 client.6679 _unmount:6687 -3> 2023-11-01T06:36:01.886+0000 7f25b1051640 0 client.6679 _unmount:6702 -2> 2023-11-01T06:36:01.886+0000 7f25b1051640 0 client.6679 flush_caps_sync:4823 -1> 2023-11-01T06:36:01.898+0000 7f25b1051640 -1 /home/jenkins-build/build/workspace/ceph-api/src/client/MetaSession.h: In function 'MetaSession::~MetaSession()' thread 7f25b1051640 time 2023-11-01T06:36:01.886870+0000 /home/jenkins-build/build/workspace/ceph-api/src/client/MetaSession.h: 66: FAILED ceph_assert(caps.empty()) ceph version Development (no_version) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a1) [0x7f25eec0c161] 2: /home/jenkins-build/build/workspace/ceph-api/build/lib/libceph-common.so.2(+0x16df3f9) [0x7f25eec0c3f9] 3: /home/jenkins-build/build/workspace/ceph-api/build/lib/libcephfs.so.2(+0x18305b) [0x7f25e119e05b] 4: /home/jenkins-build/build/workspace/ceph-api/build/lib/libcephfs.so.2(+0xf1ef9) [0x7f25e110cef9] 5: /home/jenkins-build/build/workspace/ceph-api/build/lib/libcephfs.so.2(+0x106b97) [0x7f25e1121b97] 6: /home/jenkins-build/build/workspace/ceph-api/build/lib/libcephfs.so.2(+0x108053) [0x7f25e1123053] 7: (ceph_mount_info::shutdown()+0x2c) [0x7f25e10c283c] 8: ceph_shutdown() </pre>