Bug #44297
mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymous connection")
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
on nautilus->octopus/master upgrade
2020-02-25T18:14:31.772+0000 7fc6d6a21700 -1 /build/ceph-15.1.0-1203-gc0c0508/src/mon/Monitor.cc: In function 'virtual int AnonConnection::send_message(Message*)' thread 7fc6d6a21700 time 2020-02-25T18:14:31.771106+0000 /build/ceph-15.1.0-1203-gc0c0508/src/mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymous connection") ceph version 15.1.0-1203-gc0c0508 (c0c0508e4e8f39832936b50e37613b72e3465e91) octopus (rc) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7fc6db971fb0] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc6db97218b] 3: (AnonConnection::send_keepalive()+0) [0x557cd4b3f5c0] 4: (()+0x32cd92) [0x7fc6dba37d92] 5: (()+0x32c07b) [0x7fc6dba3707b] 6: (AdminSocketHook::call_async(std::basic_string_view<char, std::char_traits<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, ceph::Formatter*, ceph::buffer::v14_2_0::list const&, std::function<void (int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v14_2_0::list&)>)+0x143) [0x557cd4b67613] 7: (AdminSocket::execute_command(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ceph::buffer::v14_2_0::list const&, std::function<void (int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v14_2_0::list&)>)+0x8dd) [0x7fc6dba399bd] 8: (AdminSocket::do_tell_queue()+0x2e5) [0x7fc6dba3a6d5] 9: (AdminSocket::entry()+0x31f) [0x7fc6dba3bdef] 10: (()+0xbd66f) [0x7fc6da30666f] 11: (()+0x76db) [0x7fc6da7dd6db] 12: (clone()+0x3f) [0x7fc6d99c388f]
/a/sage-2020-02-25_15:51:04-rados-wip-sage2-testing-2020-02-25-0704-distro-basic-smithi/4801786
History
#1 Updated by Sage Weil over 3 years ago
- Project changed from Ceph to RADOS
#2 Updated by Sage Weil over 3 years ago
The command is passed from a nautilus monitor:
-150> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 1 -- v1:172.21.15.117:6789/0 <== mon.2 v1:172.21.15.203:6789/0 8 ==== forward(mon_command({"prefix": "smart", "format": "json", "devid": ""} v 0) v1 caps allow profile mgr tid 398 con_features 4540138292836630527) v4 ==== 297+0+0 (unknown 1276046022 0 0) 0x557cd7ac7f80 con 0x557cd6b5b400
-149> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 mon.b@1(leader) e1 _ms_dispatch existing session 0x557cd6ab8d00 for mon.2
-148> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 mon.b@1(leader) e1 entity caps allow *
-147> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 is_capable service=mon command= read addr v1:172.21.15.203:6789/0 on cap allow *
-146> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 allow so far , doing grant allow *
-145> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 allow all
-144> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 10 mon.b@1(leader) e1 received forwarded message from mgr v1:172.21.15.110:0/16681 via mon.2 v1:172.21.15.203:6789/0
-143> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 is_capable service=mon command= exec addr v1:172.21.15.203:6789/0 on cap allow *
-142> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 allow so far , doing grant allow *
-141> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 allow all
-140> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 10 mon.b@1(leader) e1 caps are allow profile mgr
-139> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 10 mon.b@1(leader) e1 entity name '' type 0
-138> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 10 mon.b@1(leader) e1 mesg 0x557cd8185e00 from v1:172.21.15.203:6789/0
-137> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 mon.b@1(leader) e1 _ms_dispatch existing session 0x557cd827f840 for mgr.84111
-136> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 mon.b@1(leader) e1 entity caps allow profile mgr
-135> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 5 mon.b@1(leader) e1 handle_command passing command to tell/asok
Looks like the new asok/tell behavior, which assumes the command is local, is colliding with the nautilus behavior, which was semi-broken and would occasionally forward these commands around.
#3 Updated by Sage Weil over 3 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 33542
#4 Updated by Sage Weil over 3 years ago
- Status changed from Fix Under Review to Resolved