Project

General

Profile

Actions

Bug #44297

closed

mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymous connection")

Added by Sage Weil about 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

on nautilus->octopus/master upgrade


2020-02-25T18:14:31.772+0000 7fc6d6a21700 -1 /build/ceph-15.1.0-1203-gc0c0508/src/mon/Monitor.cc: In function 'virtual int AnonConnection::send_message(Message*)' thread 7fc6d6a21700 time 2020-02-25T18:14:31.771106+0000
/build/ceph-15.1.0-1203-gc0c0508/src/mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymous connection")

 ceph version 15.1.0-1203-gc0c0508 (c0c0508e4e8f39832936b50e37613b72e3465e91) octopus (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7fc6db971fb0]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fc6db97218b]
 3: (AnonConnection::send_keepalive()+0) [0x557cd4b3f5c0]
 4: (()+0x32cd92) [0x7fc6dba37d92]
 5: (()+0x32c07b) [0x7fc6dba3707b]
 6: (AdminSocketHook::call_async(std::basic_string_view<char, std::char_traits<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, ceph::Formatter*, ceph::buffer::v14_2_0::list const&, std::function<void (int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v14_2_0::list&)>)+0x143) [0x557cd4b67613]
 7: (AdminSocket::execute_command(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ceph::buffer::v14_2_0::list const&, std::function<void (int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v14_2_0::list&)>)+0x8dd) [0x7fc6dba399bd]
 8: (AdminSocket::do_tell_queue()+0x2e5) [0x7fc6dba3a6d5]
 9: (AdminSocket::entry()+0x31f) [0x7fc6dba3bdef]
 10: (()+0xbd66f) [0x7fc6da30666f]
 11: (()+0x76db) [0x7fc6da7dd6db]
 12: (clone()+0x3f) [0x7fc6d99c388f]

/a/sage-2020-02-25_15:51:04-rados-wip-sage2-testing-2020-02-25-0704-distro-basic-smithi/4801786

Actions #1

Updated by Sage Weil about 4 years ago

  • Project changed from Ceph to RADOS
Actions #2

Updated by Sage Weil about 4 years ago

The command is passed from a nautilus monitor:

  -150> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700  1 -- v1:172.21.15.117:6789/0 <== mon.2 v1:172.21.15.203:6789/0 8 ==== forward(mon_command({"prefix": "smart", "format": "json", "devid": ""} v 0) v1 caps allow profile mgr tid 398 con_features 4540138292836630527) v4 ==== 297+0+0 (unknown 1276046022 0 0) 0x557cd7ac7f80 con 0x557cd6b5b400
  -149> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 mon.b@1(leader) e1 _ms_dispatch existing session 0x557cd6ab8d00 for mon.2
  -148> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 mon.b@1(leader) e1  entity  caps allow *
  -147> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 is_capable service=mon command= read addr v1:172.21.15.203:6789/0 on cap allow *
  -146> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20  allow so far , doing grant allow *
  -145> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20  allow all
  -144> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 10 mon.b@1(leader) e1 received forwarded message from mgr v1:172.21.15.110:0/16681 via mon.2 v1:172.21.15.203:6789/0
  -143> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 is_capable service=mon command= exec addr v1:172.21.15.203:6789/0 on cap allow *
  -142> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20  allow so far , doing grant allow *
  -141> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20  allow all
  -140> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 10 mon.b@1(leader) e1  caps are allow profile mgr
  -139> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 10 mon.b@1(leader) e1  entity name '' type 0
  -138> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 10 mon.b@1(leader) e1  mesg 0x557cd8185e00 from v1:172.21.15.203:6789/0
  -137> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 mon.b@1(leader) e1 _ms_dispatch existing session 0x557cd827f840 for mgr.84111
  -136> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700 20 mon.b@1(leader) e1  entity  caps allow profile mgr
  -135> 2020-02-25T18:14:31.708+0000 7fc6cf1b6700  5 mon.b@1(leader) e1 handle_command passing command to tell/asok

Looks like the new asok/tell behavior, which assumes the command is local, is colliding with the nautilus behavior, which was semi-broken and would occasionally forward these commands around.

Actions #3

Updated by Sage Weil about 4 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 33542
Actions #4

Updated by Sage Weil about 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF