Project

General

Profile

Bug #51114

TestAMQP.ClosedConnection failing in master

Added by Kefu Chai 4 months ago. Updated 4 months ago.

Status:
Pending Backport
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Tags:
amqp
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

[ RUN      ] TestAMQP.ClosedConnection
../src/rgw/rgw_amqp.cc: In function 'rgw::amqp::connection_ptr_t& rgw::amqp::create_connection(rgw::amqp::connection_ptr_t&, const amqp_connection_info&)' thread 7f16c27c4700 time 2021-06-07T02:49:39.821685+0000
../src/rgw/rgw_amqp.cc: 391: FAILED ceph_assert(conn && !conn->marked_for_deletion)
 ceph version Development (no_version) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a1) [0x7f16c55feca3]
 2: /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libceph-common.so.2(+0x1697f41) [0x7f16c55fef41]
 3: (rgw::amqp::create_connection(boost::intrusive_ptr<rgw::amqp::connection_t>&, amqp_connection_info const&)+0x6a) [0x561b553c53c7]
 4: (rgw::amqp::Manager::run()+0x543) [0x561b553c912d]
 5: (void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&)+0x6a) [0x561b553db7ce]
 6: (std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&)+0x3b) [0x561b553db6fc]
 7: (void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>)+0x47) [0x561b553db65b]
 8: (std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()()+0x1c) [0x561b553db5e4]
 9: (std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run()+0x20) [0x561b553da57a]
 10: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6d84) [0x7f16c3e5cd84]
 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f16c6690609]
 12: clone()

Related issues

Copied to rgw - Backport #51351: pacific: TestAMQP.ClosedConnection failing in master New

History

#1 Updated by Kefu Chai 4 months ago

Yuval, would you kindly take a look?

#2 Updated by Yuval Lifshitz 4 months ago

  • Status changed from New to In Progress

#3 Updated by Yuval Lifshitz 4 months ago

  • Tags set to amqp
this issue indicates a real race condition that may happen when deleting a connection explicitly using the "disconnect()" API.
note that this API is currently only used in the unit tests. however, a different deletion mechanism is needed to cleanup idle connections, as described here:https://tracker.ceph.com/issues/49033.
so in this issue:
  • remove the "disconnect()" API
  • add a time based idle connection cleanup mechanism

#4 Updated by Yuval Lifshitz 4 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 41831

#5 Updated by Casey Bodley 4 months ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to pacific

#6 Updated by Backport Bot 4 months ago

  • Copied to Backport #51351: pacific: TestAMQP.ClosedConnection failing in master added

Also available in: Atom PDF