Bug #38577
Messenger/MessengerTest.MissingServerIdenTest2/0 msg/async/AsyncMessenger.cc: 312: FAILED ceph_assert(!did_bind)
0%
Description
2019-03-04T22:47:10.446 INFO:teuthology.orchestra.run.smithi193.stdout:[ RUN ] Messenger/MessengerTest.MissingServerIdenTest2/0 ... 2019-03-04T22:47:10.556 INFO:teuthology.orchestra.run.smithi193.stderr:2019-03-04 22:47:10.114 7f75c184f700 -1 ceph_test_msgr intercept conn(0x1c5ba80) intercept called on step=14 2019-03-04T22:47:10.556 INFO:teuthology.orchestra.run.smithi193.stderr:2019-03-04 22:47:10.114 7f75c184f700 -1 ceph_test_msgr intercept conn(0x1c5ba80) resuming step=14 with decision=0 2019-03-04T22:47:10.556 INFO:teuthology.orchestra.run.smithi193.stderr:2019-03-04 22:47:10.118 7f75c084d700 -1 ceph_test_msgr ms_handle_fast_connect 0x1c5ad00 2019-03-04T22:47:10.557 INFO:teuthology.orchestra.run.smithi193.stderr:2019-03-04 22:47:10.118 7f75c184f700 -1 ceph_test_msgr intercept conn(0x1c5ba80) intercept called on step=15 2019-03-04T22:47:10.557 INFO:teuthology.orchestra.run.smithi193.stderr:2019-03-04 22:47:10.118 7f75c184f700 -1 ceph_test_msgr intercept conn(0x1c5ba80) resuming step=15 with decision=0 2019-03-04T22:47:10.557 INFO:teuthology.orchestra.run.smithi193.stderr:2019-03-04 22:47:10.118 7f75c084d700 -1 ceph_test_msgr intercept conn(0x1c5ad00) intercept called on step=15 2019-03-04T22:47:10.557 INFO:teuthology.orchestra.run.smithi193.stderr:2019-03-04 22:47:10.118 7f75c084d700 -1 ceph_test_msgr intercept conn(0x1c5ad00) pausing on step=15 2019-03-04T22:47:10.557 INFO:teuthology.orchestra.run.smithi193.stderr:/build/ceph-14.1.0-354-g56a9456/src/msg/async/AsyncMessenger.cc: In function 'virtual AsyncMessenger::~AsyncMessenger()' thread 7f75cfdbcc80 time 2019-03-04 22:47:10.122483 2019-03-04T22:47:10.557 INFO:teuthology.orchestra.run.smithi193.stderr:/build/ceph-14.1.0-354-g56a9456/src/msg/async/AsyncMessenger.cc: 312: FAILED ceph_assert(!did_bind) 2019-03-04T22:47:10.557 INFO:teuthology.orchestra.run.smithi193.stderr: ceph version 14.1.0-354-g56a9456 (56a9456b13e8fb2c6ac1e7abb4becfff314c64e5) nautilus (dev) 2019-03-04T22:47:10.557 INFO:teuthology.orchestra.run.smithi193.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f75c6a74b62] 2019-03-04T22:47:10.558 INFO:teuthology.orchestra.run.smithi193.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f75c6a74d3d] 2019-03-04T22:47:10.558 INFO:teuthology.orchestra.run.smithi193.stderr: 3: (AsyncMessenger::~AsyncMessenger()+0x24d) [0x7f75c6d5343d] 2019-03-04T22:47:10.558 INFO:teuthology.orchestra.run.smithi193.stderr: 4: (AsyncMessenger::~AsyncMessenger()+0x9) [0x7f75c6d53469] 2019-03-04T22:47:10.558 INFO:teuthology.orchestra.run.smithi193.stderr: 5: (MessengerTest::TearDown()+0xdc) [0x462cbc] 2019-03-04T22:47:10.558 INFO:teuthology.orchestra.run.smithi193.stderr: 6: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x4a) [0x4a0f1a] 2019-03-04T22:47:10.558 INFO:teuthology.orchestra.run.smithi193.stderr: 7: (testing::TestInfo::Run()+0xaf) [0x49508f] 2019-03-04T22:47:10.558 INFO:teuthology.orchestra.run.smithi193.stderr: 8: (testing::TestCase::Run()+0xb5) [0x4951c5] 2019-03-04T22:47:10.559 INFO:teuthology.orchestra.run.smithi193.stderr: 9: (testing::internal::UnitTestImpl::RunAllTests()+0x468) [0x4956b8] 2019-03-04T22:47:10.559 INFO:teuthology.orchestra.run.smithi193.stderr: 10: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x4a) [0x4a13da] 2019-03-04T22:47:10.559 INFO:teuthology.orchestra.run.smithi193.stderr: 11: (testing::UnitTest::Run()+0x5d) [0x49583d] 2019-03-04T22:47:10.559 INFO:teuthology.orchestra.run.smithi193.stderr: 12: (main()+0x394) [0x440124] 2019-03-04T22:47:10.559 INFO:teuthology.orchestra.run.smithi193.stderr: 13: (__libc_start_main()+0xf0) [0x7f75c5fde830] 2019-03-04T22:47:10.559 INFO:teuthology.orchestra.run.smithi193.stderr: 14: (_start()+0x29) [0x440c29] 2019-03-04T22:47:10.559 INFO:teuthology.orchestra.run.smithi193.stderr:2019-03-04 22:47:10.118 7f75cfdbcc80 -1 /build/ceph-14.1.0-354-g56a9456/src/msg/async/AsyncMessenger.cc: In function 'virtual AsyncMessenger::~AsyncMessenger()' thread 7f75cfdbcc80 time 2019-03-04 22:47:10.122483 2019-03-04T22:47:10.560 INFO:teuthology.orchestra.run.smithi193.stderr:/build/ceph-14.1.0-354-g56a9456/src/msg/async/AsyncMessenger.cc: 312: FAILED ceph_assert(!did_bind) 2019-03-04T22:47:10.560 INFO:teuthology.orchestra.run.smithi193.stderr: 2019-03-04T22:47:10.560 INFO:teuthology.orchestra.run.smithi193.stderr: ceph version 14.1.0-354-g56a9456 (56a9456b13e8fb2c6ac1e7abb4becfff314c64e5) nautilus (dev) 2019-03-04T22:47:10.560 INFO:teuthology.orchestra.run.smithi193.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f75c6a74b62] 2019-03-04T22:47:10.560 INFO:teuthology.orchestra.run.smithi193.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f75c6a74d3d] 2019-03-04T22:47:10.560 INFO:teuthology.orchestra.run.smithi193.stderr: 3: (AsyncMessenger::~AsyncMessenger()+0x24d) [0x7f75c6d5343d] 2019-03-04T22:47:10.560 INFO:teuthology.orchestra.run.smithi193.stderr: 4: (AsyncMessenger::~AsyncMessenger()+0x9) [0x7f75c6d53469] 2019-03-04T22:47:10.560 INFO:teuthology.orchestra.run.smithi193.stderr: 5: (MessengerTest::TearDown()+0xdc) [0x462cbc] 2019-03-04T22:47:10.561 INFO:teuthology.orchestra.run.smithi193.stderr: 6: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x4a) [0x4a0f1a] 2019-03-04T22:47:10.561 INFO:teuthology.orchestra.run.smithi193.stderr: 7: (testing::TestInfo::Run()+0xaf) [0x49508f] 2019-03-04T22:47:10.561 INFO:teuthology.orchestra.run.smithi193.stderr: 8: (testing::TestCase::Run()+0xb5) [0x4951c5] 2019-03-04T22:47:10.561 INFO:teuthology.orchestra.run.smithi193.stderr: 9: (testing::internal::UnitTestImpl::RunAllTests()+0x468) [0x4956b8]
/a/sage-2019-03-04_18:38:36-rados-wip-sage2-testing-2019-03-04-1033-distro-basic-smithi/3668604
Related issues
History
#1 Updated by Greg Farnum over 4 years ago
- Project changed from RADOS to Messengers
#2 Updated by Greg Farnum over 4 years ago
- Category set to AsyncMessenger
#3 Updated by Sage Weil over 4 years ago
/a/sage-2019-03-22_15:28:27-rados-wip-sage4-testing-2019-03-22-0507-distro-basic-smithi/3761317
#4 Updated by Neha Ojha over 4 years ago
/a/dzafman-2019-03-25_19:13:04-rados-wip-zafman-testing-distro-basic-smithi/3773531/
#5 Updated by Neha Ojha over 4 years ago
/a/yuriw-2019-06-21_18:33:43-rados-wip-yuri3-testing-2019-06-20-2323-nautilus-distro-basic-smithi/4057967/
#6 Updated by Sage Weil over 4 years ago
/a/sage-2019-07-23_18:50:19-rados-wip-sage-testing-2019-07-23-0925-distro-basic-smithi/4142348
#7 Updated by Sage Weil over 4 years ago
/a/sage-2019-08-22_14:28:32-rados-wip-sage2-testing-2019-08-21-1708-distro-basic-smithi/4241121
#8 Updated by Sage Weil over 4 years ago
- Duplicated by Bug #41044: test/msgr/test_msgr.cc:794: Failure added
#9 Updated by Patrick Donnelly about 4 years ago
- Status changed from 12 to New
#10 Updated by Sage Weil almost 4 years ago
- Status changed from New to Can't reproduce
#11 Updated by Sage Weil over 3 years ago
- Status changed from Can't reproduce to In Progress
2020-03-11T16:21:07.866 INFO:teuthology.orchestra.run.smithi151.stderr: -12> 2020-03-11T16:21:07.612+0000 7fcdf4e7d700 -1 ceph_test_msgr ms_handle_reset 0x5644693e2800 2020-03-11T16:21:07.866 INFO:teuthology.orchestra.run.smithi151.stderr: -11> 2020-03-11T16:21:07.612+0000 7fcdf6680700 -1 ceph_test_msgr intercept conn(0x5644693e2400) intercept called on step=13 2020-03-11T16:21:07.866 INFO:teuthology.orchestra.run.smithi151.stderr: -10> 2020-03-11T16:21:07.612+0000 7fcdf6680700 -1 ceph_test_msgr intercept conn(0x5644693e2400) resuming step=13 with decision=0 2020-03-11T16:21:07.866 INFO:teuthology.orchestra.run.smithi151.stderr: -9> 2020-03-11T16:21:07.612+0000 7fcdf3e7b700 -1 ceph_test_msgr ms_handle_reset 0x5644693e1c00 2020-03-11T16:21:07.867 INFO:teuthology.orchestra.run.smithi151.stderr: -8> 2020-03-11T16:21:07.612+0000 7fcdf567e700 -1 ceph_test_msgr intercept conn(0x5644693e1000) intercept called on step=14 2020-03-11T16:21:07.867 INFO:teuthology.orchestra.run.smithi151.stderr: -7> 2020-03-11T16:21:07.612+0000 7fcdf567e700 -1 ceph_test_msgr intercept conn(0x5644693e1000) resuming step=14 with decision=0 2020-03-11T16:21:07.867 INFO:teuthology.orchestra.run.smithi151.stderr: -6> 2020-03-11T16:21:07.612+0000 7fcdf567e700 -1 ceph_test_msgr intercept conn(0x5644693e1000) intercept called on step=15 2020-03-11T16:21:07.867 INFO:teuthology.orchestra.run.smithi151.stderr: -5> 2020-03-11T16:21:07.612+0000 7fcdf567e700 -1 ceph_test_msgr intercept conn(0x5644693e1000) resuming step=15 with decision=0 2020-03-11T16:21:07.867 INFO:teuthology.orchestra.run.smithi151.stderr: -4> 2020-03-11T16:21:07.612+0000 7fcdf6680700 -1 ceph_test_msgr ms_handle_fast_connect 0x5644693e2400 2020-03-11T16:21:07.867 INFO:teuthology.orchestra.run.smithi151.stderr: -3> 2020-03-11T16:21:07.612+0000 7fcdf6680700 -1 ceph_test_msgr intercept conn(0x5644693e2400) intercept called on step=15 2020-03-11T16:21:07.867 INFO:teuthology.orchestra.run.smithi151.stderr: -2> 2020-03-11T16:21:07.612+0000 7fcdf6680700 -1 ceph_test_msgr intercept conn(0x5644693e2400) pausing on step=15 2020-03-11T16:21:07.867 INFO:teuthology.orchestra.run.smithi151.stderr: -1> 2020-03-11T16:21:07.620+0000 7fce043dc180 -1 /build/ceph-15.1.0-2019-geb40947/src/msg/async/AsyncMessenger.cc: In function 'virtual AsyncMessenger::~AsyncMessenger()' thread 7fce043dc180 time 2020-03-11T16:21:07.615278+0000 2020-03-11T16:21:07.868 INFO:teuthology.orchestra.run.smithi151.stderr:/build/ceph-15.1.0-2019-geb40947/src/msg/async/AsyncMessenger.cc: 316: FAILED ceph_assert(!did_bind) 2020-03-11T16:21:07.868 INFO:teuthology.orchestra.run.smithi151.stderr: 2020-03-11T16:21:07.868 INFO:teuthology.orchestra.run.smithi151.stderr: ceph version 15.1.0-2019-geb40947 (eb40947502c881503d30adc3440afd9b2204753e) octopus (rc) 2020-03-11T16:21:07.868 INFO:teuthology.orchestra.run.smithi151.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7fcdfa8a5c70] 2020-03-11T16:21:07.868 INFO:teuthology.orchestra.run.smithi151.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fcdfa8a5e4b] 2020-03-11T16:21:07.868 INFO:teuthology.orchestra.run.smithi151.stderr: 3: (()+0x4fcb55) [0x7fcdfab39b55] 2020-03-11T16:21:07.868 INFO:teuthology.orchestra.run.smithi151.stderr: 4: (AsyncMessenger::~AsyncMessenger()+0x9) [0x7fcdfab39b69] 2020-03-11T16:21:07.869 INFO:teuthology.orchestra.run.smithi151.stderr: 5: (MessengerTest::TearDown()+0xd6) [0x564466953766] 2020-03-11T16:21:07.869 INFO:teuthology.orchestra.run.smithi151.stderr: 6: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x4a) [0x564466998faa] 2020-03-11T16:21:07.869 INFO:teuthology.orchestra.run.smithi151.stderr: 7: (testing::TestInfo::Run()+0x135) [0x56446698e515] 2020-03-11T16:21:07.869 INFO:teuthology.orchestra.run.smithi151.stderr: 8: (testing::TestSuite::Run()+0xb5) [0x56446698e5f5] 2020-03-11T16:21:07.869 INFO:teuthology.orchestra.run.smithi151.stderr: 9: (testing::internal::UnitTestImpl::RunAllTests()+0x488) [0x56446698eb18] 2020-03-11T16:21:07.869 INFO:teuthology.orchestra.run.smithi151.stderr: 10: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x4a) [0x5644669994ba] 2020-03-11T16:21:07.869 INFO:teuthology.orchestra.run.smithi151.stderr: 11: (testing::UnitTest::Run()+0x98) [0x56446698ed28] 2020-03-11T16:21:07.870 INFO:teuthology.orchestra.run.smithi151.stderr: 12: (main()+0x32f) [0x5644669325cf] 2020-03-11T16:21:07.870 INFO:teuthology.orchestra.run.smithi151.stderr: 13: (__libc_start_main()+0xe7) [0x7fcdf9522b97] 2020-03-11T16:21:07.870 INFO:teuthology.orchestra.run.smithi151.stderr: 14: (_start()+0x2a) [0x5644669333ea] 2020-03-11T16:21:07.870 INFO:teuthology.orchestra.run.smithi151.stderr: 2020-03-11T16:21:07.870 INFO:teuthology.orchestra.run.smithi151.stderr: 0> 2020-03-11T16:21:07.624+0000 7fce043dc180 -1 *** Caught signal (Aborted) **
/a/sage-2020-03-11_14:29:21-rados-wip-sage-testing-2020-03-11-0719-distro-basic-smithi/4847254
#12 Updated by Neha Ojha over 3 years ago
Seen in nautilus: /a/yuriw-2020-04-03_18:09:11-rados-wip-yuri-testing-2020-04-01-2252-nautilus-distro-basic-smithi/4923025/
#13 Updated by Neha Ojha over 3 years ago
/a/yuriw-2020-05-21_22:41:21-rados-wip-yuri2-testing-2020-05-21-2005-nautilus-distro-basic-smithi/5078950
#14 Updated by Neha Ojha over 3 years ago
- Backport set to nautilus,octopus
/a/yuriw-2020-07-13_17:52:35-rados-wip-yuri3-testing-2020-07-13-1505-nautilus-distro-basic-smithi/5222691
#15 Updated by Laura Flores over 1 year ago
- Backport changed from nautilus,octopus to nautilus,octopus,pacific
Checked the Telemetry Spec Search page (http://telemetry.front.sepia.ceph.com:4000/d/Nvj6XTaMk/spec-search?orgId=1) and didn't find any evidence of this crash being reported by clusters who have opted into Telemetry, so it's possible that this is just a test issue. Either that, or a very rare issue.
/a/yuriw-2022-08-01_22:58:38-rados-wip-yuri8-testing-2022-08-01-1413-pacific-distro-default-smithi/6955109
2022-08-02T02:30:02.972 INFO:teuthology.orchestra.run.smithi089.stdout:[ RUN ] Messenger/MessengerTest.ReconnectRaceTest/0
...
2022-08-02T02:30:03.015 INFO:teuthology.orchestra.run.smithi089.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10-531-g7061d0e7/rpm/el8/BUILD/ceph-16.2.10-531-g7061d0e7/src/test/msgr/test_msgr.cc:873: Failure
2022-08-02T02:30:03.016 INFO:teuthology.orchestra.run.smithi089.stdout:Expected equality of these values:
2022-08-02T02:30:03.016 INFO:teuthology.orchestra.run.smithi089.stdout: cli_interceptor->count_step(c2s.get(), Interceptor::STEP::SESSION_RETRY)
2022-08-02T02:30:03.016 INFO:teuthology.orchestra.run.smithi089.stdout: Which is: 0
2022-08-02T02:30:03.016 INFO:teuthology.orchestra.run.smithi089.stdout: 1u
2022-08-02T02:30:03.016 INFO:teuthology.orchestra.run.smithi089.stdout: Which is: 1
2022-08-02T02:30:03.054 DEBUG:teuthology.orchestra.run:got remote process result: None
2022-08-02T02:30:03.055 INFO:teuthology.orchestra.run.smithi089.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10-531-g7061d0e7/rpm/el8/BUILD/ceph-16.2.10-531-g7061d0e7/src/msg/async/AsyncMessenger.cc: In function 'virtual AsyncMessenger::~AsyncMessenger()' thread 7fb2ad001fc0 time 2022-08-02T02:30:02.929434+0000
2022-08-02T02:30:03.055 INFO:teuthology.orchestra.run.smithi089.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10-531-g7061d0e7/rpm/el8/BUILD/ceph-16.2.10-531-g7061d0e7/src/msg/async/AsyncMessenger.cc: 316: FAILED ceph_assert(!did_bind)
2022-08-02T02:30:03.055 INFO:teuthology.orchestra.run.smithi089.stderr: ceph version 16.2.10-531-g7061d0e7 (7061d0e7ffd836dc9c76dfa4c41c6ba60edca507) pacific (stable)
2022-08-02T02:30:03.056 INFO:teuthology.orchestra.run.smithi089.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fb2a25c6c68]
2022-08-02T02:30:03.056 INFO:teuthology.orchestra.run.smithi089.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x277e82) [0x7fb2a25c6e82]
2022-08-02T02:30:03.056 INFO:teuthology.orchestra.run.smithi089.stderr: 3: /usr/lib64/ceph/libceph-common.so.2(+0x570401) [0x7fb2a28bf401]
2022-08-02T02:30:03.057 INFO:teuthology.orchestra.run.smithi089.stderr: 4: (AsyncMessenger::~AsyncMessenger()+0xd) [0x7fb2a28bf41d]
2022-08-02T02:30:03.057 INFO:teuthology.orchestra.run.smithi089.stderr: 5: (MessengerTest::TearDown()+0xdc) [0x55c141ce5e2c]
2022-08-02T02:30:03.057 INFO:teuthology.orchestra.run.smithi089.stderr: 6: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x4e) [0x55c141d2dd7e]
2022-08-02T02:30:03.057 INFO:teuthology.orchestra.run.smithi089.stderr: 7: (testing::TestInfo::Run()+0x135) [0x55c141d20b15]
2022-08-02T02:30:03.058 INFO:teuthology.orchestra.run.smithi089.stderr: 8: (testing::TestSuite::Run()+0xc1) [0x55c141d21071]
2022-08-02T02:30:03.058 INFO:teuthology.orchestra.run.smithi089.stderr: 9: (testing::internal::UnitTestImpl::RunAllTests()+0x445) [0x55c141d22c85]
2022-08-02T02:30:03.058 INFO:teuthology.orchestra.run.smithi089.stderr: 10: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x4e) [0x55c141d2e2ee]
#16 Updated by Leonid Usov 4 months ago
- Related to Bug #62395: AsyncMessenger::wait() isn't checking for spurious condition wakeup added
#17 Updated by Leonid Usov 4 months ago
This is most probably caused by https://tracker.ceph.com/issues/62395, the fix is trivial and awaiting review
#18 Updated by Aishwarya Mathuria 4 months ago
/a/yuriw-2023-08-16_22:40:18-rados-wip-yuri2-testing-2023-08-16-1142-pacific-distro-default-smithi/7371026
#19 Updated by Leonid Usov 2 months ago
- Status changed from In Progress to Resolved