Project

General

Profile

Actions

Bug #53918

closed

librados segfault during AMQP notification tests

Added by Casey Bodley over 2 years ago. Updated about 2 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://qa-proxy.ceph.com/teuthology/cbodley-2022-01-18_13:20:34-rgw-wip-cbodley-testing-distro-default-smithi/6624164/teuthology.log

2022-01-18T14:04:07.503 DEBUG:teuthology.orchestra.run.smithi118:bucket notification tests against different endpoints> BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a amqp_test
2022-01-18T14:04:17.342 INFO:teuthology.orchestra.run.smithi118.stderr:test s3 notification filter on master ... ok
2022-01-18T14:04:19.403 INFO:tasks.rgw.client.0.smithi118.stdout:*** Caught signal (Segmentation fault) **
2022-01-18T14:04:19.404 INFO:tasks.rgw.client.0.smithi118.stdout: in thread 7efbffc98700 thread_name:msgr-worker-1
2022-01-18T14:04:19.405 INFO:tasks.rgw.client.0.smithi118.stdout: ceph version 17.0.0-10248-gdf3b9eef (df3b9eef0cec2688636dd155c474732020406c32) quincy (dev)
2022-01-18T14:04:19.405 INFO:tasks.rgw.client.0.smithi118.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7efc0af02b20]
2022-01-18T14:04:19.406 INFO:tasks.rgw.client.0.smithi118.stdout: 2: /lib64/librados.so.2(+0xfb9b6) [0x7efc0d2319b6]
2022-01-18T14:04:19.406 INFO:tasks.rgw.client.0.smithi118.stdout: 3: /lib64/librados.so.2(+0xfc9ab) [0x7efc0d2329ab]
2022-01-18T14:04:19.406 INFO:tasks.rgw.client.0.smithi118.stdout: 4: /lib64/librados.so.2(+0x101b66) [0x7efc0d237b66]
2022-01-18T14:04:19.406 INFO:tasks.rgw.client.0.smithi118.stdout: 5: (DispatchQueue::fast_dispatch(boost::intrusive_ptr<Message> const&)+0x1a0) [0x7efc0bafc470]
2022-01-18T14:04:19.407 INFO:tasks.rgw.client.0.smithi118.stdout: 6: (ProtocolV2::handle_message()+0x12c6) [0x7efc0bbe4996]
2022-01-18T14:04:19.407 INFO:tasks.rgw.client.0.smithi118.stdout: 7: (ProtocolV2::handle_read_frame_dispatch()+0x238) [0x7efc0bbf7278]
2022-01-18T14:04:19.407 INFO:tasks.rgw.client.0.smithi118.stdout: 8: (ProtocolV2::_handle_read_frame_epilogue_main()+0x85) [0x7efc0bbf7365]
2022-01-18T14:04:19.407 INFO:tasks.rgw.client.0.smithi118.stdout: 9: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer>&&, int)+0x204) [0x7efc0bbf8914]
2022-01-18T14:04:19.407 INFO:tasks.rgw.client.0.smithi118.stdout: 10: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x3c) [0x7efc0bbe008c]
2022-01-18T14:04:19.408 INFO:tasks.rgw.client.0.smithi118.stdout: 11: (AsyncConnection::process()+0x789) [0x7efc0bba6749]
2022-01-18T14:04:19.408 INFO:tasks.rgw.client.0.smithi118.stdout: 12: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x1547) [0x7efc0bc065d7]
2022-01-18T14:04:19.408 INFO:tasks.rgw.client.0.smithi118.stdout: 13: /usr/lib64/ceph/libceph-common.so.2(+0x61809e) [0x7efc0bc0e09e]
2022-01-18T14:04:19.409 INFO:tasks.rgw.client.0.smithi118.stdout: 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7efc09f2bba3]
2022-01-18T14:04:19.409 INFO:tasks.rgw.client.0.smithi118.stdout: 15: /lib64/libpthread.so.0(+0x814a) [0x7efc0aef814a]
2022-01-18T14:04:19.409 INFO:tasks.rgw.client.0.smithi118.stdout: 16: clone()

unfortunately we don't have symbols for librados.so, and radosgw is running with debug_ms=0/0 so even the crash dump doesn't contain messenger logs

if it continues to reproduce, i'll try to collect logs and reassign accordingly

Actions #2

Updated by Casey Bodley about 2 years ago

  • Status changed from New to Need More Info
Actions #3

Updated by Casey Bodley about 2 years ago

  • Status changed from Need More Info to Can't reproduce

it's been a month, i don't think we've seen it again

Actions

Also available in: Atom PDF