Project

General

Profile

Bug #18664

ceph_test_msgr SIGABRT in decode_message, SimpleMessenger, can't decode unknown message type 0 MSG_AUTH=17

Added by Sage Weil about 7 years ago. Updated over 4 years ago.

Status:
New
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-01-24T20:43:20.389 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.937764 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd6b5c4f00 send m=0x7fbd368f21c0 i=680
2017-01-24T20:43:20.389 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.937922 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd6b5c7100 send m=0x7fbd291a5d40 i=681
2017-01-24T20:43:20.389 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.938034 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd29545d00 send m=0x7fbd291a52c0 i=682
2017-01-24T20:43:20.389 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.938053 7fbd27a55f00 -1  ceph_test_msgr Op 1500:
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.938057 7fbd27a55f00 -1  ceph_test_msgr available_connections: 15 inflight messages: 12
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.938155 7fbcb4e4a700 -1  ceph_test_msgr ms_fast_dispatch conn=0x7fbd6b5c7d00reply=^A i = 666
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.939210 7fbcb3e3a700  0 -- 127.0.0.1:16814/12764 >> 127.0.0.1:16838/12788 pipe(0x7fbd6119e000 sd=172 :16814 s=0 pgs=0 cs=0 l=0 c=0x7fbd6b5c7800).accept connect_seq 0 vs existing 0 state connecting
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.939573 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd29545200 send m=0x7fbd291a5100 i=683
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.939610 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd29542d00 send m=0x7fbd291a5800 i=684
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.939628 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd29542a00 send m=0x7fbd291a5640 i=685
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.940875 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd29542200 send m=0x7fbd291a4f40 i=686
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.940898 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd29545d00 send m=0x7fbd291a4a00 i=687
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.940975 7fbcb7c78700  0 can't decode unknown message type 0 MSG_AUTH=17
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941004 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd29543d00 send m=0x7fbd291a4140 i=688
2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941096 7fbd27a55f00 -1  ceph_test_msgr Op 1510:
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941099 7fbd27a55f00 -1  ceph_test_msgr available_connections: 14 inflight messages: 13
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941109 7fbd27a55f00 -1  ceph_test_msgr send_message_wrap conn=0x7fbd6b5c4f00 send m=0x7fbd291a28c0 i=689
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941949 7fbcb7c78700 -1 *** Caught signal (Aborted) **
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: in thread 7fbcb7c78700 thread_name:ms_pipe_read
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr:
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: ceph version 11.1.0-6995-gf750cad (f750cadc8d0d0816a64e48339f9bd2dce7b2034a)
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 1: (()+0x5ab9f) [0x7fbd27acfb9f]
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 2: (()+0xf370) [0x7fbd27204370]
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 3: (gsignal()+0x37) [0x7fbd1b80b1d7]
2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 4: (abort()+0x148) [0x7fbd1b80c8c8]
2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 5: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&)+0x18a7) [0x7fbd1dcfd157]
2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 6: (Pipe::read_message(Message**, AuthSessionHandler*)+0x1441) [0x7fbd1dd6a4e1]
2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 7: (Pipe::reader()+0xbb1) [0x7fbd1dd76dd1]
2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 8: (Pipe::Reader::entry()+0xd) [0x7fbd1dd80c6d]
2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 9: (()+0x7dc5) [0x7fbd271fcdc5]
2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 10: (clone()+0x6d) [0x7fbd1b8cd73d]
2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

/a/sage-2017-01-24_19:52:02-rados-wip-sage-testing---basic-smithi/743886

Related issues

Duplicated by Messengers - Bug #20361: ceph_test_msgr: can't decode unknown message type 0 MSG_AUTH=17 Duplicate 06/20/2017

History

#1 Updated by Sage Weil about 7 years ago

  • Subject changed from ceph_test_msgr SIGABRT in decode_message, can't decode unknown message type 0 MSG_AUTH=17 to ceph_test_msgr SIGABRT in decode_message, SimpleMessenger, can't decode unknown message type 0 MSG_AUTH=17

#2 Updated by Greg Farnum almost 7 years ago

  • Priority changed from Urgent to Normal

Apparently this hasn't reproduced?

#3 Updated by Greg Farnum almost 7 years ago

  • Duplicated by Bug #20361: ceph_test_msgr: can't decode unknown message type 0 MSG_AUTH=17 added

#4 Updated by Greg Farnum almost 7 years ago

  • Priority changed from Normal to High

/a/sage-2017-06-20_17:41:31-rados-wip-sage-testing-distro-basic-smithi/1306218

#5 Updated by Sage Weil over 6 years ago

  • Status changed from New to Can't reproduce

#6 Updated by Sage Weil over 6 years ago

  • Status changed from Can't reproduce to 12

/a/sage-2017-09-27_21:32:47-rados-wip-sage-testing-2017-09-27-1558-distro-basic-smithi/1680113

#7 Updated by Sage Weil about 6 years ago

/a/sage-2018-01-09_00:16:55-rados-wip-sage-testing-2018-01-08-1631-distro-basic-smithi/2044757

#8 Updated by Neha Ojha over 5 years ago

Similar failure seen in luminous:


2018-10-23T16:55:17.223 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.224198 7f14439c83c0 -1  ceph_test_msgr send_message_wrap conn=0x55772152fc20 send m=0x557750c74f00 i=889
2018-10-23T16:55:17.224 INFO:teuthology.orchestra.run.smithi193.stderr:*** Caught signal (Aborted) **
2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr: in thread 7f13d0321700 thread_name:ms_pipe_read
2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225272 7f13d0321700  0 can't decode unknown message type 0 MSG_AUTH=17
2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225315 7f13d405e700  0 -- 127.0.0.1:16833/11971 >> 127.0.0.1:16801/11939 pipe(0x557721db0800 sd=193 :16833 s=2 pgs=15 cs=1 l=0 c=0x55772152e7e0).fault with nothing to send, going to standby
2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225434 7f14439c83c0 -1  ceph_test_msgr Op 1810:
2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225443 7f14439c83c0 -1  ceph_test_msgr available_connections: 16 inflight messages: 15
2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225497 7f14439c83c0 -1  ceph_test_msgr send_message_wrap conn=0x557732f0bc20 send m=0x557750c76760 i=890
2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225712 7f13d203e700 -1  ceph_test_msgr ms_fast_dispatch conn=0x55773f507560reply=
2018-10-23T16:55:17.243 INFO:teuthology.orchestra.run.smithi193.stderr: ceph version 12.2.8-504-g943c4b5 (943c4b5a87c3f117458b5b6297beadde2a785ccf) luminous (stable)
2018-10-23T16:55:17.243 INFO:teuthology.orchestra.run.smithi193.stderr: 1: (()+0x58a54) [0x55771ecdfa54]
2018-10-23T16:55:17.243 INFO:teuthology.orchestra.run.smithi193.stderr: 2: (()+0x11390) [0x7f14435b4390]
2018-10-23T16:55:17.243 INFO:teuthology.orchestra.run.smithi193.stderr: 3: (gsignal()+0x38) [0x7f1439ade428]
2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 4: (abort()+0x16a) [0x7f1439ae002a]
2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 5: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x27d1) [0x7f143a922bf1]
2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 6: (Pipe::read_message(Message**, AuthSessionHandler*)+0x187c) [0x7f143a9f23cc]
2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 7: (Pipe::reader()+0x1170) [0x7f143a9fea50]
2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 8: (Pipe::Reader::entry()+0xd) [0x7f143aa08fed]
2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 9: (()+0x76ba) [0x7f14435aa6ba]
2018-10-23T16:55:17.245 INFO:teuthology.orchestra.run.smithi193.stderr: 10: (clone()+0x6d) [0x7f1439bb041d]
2018-10-23T16:55:17.245 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.226173 7f13d0321700 -1 *** Caught signal (Aborted) **
2018-10-23T16:55:17.245 INFO:teuthology.orchestra.run.smithi193.stderr: in thread 7f13d0321700 thread_name:ms_pipe_read

/a/yuriw-2018-10-23_15:16:03-rados-wip-yuri-testing-2018-10-22-2055-luminous-distro-basic-smithi/3175376/

#9 Updated by Neha Ojha over 4 years ago

  • Project changed from Ceph to Messengers

/a/nojha-2019-08-26_20:27:46-rados-wip-bluefs-shared-alloc-luminous-2019-08-26-distro-basic-smithi/4255419/

#10 Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New

Also available in: Atom PDF