Ceph

probably https://github.com/ceph/ceph/pull/14578 could fix this.

maybe this is related (used message after being queued for send?)

2017-04-19T17:46:40.298 INFO:tasks.ceph.mgr.x.smithi179.stderr:*** Caught signal (Segmentation fault) **
2017-04-19T17:46:40.298 INFO:tasks.ceph.mgr.x.smithi179.stderr: in thread 7f4f18859700 thread_name:ms_pipe_write
2017-04-19T17:46:40.299 INFO:tasks.workunit.client.0.smithi116.stdout:op 5316 completed, throughput=5MB/sec
2017-04-19T17:46:40.299 INFO:tasks.workunit.client.0.smithi116.stdout:READ : oid=obj-exX3wrorcH28BIi off=348076 len=181124
2017-04-19T17:46:40.299 INFO:tasks.ceph.mgr.x.smithi179.stderr: ceph version 12.0.0-2827-g8f6cef9 (8f6cef90871eec5261645a47bc4090c7ffb3a1e4)
2017-04-19T17:46:40.300 INFO:tasks.ceph.mgr.x.smithi179.stderr: 1: (()+0x296a97) [0x7f4f266c9a97]
2017-04-19T17:46:40.300 INFO:tasks.ceph.mgr.x.smithi179.stderr: 2: (()+0x10330) [0x7f4f24c90330]
2017-04-19T17:46:40.300 INFO:tasks.ceph.mgr.x.smithi179.stderr: 3: (ceph::buffer::ptr::ptr(ceph::buffer::ptr const&)+0) [0x7f4f266cb5b0]
2017-04-19T17:46:40.300 INFO:tasks.ceph.mgr.x.smithi179.stderr: 4: (Pipe::writer()+0x821) [0x7f4f269257b1]
2017-04-19T17:46:40.300 INFO:tasks.ceph.mgr.x.smithi179.stderr: 5: (Pipe::Writer::entry()+0xd) [0x7f4f26927f4d]
2017-04-19T17:46:40.300 INFO:tasks.ceph.mgr.x.smithi179.stderr: 6: (()+0x8184) [0x7f4f24c88184]
2017-04-19T17:46:40.300 INFO:tasks.ceph.mgr.x.smithi179.stderr: 7: (clone()+0x6d) [0x7f4f23d6e37d]
2017-04-19T17:46:40.300 INFO:tasks.ceph.mgr.x.smithi179.stderr:2017-04-19 17:46:40.265515 7f4f18859700 -1 *** Caught signal (Segmentation fault) **
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: in thread 7f4f18859700 thread_name:ms_pipe_write
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr:
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: ceph version 12.0.0-2827-g8f6cef9 (8f6cef90871eec5261645a47bc4090c7ffb3a1e4)
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: 1: (()+0x296a97) [0x7f4f266c9a97]
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: 2: (()+0x10330) [0x7f4f24c90330]
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: 3: (ceph::buffer::ptr::ptr(ceph::buffer::ptr const&)+0) [0x7f4f266cb5b0]
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: 4: (Pipe::writer()+0x821) [0x7f4f269257b1]
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: 5: (Pipe::Writer::entry()+0xd) [0x7f4f26927f4d]
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: 6: (()+0x8184) [0x7f4f24c88184]
2017-04-19T17:46:40.301 INFO:tasks.ceph.mgr.x.smithi179.stderr: 7: (clone()+0x6d) [0x7f4f23d6e37d]
2017-04-19T17:46:40.302 INFO:tasks.ceph.mgr.x.smithi179.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-04-19T17:46:40.302 INFO:tasks.ceph.mgr.x.smithi179.stderr:

(gdb) bt
#0  0x00007f4f24c901fb in raise (sig=11) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x00007f4f266c9b65 in reraise_fatal (signum=11) at /build/ceph-12.0.0-2827-g8f6cef9/src/global/signal_handler.cc:74
#2  handle_fatal_signal (signum=11) at /build/ceph-12.0.0-2827-g8f6cef9/src/global/signal_handler.cc:138
#3  <signal handler called>
#4  ceph::buffer::ptr::ptr (this=0x7f4f317438b0, p=...) at /build/ceph-12.0.0-2827-g8f6cef9/src/common/buffer.cc:792
#5  0x00007f4f269257b1 in _List_node<ceph::buffer::ptr const&> (this=<optimized out>) at /usr/include/c++/4.8/bits/stl_list.h:114
#6  construct<std::_List_node<ceph::buffer::ptr>, ceph::buffer::ptr const&> (__p=<optimized out>, this=0x7f4f188581b0) at /usr/include/c++/4.8/ext/new_allocator.h:120
#7  _M_create_node<ceph::buffer::ptr const&> (this=0x7f4f188581b0) at /usr/include/c++/4.8/bits/stl_list.h:505
#8  _M_insert<ceph::buffer::ptr const&> (__position=..., this=0x7f4f188581b0) at /usr/include/c++/4.8/bits/stl_list.h:1561
#9  emplace_back<ceph::buffer::ptr const&> (this=0x7f4f188581b0) at /usr/include/c++/4.8/bits/stl_list.h:1026
#10 _M_initialize_dispatch<std::_List_const_iterator<ceph::buffer::ptr> > (__last=..., __first=<error reading variable: Cannot access memory at address 0x10>, this=0x7f4f188581b0) at /usr/include/c++/4.8/bits/stl_list.h:1491
Python Exception <class 'IndexError'> list index out of range: 
#11 list (__x=std::list, this=0x7f4f188581b0) at /usr/include/c++/4.8/bits/stl_list.h:584
#12 list (other=..., this=0x7f4f188581b0) at /build/ceph-12.0.0-2827-g8f6cef9/src/include/buffer.h:661
#13 Pipe::writer (this=0x7f4f31627400) at /build/ceph-12.0.0-2827-g8f6cef9/src/msg/simple/Pipe.cc:1946
#14 0x00007f4f26927f4d in Pipe::Writer::entry (this=<optimized out>) at /build/ceph-12.0.0-2827-g8f6cef9/src/msg/simple/Pipe.h:62
#15 0x00007f4f24c88184 in start_thread (arg=0x7f4f18859700) at pthread_create.c:312
#16 0x00007f4f23d6e37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) f 13
#13 Pipe::writer (this=0x7f4f31627400) at /build/ceph-12.0.0-2827-g8f6cef9/src/msg/simple/Pipe.cc:1946
1946    /build/ceph-12.0.0-2827-g8f6cef9/src/msg/simple/Pipe.cc: No such file or directory.
(gdb) p m
$1 = (Message *) 0x7f4f316c4b40

   -27> 2017-04-19 17:46:40.263555 7f4f1d075700 10 register_pg  will create 0.13 primary 1 acting [1,4]
   -26> 2017-04-19 17:46:40.263558 7f4f1d075700 10 register_pg  will create 0.14 primary 1 acting [1,2]
   -25> 2017-04-19 17:46:40.263561 7f4f1d075700 10 register_pg  will create 0.15 primary 2 acting [2]
   -24> 2017-04-19 17:46:40.263564 7f4f1d075700 10 register_pg  will create 0.16 primary 3 acting [3]
   -23> 2017-04-19 17:46:40.263567 7f4f1d075700 10 register_pg  will create 0.17 primary 3 acting [3,2]
   -22> 2017-04-19 17:46:40.263568 7f4f1d075700 10 register_new_pgs registered 24 new pgs, removed 0 uncreated pgs
   -21> 2017-04-19 17:46:40.263620 7f4f1d075700  4 mgr init waiting for FSMap...
   -20> 2017-04-19 17:46:40.263912 7f4f2087c700  1 -- 172.21.15.179:0/2878241838 <== mon.0 172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (1463616380 0 0) 0x7f4f316c4d80 con 0x7f4f31ad0a00
   -19> 2017-04-19 17:46:40.263931 7f4f2087c700 10 cephx: set_have_need_key no handler for service mon
   -18> 2017-04-19 17:46:40.263932 7f4f2087c700 10 cephx: set_have_need_key no handler for service mds
   -17> 2017-04-19 17:46:40.263932 7f4f2087c700 10 cephx: set_have_need_key no handler for service osd
   -16> 2017-04-19 17:46:40.263933 7f4f2087c700 10 cephx: set_have_need_key no handler for service mgr
   -15> 2017-04-19 17:46:40.263933 7f4f2087c700 10 cephx: set_have_need_key no handler for service auth
   -14> 2017-04-19 17:46:40.263934 7f4f2087c700 10 cephx: validate_tickets want 55 have 0 need 55
   -13> 2017-04-19 17:46:40.263935 7f4f2087c700 10 cephx client: handle_response ret = 0
   -12> 2017-04-19 17:46:40.263936 7f4f2087c700 10 cephx client:  got initial server challenge af98ba89ba0181b4
   -11> 2017-04-19 17:46:40.263938 7f4f2087c700 10 cephx client: validate_tickets: want=55 need=55 have=0
   -10> 2017-04-19 17:46:40.263938 7f4f2087c700 10 cephx: set_have_need_key no handler for service mon
    -9> 2017-04-19 17:46:40.263939 7f4f2087c700 10 cephx: set_have_need_key no handler for service mds
    -8> 2017-04-19 17:46:40.263939 7f4f2087c700 10 cephx: set_have_need_key no handler for service osd
    -7> 2017-04-19 17:46:40.263940 7f4f2087c700 10 cephx: set_have_need_key no handler for service mgr
    -6> 2017-04-19 17:46:40.263940 7f4f2087c700 10 cephx: set_have_need_key no handler for service auth
    -5> 2017-04-19 17:46:40.263941 7f4f2087c700 10 cephx: validate_tickets want 55 have 0 need 55
    -4> 2017-04-19 17:46:40.263941 7f4f2087c700 10 cephx client: want=55 need=55 have=0
    -3> 2017-04-19 17:46:40.263943 7f4f2087c700 10 cephx client: build_request
    -2> 2017-04-19 17:46:40.263979 7f4f2087c700 10 cephx client: get auth session key: client_challenge 4b12b1d9b0b6bbc4
    -1> 2017-04-19 17:46:40.263983 7f4f2087c700  1 -- 172.21.15.179:0/2878241838 --> 172.21.15.179:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f4f316c4b40 con 0x7f4f31ad0a00
     0> 2017-04-19 17:46:40.265515 7f4f18859700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f4f18859700 thread_name:ms_pipe_write

 ceph version 12.0.0-2827-g8f6cef9 (8f6cef90871eec5261645a47bc4090c7ffb3a1e4)
 1: (()+0x296a97) [0x7f4f266c9a97]
 2: (()+0x10330) [0x7f4f24c90330]
 3: (ceph::buffer::ptr::ptr(ceph::buffer::ptr const&)+0) [0x7f4f266cb5b0]
 4: (Pipe::writer()+0x821) [0x7f4f269257b1]
 5: (Pipe::Writer::entry()+0xd) [0x7f4f26927f4d]
 6: (()+0x8184) [0x7f4f24c88184]
 7: (clone()+0x6d) [0x7f4f23d6e37d]

/a/sage-2017-04-28_00:12:30-rados-wip-sage-testing2---basic-smithi/1075424
asyncmessenger

/a/sage-2017-04-28_19:45:54-rados-wip-sage-testing---basic-smithi/1077701

/a/sage-2017-05-09_03:37:05-rados-wip-crush-compat---basic-smithi/1115821

/a/sage-2017-05-12_14:53:12-rados-wip-sage-testing---basic-smithi/1171581

/a/sage-2017-05-23_06:26:57-rados-wip-sage-testing---basic-smithi/1220932

/a/sage-2017-05-25_17:58:50-rados-wip-sage-testing2---basic-smithi/1229304

shortly after it's deactivated:

2017-05-25T19:28:43.912 INFO:tasks.ceph.mgr.x.smithi063.stderr:2017-05-25 19:28:43.867704 7f3fddff2700 -1 mgr handle_mgr_map I was active but no longer am
2017-05-25T19:28:44.288 INFO:tasks.ceph.mgr.x.smithi063.stderr:*** Caught signal (Segmentation fault) **
2017-05-25T19:28:44.290 INFO:tasks.ceph.mgr.x.smithi063.stderr: in thread 7f3fdb7ed700 thread_name:ms_pipe_read

and

2017-05-29T23:00:12.134 INFO:tasks.ceph.mgr.x.smithi079.stderr:2017-05-29 23:00:12.135345 7f4e97afc700 -1 mgr handle_mgr_map I was active but no longer am
2017-05-29T23:00:12.336 INFO:teuthology.orchestra.run.smithi075.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","monmap":{"epoch":2,"fsid":"68e52685-205e-4fbd-ace1-e54f094042a8","modified":"2017-05-29 22:50:33.790350","created":"2017-05-29 22:50:19.287092","feature
s":{"persistent":["kraken","luminous"],"optional":[]},"mons":[{"rank":0,"name":"a","addr":"172.21.15.75:6789/0","public_addr":"172.21.15.75:6789/0"},{"rank":1,"name":"b","addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0"},{"rank":2,"name":"c","addr":"172.21.15.75:6790/0","public_addr":"172.21.15.75:67
90/0"}]}}
2017-05-29T23:00:12.643 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","monmap":{"epoch":2,"fsid":"68e52685-205e-4fbd-ace1-e54f094042a8","modified":"2017-05-29 22:50:33.790350","created":"2017-05-29 22:50:19.287092","fe
atures":{"persistent":["kraken","luminous"],"optional":[]},"mons":[{"rank":0,"name":"a","addr":"172.21.15.75:6789/0","public_addr":"172.21.15.75:6789/0"},{"rank":1,"name":"b","addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0"},{"rank":2,"name":"c","addr":"172.21.15.75:6790/0","public_addr":"172.21.15.
75:6790/0"}]}}

2017-05-29T23:00:12.643 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3
2017-05-29T23:00:12.643 INFO:teuthology.orchestra.run.smithi075:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.15.75:6789 mon_status'
2017-05-29T23:00:12.644 INFO:tasks.ceph.mgr.x.smithi079.stderr:*** Caught signal (Segmentation fault) **
2017-05-29T23:00:12.644 INFO:tasks.ceph.mgr.x.smithi079.stderr: in thread 7f4e930f1700 thread_name:mgr-fin

2017-05-31T21:44:11.886 INFO:tasks.ceph.mgr.x.smithi166.stderr:2017-05-31 21:44:11.566469 7f8ec5ddb700 -1 mgr handle_mgr_map I was active but no longer am
2017-05-31T21:44:12.125 INFO:tasks.ceph.mgr.x.smithi166.stderr:*** Caught signal (Segmentation fault) **
2017-05-31T21:44:12.128 INFO:tasks.ceph.mgr.x.smithi166.stderr: in thread 7f8ec8de1700 thread_name:msgr-worker-0
2017-05-31T21:44:12.300 INFO:tasks.ceph.mgr.x.smithi166.stderr:daemon-helper: command crashed with signal 11

go ta stack trace and core this time!

2017-06-01T05:58:58.807 INFO:tasks.ceph.mgr.x.smithi086.stderr:2017-06-01 05:58:58.808442 7f935c765700 -1 mgr handle_mgr_map I was active but no longer am
2017-06-01T05:58:58.823 INFO:tasks.ceph.mgr.x.smithi086.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.2-1874-g9581c1e/rpm/el7/BUILD/ceph-12.0.2-1874-g9581c1e/src/msg/DispatchQueue.h: In function 'Dis
patchQueue::~DispatchQueue()' thread 7f935c765700 time 2017-06-01 05:58:58.824253
2017-06-01T05:58:58.826 INFO:tasks.ceph.mgr.x.smithi086.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.2-1874-g9581c1e/rpm/el7/BUILD/ceph-12.0.2-1874-g9581c1e/src/msg/DispatchQueue.h: 228: FAILED asse
rt(mqueue.empty())
2017-06-01T05:58:58.829 INFO:tasks.ceph.mgr.x.smithi086.stderr: ceph version  12.0.2-1874-g9581c1e (9581c1ec2323fe8aeeb9e60dc3397298b2350970) luminous (dev)
2017-06-01T05:58:58.832 INFO:tasks.ceph.mgr.x.smithi086.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x556325792310]
2017-06-01T05:58:58.841 INFO:tasks.ceph.mgr.x.smithi086.stderr: 2: (DispatchQueue::~DispatchQueue()+0x44) [0x55632582fe04]
2017-06-01T05:58:58.844 INFO:tasks.ceph.mgr.x.smithi086.stderr: 3: (SimpleMessenger::~SimpleMessenger()+0x140) [0x55632582d6c0]
2017-06-01T05:58:58.846 INFO:tasks.ceph.mgr.x.smithi086.stderr: 4: (SimpleMessenger::~SimpleMessenger()+0x9) [0x55632582d7d9]
2017-06-01T05:58:58.849 INFO:tasks.ceph.mgr.x.smithi086.stderr: 5: (DaemonServer::~DaemonServer()+0x26) [0x556325639bc6]
2017-06-01T05:58:58.851 INFO:tasks.ceph.mgr.x.smithi086.stderr: 6: (Mgr::~Mgr()+0x1c) [0x55632566f53c]
2017-06-01T05:58:58.855 INFO:tasks.ceph.mgr.x.smithi086.stderr: 7: (std::_Sp_counted_ptr<Mgr*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x12) [0x556325665792]
2017-06-01T05:58:58.858 INFO:tasks.ceph.mgr.x.smithi086.stderr: 8: (std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x39) [0x55632561b6e9]
2017-06-01T05:58:58.860 INFO:tasks.ceph.mgr.x.smithi086.stderr: 9: (MgrStandby::handle_mgr_map(MMgrMap*)+0x48b) [0x5563256629db]
2017-06-01T05:58:58.862 INFO:tasks.ceph.mgr.x.smithi086.stderr: 10: (MgrStandby::ms_dispatch(Message*)+0x27e) [0x5563256631de]
2017-06-01T05:58:58.864 INFO:tasks.ceph.mgr.x.smithi086.stderr: 11: (DispatchQueue::entry()+0x7a2) [0x5563259bf202]
2017-06-01T05:58:58.867 INFO:tasks.ceph.mgr.x.smithi086.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x55632582e35d]
2017-06-01T05:58:58.870 INFO:tasks.ceph.mgr.x.smithi086.stderr: 13: (()+0x7dc5) [0x7f93619a0dc5]
2017-06-01T05:58:58.873 INFO:tasks.ceph.mgr.x.smithi086.stderr: 14: (clone()+0x6d) [0x7f9360a8573d]
2017-06-01T05:58:58.875 INFO:tasks.ceph.mgr.x.smithi086.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

/a/sage-2017-06-02_08:32:01-rados-wip-sage-testing-distro-basic-smithi/1255413
/a/sage-2017-06-02_08:32:01-rados-wip-sage-testing-distro-basic-smithi/1255363