Ceph

I can't seem to actually find this thread in the core, which is odd. Marking as Need More Info for the moment.

Haven't seen it recur lately, marking can't reproduce.

Program terminated with signal SIGSEGV, Segmentation fault.
#0  PushOp::print (this=this@entry=0xffffffff00000013, out=...) at /usr/include/c++/4.8/bits/stl_map.h:435
435           { return _M_t.size(); }
(gdb) bt
#0  PushOp::print (this=this@entry=0xffffffff00000013, out=...) at /usr/include/c++/4.8/bits/stl_map.h:435
#1  0x00000000006ff6ee in operator<< (out=..., op=...) at osd/osd_types.cc:4154
#2  0x00000000007ca1c2 in ReplicatedBackend::send_pushes (this=this@entry=0x24a8a00, prio=10, pushes=...) at osd/ReplicatedPG.cc:8444
#3  0x00000000007caaae in ReplicatedBackend::do_pull (this=this@entry=0x24a8a00, op=...) at osd/ReplicatedPG.cc:2200
#4  0x000000000090415e in ReplicatedBackend::handle_message (this=0x24a8a00, op=...) at osd/ReplicatedBackend.cc:138
#5  0x000000000079df6b in ReplicatedPG::do_request (this=0x257ec00, op=..., handle=...) at osd/ReplicatedPG.cc:1113
#6  0x00000000005fca01 in OSD::dequeue_op (this=0x218c000, pg=..., op=..., handle=...) at osd/OSD.cc:7715
#7  0x0000000000617054 in OSD::OpWQ::_process (this=0x218ce28, pg=..., handle=...) at osd/OSD.cc:7685
#8  0x0000000000659ebc in ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process (this=0x218ce28, handle=...) at ./common/WorkQueue.h:190
#9  0x0000000000a4d951 in ThreadPool::worker (this=0x218c470, wt=0x2154220) at common/WorkQueue.cc:125
#10 0x0000000000a4e840 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#11 0x00007fb7a175e182 in start_thread (arg=0x7fb78b678700) at pthread_create.c:312
#12 0x00007fb79fed238d in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()

incomplete log, but the core is good. ubuntu@teuthology:/a/sage-2014-08-10_18:40:12-rados-firefly-next-distro-basic-multi/414840

#1  0x00000000006ff6ee in operator<< (out=..., op=...) at osd/osd_types.cc:4154
4154    osd/osd_types.cc: No such file or directory.
(gdb) p op
$6 = (const PushOp &) @0xffffffff00000013: <error reading variable>
(gdb) up
#2  0x00000000007ca1c2 in ReplicatedBackend::send_pushes (this=this@entry=0x24a8a00, prio=10, pushes=...) at osd/ReplicatedPG.cc:8444
8444    osd/ReplicatedPG.cc: No such file or directory.
(gdb) p j
$7 = {_M_current = 0xffffffff00000013}
(gdb) p pushes
$9 = 0

core is on vpm022 with matching packages installed.