Bug #1473: osd assert failure: FAILED assert(0 == "oi disagrees with stat, or error code on stat") - Ceph - Ceph

Actions

Copy link

Bug #1473

closed

osd assert failure: FAILED assert(0 == "oi disagrees with stat, or error code on stat")

Added by Sam Lang over 12 years ago. Updated over 12 years ago.

Status:

Resolved

Priority:

High

Assignee:

Category:

OSD

Target version:

v0.38

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2011-08-31 10:04:29.898531 7fa020ddd700 -- 192.168.101.11:6801/7607 >> 192.168.101.14:6806/2173 pipe(0x78a7000 sd=34 pgs=521 cs=29 l=0).fault initiating reconnect
2011-08-31 10:04:29.898962 7fa01ca9a700 -- 192.168.101.11:6802/7607 >> 192.168.101.14:6808/2174 pipe(0x7154a00 sd=51 pgs=2 cs=2 l=0).fault first fault
2011-08-31 10:04:29.899049 7fa01fdcd700 -- 192.168.101.11:6801/7607 >> 192.168.101.14:6806/2173 pipe(0x78a7000 sd=34 pgs=521 cs=30 l=0).fault first fault
2011-08-31 10:04:50.199412 7fa02600c700 log [ERR] : 0.211 10000000158.00000000/head oi.size 18425 but stat got 0 size 17785
../../src/osd/ReplicatedPG.cc: In function 'int ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp>&, ceph::bufferlist&)', in thread '0x7fa02600c700'
../../src/osd/ReplicatedPG.cc: 1332: FAILED assert(0 == "oi disagrees with stat, or error code on stat")
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91ba61]
2: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x2051) [0x78ce93]
3: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0xca) [0x7968d0]
4: (ReplicatedPG::do_op(MOSDOp*)+0x20fe) [0x7864c0]
5: (OSD::dequeue_op(PG*)+0x2ef) [0x846f5d]
6: (OSD::OpWQ::_process(PG*)+0x27) [0x8526cd]
7: (ThreadPool::WorkQueue<PG>::_void_process(void*)+0x2e) [0x899e46]
8: (ThreadPool::worker()+0x422) [0x8b4d02]
9: (ThreadPool::WorkThread::entry()+0x1c) [0x851168]
10: (Thread::_entry_func(void*)+0x23) [0x8a0e21]
11: (()+0x6d8c) [0x7fa034be5d8c]
12: (clone()+0x6d) [0x7fa03342704d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91ba61]
2: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x2051) [0x78ce93]
3: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0xca) [0x7968d0]
4: (ReplicatedPG::do_op(MOSDOp*)+0x20fe) [0x7864c0]
5: (OSD::dequeue_op(PG*)+0x2ef) [0x846f5d]
6: (OSD::OpWQ::_process(PG*)+0x27) [0x8526cd]
7: (ThreadPool::WorkQueue<PG>::_void_process(void*)+0x2e) [0x899e46]
8: (ThreadPool::worker()+0x422) [0x8b4d02]
9: (ThreadPool::WorkThread::entry()+0x1c) [0x851168]
10: (Thread::_entry_func(void*)+0x23) [0x8a0e21]
11: (()+0x6d8c) [0x7fa034be5d8c]
12: (clone()+0x6d) [0x7fa03342704d]

Caught signal (Aborted) *
in thread 0x7fa02600c700
ceph version (commit:)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x8f3031]
2: /usr/ceph/bin/cosd() [0x91c0d3]
3: (()+0xfc60) [0x7fa034beec60]
4: (gsignal()+0x35) [0x7fa033374d05]
5: (abort()+0x186) [0x7fa033378ab6]
6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7fa033c2b6dd]
7: (()+0xb9926) [0x7fa033c29926]
8: (()+0xb9953) [0x7fa033c29953]
9: (()+0xb9a5e) [0x7fa033c29a5e]
10: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1f3) [0x91bbcb]
11: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x2051) [0x78ce93]
12: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0xca) [0x7968d0]
13: (ReplicatedPG::do_op(MOSDOp*)+0x20fe) [0x7864c0]
14: (OSD::dequeue_op(PG*)+0x2ef) [0x846f5d]
15: (OSD::OpWQ::_process(PG*)+0x27) [0x8526cd]
16: (ThreadPool::WorkQueue<PG>::_void_process(void*)+0x2e) [0x899e46]
17: (ThreadPool::worker()+0x422) [0x8b4d02]
18: (ThreadPool::WorkThread::entry()+0x1c) [0x851168]
19: (Thread::_entry_func(void*)+0x23) [0x8a0e21]
20: (()+0x6d8c) [0x7fa034be5d8c]
21: (clone()+0x6d) [0x7fa03342704d]

(gdb) bt
#0 0x00007fa034beeb3b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000091bf26 in reraise_fatal (signum=6) at ../../src/global/signal_handler.cc:59
#2 0x000000000091c144 in handle_fatal_signal (signum=6) at ../../src/global/signal_handler.cc:106
#3 <signal handler called>
#4 0x00007fa033374d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fa033378ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007fa033c2b6dd in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fa033c29926 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007fa033c29953 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007fa033c29a5e in _cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x000000000091bbcb in ceph::_ceph_assert_fail (assertion=0xa2d288 "0 == \"oi disagrees with stat, or error code on stat\"",
file=0xa2c981 "../../src/osd/ReplicatedPG.cc", line=1332,
func=0xa36a20 "int ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp>&, ceph::bufferlist&)")
at ../../src/common/assert.cc:70
#11 0x000000000078ce93 in ReplicatedPG::do_osd_ops (this=0x7890000, ctx=0x35aab00, ops=..., odata=...)
at ../../src/osd/ReplicatedPG.cc:1332
#12 0x00000000007968d0 in ReplicatedPG::prepare_transaction (this=0x7890000, ctx=0x35aab00) at ../../src/osd/ReplicatedPG.cc:2472
#13 0x00000000007864c0 in ReplicatedPG::do_op (this=0x7890000, op=0x3245000) at ../../src/osd/ReplicatedPG.cc:595
#14 0x0000000000846f5d in OSD::dequeue_op (this=0x2169000, pg=0x7890000) at ../../src/osd/OSD.cc:5146
#15 0x00000000008526cd in OSD::OpWQ::_process (this=0x2169a48, pg=0x7890000) at ../../src/osd/OSD.h:342
#16 0x0000000000899e46 in WorkQueue<PG>::_void_process (this=0x2169a48, p=0x7890000) at ../../src/common/WorkQueue.h:62
#17 0x00000000008b4d02 in ThreadPool::worker (this=0x2169400) at ../../src/common/WorkQueue.cc:54
#18 0x0000000000851168 in ThreadPool::WorkThread::entry (this=0x2131ca0) at ../../src/common/WorkQueue.h:119
#19 0x00000000008a0e21 in Thread::_entry_func (arg=0x2131ca0) at ../../src/common/Thread.cc:45
#20 0x00007fa034be5d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007fa03342704d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#22 0x0000000000000000 in ?? ()

Actions

Copy link

Updated by Sam Lang over 12 years ago

If I comment out the if(1) check that is being done (and causing the assert failure), I'm able to restart the osds that have died, but (2-3 seconds after restart) I get this crash on one of the osds that had previously crashed.

2011-08-31 15:15:49.018859 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6809/10204 not 192.168.101.11:6809/7922 - wrong node!
2011-08-31 15:15:49.018890 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=28 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.018971 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.019016 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.019159 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=29 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/10755 not 192.168.101.111:6801/8386 - wrong node!
2011-08-31 15:15:49.019173 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=29 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.019377 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6809/10204 not 192.168.101.11:6809/7922 - wrong node!
2011-08-31 15:15:49.019632 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=29 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/10755 not 192.168.101.111:6801/8386 - wrong node!
2011-08-31 15:15:49.019652 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.020096 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6813/10429 not 192.168.101.11:6813/8154 - wrong node!
2011-08-31 15:15:49.020120 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=30 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.020479 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6813/10881 not 192.168.101.114:6813/2714 - wrong node!
2011-08-31 15:15:49.020519 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.020561 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6809/12553 not 192.168.101.115:6809/27652 - wrong node!
2011-08-31 15:15:49.020598 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=35 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6813/10429 not 192.168.101.11:6813/8154 - wrong node!
2011-08-31 15:15:49.020621 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=37 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.020651 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6813/12675 not 192.168.101.115:6813/27765 - wrong node!
2011-08-31 15:15:49.020680 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=29 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/10755 not 192.168.101.111:6801/8386 - wrong node!
2011-08-31 15:15:49.020752 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6809/10204 not 192.168.101.11:6809/7922 - wrong node!
2011-08-31 15:15:49.020800 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6814/24193 not 192.168.101.12:6814/24832 - wrong node!
2011-08-31 15:15:49.020820 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.020837 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.020898 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6813/10881 not 192.168.101.114:6813/2714 - wrong node!
2011-08-31 15:15:49.020935 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6810/23976 not 192.168.101.12:6810/24750 - wrong node!
2011-08-31 15:15:49.020956 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.021007 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.021038 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6809/12553 not 192.168.101.115:6809/27652 - wrong node!
2011-08-31 15:15:49.021096 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6814/24193 not 192.168.101.12:6814/24832 - wrong node!
2011-08-31 15:15:49.021219 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6810/23976 not 192.168.101.12:6810/24750 - wrong node!
2011-08-31 15:15:49.021391 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6813/12675 not 192.168.101.115:6813/27765 - wrong node!
2011-08-31 15:15:49.021961 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.021976 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.021991 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6813/12675 not 192.168.101.115:6813/27765 - wrong node!
2011-08-31 15:15:49.022032 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=42 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6809/12553 not 192.168.101.115:6809/27652 - wrong node!
2011-08-31 15:15:49.022278 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.022848 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.022885 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.022939 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.022967 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023158 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.023194 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023223 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.023291 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:49.023339 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023375 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.023471 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:49.023504 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023527 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.023642 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.023675 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023773 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:49.023908 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:49.023982 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.221244 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=35 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6813/10429 not 192.168.101.11:6813/8154 - wrong node!
2011-08-31 15:15:49.221517 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6813/10881 not 192.168.101.114:6813/2714 - wrong node!
2011-08-31 15:15:49.221605 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6814/24193 not 192.168.101.12:6814/24832 - wrong node!
2011-08-31 15:15:49.221811 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6810/23976 not 192.168.101.12:6810/24750 - wrong node!
2011-08-31 15:15:49.222744 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.223818 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.224003 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.224094 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.224215 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:49.224556 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:49.224701 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.421342 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/10755 not 192.168.101.111:6801/8386 - wrong node!
2011-08-31 15:15:49.421391 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=29 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6809/10204 not 192.168.101.11:6809/7922 - wrong node!
2011-08-31 15:15:49.421442 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.422568 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6813/12675 not 192.168.101.115:6813/27765 - wrong node!
2011-08-31 15:15:49.422600 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=42 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6809/12553 not 192.168.101.115:6809/27652 - wrong node!
2011-08-31 15:15:49.621915 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=35 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6813/10429 not 192.168.101.11:6813/8154 - wrong node!
2011-08-31 15:15:49.621996 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6814/24193 not 192.168.101.12:6814/24832 - wrong node!
2011-08-31 15:15:49.622111 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6813/10881 not 192.168.101.114:6813/2714 - wrong node!
2011-08-31 15:15:49.622154 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6810/23976 not 192.168.101.12:6810/24750 - wrong node!
2011-08-31 15:15:49.623330 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.624586 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.624718 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.624817 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.624858 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:49.625154 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:49.625300 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.661046 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.662473 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.665093 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=1).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.665132 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=1).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.665158 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.665209 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=1).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.665238 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.665264 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=1).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:50.425556 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:50.749669 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=1).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
../../src/osd/PG.cc: In function 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)', in thread '0x7fae4ad81700'
../../src/osd/PG.cc: 3892: FAILED assert(0 == "we got a bad state machine event")
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91b861]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x93e87c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x977d03]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x977735]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9771fd]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x976d02]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x97669e]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97628c]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x975c77]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x9757dd]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x975180]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x974afc]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x9741a6]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x9734ef]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x971d44]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97000b]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96d262]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ccc7]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c8ff]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c40b]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7fd91a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f62dc]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7dfed3]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x952157]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x947f6e]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x850714]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x83e139]
28: (OSD::_dispatch(Message*)+0x737) [0x82ff93]
29: (OSD::ms_dispatch(Message*)+0x146) [0x82ecb4]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8d93a6]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c32c8]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x775f6c]
33: (Thread::_entry_func(void*)+0x23) [0x8a0c29]
34: (()+0x6d8c) [0x7fae57855d8c]
35: (clone()+0x6d) [0x7fae5609704d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91b861]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x93e87c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x977d03]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x977735]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9771fd]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x976d02]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x97669e]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97628c]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x975c77]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x9757dd]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x975180]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x974afc]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x9741a6]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x9734ef]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x971d44]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97000b]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96d262]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ccc7]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c8ff]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c40b]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7fd91a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f62dc]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7dfed3]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x952157]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x947f6e]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x850714]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x83e139]
28: (OSD::_dispatch(Message*)+0x737) [0x82ff93]
29: (OSD::ms_dispatch(Message*)+0x146) [0x82ecb4]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8d93a6]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c32c8]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x775f6c]
33: (Thread::_entry_func(void*)+0x23) [0x8a0c29]
34: (()+0x6d8c) [0x7fae57855d8c]
35: (clone()+0x6d) [0x7fae5609704d]

Caught signal (Aborted) *
in thread 0x7fae4ad81700
ceph version (commit:)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x8f2e31]
2: /usr/ceph/bin/cosd() [0x91bed3]
3: (()+0xfc60) [0x7fae5785ec60]
4: (gsignal()+0x35) [0x7fae55fe4d05]
5: (abort()+0x186) [0x7fae55fe8ab6]
6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7fae5689b6dd]
7: (()+0xb9926) [0x7fae56899926]
8: (()+0xb9953) [0x7fae56899953]
9: (()+0xb9a5e) [0x7fae56899a5e]
10: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1f3) [0x91b9cb]
11: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x93e87c]
12: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x977d03]
13: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x977735]
14: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9771fd]
15: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x976d02]
16: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x97669e]
17: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97628c]
18: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x975c77]
19: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x9757dd]
20: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x975180]
21: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x974afc]
22: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x9741a6]
23: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x9734ef]
24: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x971d44]
25: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97000b]
26: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96d262]
27: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ccc7]
28: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c8ff]
29: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c40b]
30: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7fd91a]
31: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f62dc]
32: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7dfed3]
33: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x952157]
34: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x947f6e]
35: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x850714]
36: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x83e139]
37: (OSD::_dispatch(Message*)+0x737) [0x82ff93]
38: (OSD::ms_dispatch(Message*)+0x146) [0x82ecb4]
39: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8d93a6]
40: (SimpleMessenger::dispatch_entry()+0x810) [0x8c32c8]
41: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x775f6c]
42: (Thread::_entry_func(void*)+0x23) [0x8a0c29]
43: (()+0x6d8c) [0x7fae57855d8c]
44: (clone()+0x6d) [0x7fae5609704d]

Actions

Copy link

Updated by Sage Weil over 12 years ago

Priority changed from Normal to High
Target version set to v0.35

Can you reproduce the second crash with the osd log enabled?

Actions

Copy link

Updated by Sage Weil over 12 years ago

Target version changed from v0.35 to v0.36

Actions

Copy link

Updated by Sage Weil over 12 years ago

Translation missing: en.field_position set to 8

Actions

Copy link

Updated by Sage Weil over 12 years ago

Translation missing: en.field_position deleted (27)
Translation missing: en.field_position set to 46

Actions

Copy link

Updated by Sage Weil over 12 years ago

Target version changed from v0.36 to v0.37

Actions

Copy link

Updated by Sage Weil over 12 years ago

Target version changed from v0.37 to v0.38

Actions

Copy link

Updated by Sage Weil over 12 years ago

Description updated (diff)

Just saw this on osd.11 on alexandria (see #1612):

osd/ReplicatedPG.cc: In function 'int ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::bufferlist&)', in thread '0x7fec8ef3b700'
osd/ReplicatedPG.cc: 1386: FAILED assert(0 == "oi disagrees with stat, or error code on stat")
 ceph version 0.36-277-g3f619dd (commit:3f619ddb55a7add161ab4fcee2ccb28bc60cb734)
 1: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x81e0) [0x4d4510]
 2: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0x5b) [0x4e44eb]
 3: (ReplicatedPG::do_op(MOSDOp*)+0x14cd) [0x4e6f2d]
 4: (OSD::dequeue_op(PG*)+0x583) [0x535953]
 5: (ThreadPool::worker()+0xa28) [0x5d03d8]
 6: (ThreadPool::WorkThread::entry()+0xd) [0x55ad7d]
 7: (()+0x68ba) [0x7fec9d6368ba]
 8: (clone()+0x6d) [0x7fec9c12602d]
 ceph version 0.36-277-g3f619dd (commit:3f619ddb55a7add161ab4fcee2ccb28bc60cb734)
 1: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x81e0) [0x4d4510]
 2: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0x5b) [0x4e44eb]
 3: (ReplicatedPG::do_op(MOSDOp*)+0x14cd) [0x4e6f2d]
 4: (OSD::dequeue_op(PG*)+0x583) [0x535953]
 5: (ThreadPool::worker()+0xa28) [0x5d03d8]
 6: (ThreadPool::WorkThread::entry()+0xd) [0x55ad7d]
 7: (()+0x68ba) [0x7fec9d6368ba]
 8: (clone()+0x6d) [0x7fec9c12602d]

Actions

Copy link