Project

General

Profile

Bug #1473

osd assert failure: FAILED assert(0 == "oi disagrees with stat, or error code on stat")

Added by Sam Lang almost 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
OSD
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

2011-08-31 10:04:29.898531 7fa020ddd700 -- 192.168.101.11:6801/7607 >> 192.168.101.14:6806/2173 pipe(0x78a7000 sd=34 pgs=521 cs=29 l=0).fault initiating reconnect
2011-08-31 10:04:29.898962 7fa01ca9a700 -- 192.168.101.11:6802/7607 >> 192.168.101.14:6808/2174 pipe(0x7154a00 sd=51 pgs=2 cs=2 l=0).fault first fault
2011-08-31 10:04:29.899049 7fa01fdcd700 -- 192.168.101.11:6801/7607 >> 192.168.101.14:6806/2173 pipe(0x78a7000 sd=34 pgs=521 cs=30 l=0).fault first fault
2011-08-31 10:04:50.199412 7fa02600c700 log [ERR] : 0.211 10000000158.00000000/head oi.size 18425 but stat got 0 size 17785
../../src/osd/ReplicatedPG.cc: In function 'int ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp>&, ceph::bufferlist&)', in thread '0x7fa02600c700'
../../src/osd/ReplicatedPG.cc: 1332: FAILED assert(0 == "oi disagrees with stat, or error code on stat")
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91ba61]
2: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x2051) [0x78ce93]
3: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0xca) [0x7968d0]
4: (ReplicatedPG::do_op(MOSDOp*)+0x20fe) [0x7864c0]
5: (OSD::dequeue_op(PG*)+0x2ef) [0x846f5d]
6: (OSD::OpWQ::_process(PG*)+0x27) [0x8526cd]
7: (ThreadPool::WorkQueue<PG>::_void_process(void*)+0x2e) [0x899e46]
8: (ThreadPool::worker()+0x422) [0x8b4d02]
9: (ThreadPool::WorkThread::entry()+0x1c) [0x851168]
10: (Thread::_entry_func(void*)+0x23) [0x8a0e21]
11: (()+0x6d8c) [0x7fa034be5d8c]
12: (clone()+0x6d) [0x7fa03342704d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91ba61]
2: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x2051) [0x78ce93]
3: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0xca) [0x7968d0]
4: (ReplicatedPG::do_op(MOSDOp*)+0x20fe) [0x7864c0]
5: (OSD::dequeue_op(PG*)+0x2ef) [0x846f5d]
6: (OSD::OpWQ::_process(PG*)+0x27) [0x8526cd]
7: (ThreadPool::WorkQueue<PG>::_void_process(void*)+0x2e) [0x899e46]
8: (ThreadPool::worker()+0x422) [0x8b4d02]
9: (ThreadPool::WorkThread::entry()+0x1c) [0x851168]
10: (Thread::_entry_func(void*)+0x23) [0x8a0e21]
11: (()+0x6d8c) [0x7fa034be5d8c]
12: (clone()+0x6d) [0x7fa03342704d]
  • Caught signal (Aborted) *
    in thread 0x7fa02600c700
    ceph version (commit:)
    1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x8f3031]
    2: /usr/ceph/bin/cosd() [0x91c0d3]
    3: (()+0xfc60) [0x7fa034beec60]
    4: (gsignal()+0x35) [0x7fa033374d05]
    5: (abort()+0x186) [0x7fa033378ab6]
    6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7fa033c2b6dd]
    7: (()+0xb9926) [0x7fa033c29926]
    8: (()+0xb9953) [0x7fa033c29953]
    9: (()+0xb9a5e) [0x7fa033c29a5e]
    10: (ceph::__ceph_assert_fail(char const
    , char const*, int, char const*)+0x1f3) [0x91bbcb]
    11: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x2051) [0x78ce93]
    12: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0xca) [0x7968d0]
    13: (ReplicatedPG::do_op(MOSDOp*)+0x20fe) [0x7864c0]
    14: (OSD::dequeue_op(PG*)+0x2ef) [0x846f5d]
    15: (OSD::OpWQ::_process(PG*)+0x27) [0x8526cd]
    16: (ThreadPool::WorkQueue<PG>::_void_process(void*)+0x2e) [0x899e46]
    17: (ThreadPool::worker()+0x422) [0x8b4d02]
    18: (ThreadPool::WorkThread::entry()+0x1c) [0x851168]
    19: (Thread::_entry_func(void*)+0x23) [0x8a0e21]
    20: (()+0x6d8c) [0x7fa034be5d8c]
    21: (clone()+0x6d) [0x7fa03342704d]

(gdb) bt
#0 0x00007fa034beeb3b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000091bf26 in reraise_fatal (signum=6) at ../../src/global/signal_handler.cc:59
#2 0x000000000091c144 in handle_fatal_signal (signum=6) at ../../src/global/signal_handler.cc:106
#3 <signal handler called>
#4 0x00007fa033374d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fa033378ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007fa033c2b6dd in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fa033c29926 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007fa033c29953 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007fa033c29a5e in _cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x000000000091bbcb in ceph::
_ceph_assert_fail (assertion=0xa2d288 "0 == \"oi disagrees with stat, or error code on stat\"",
file=0xa2c981 "../../src/osd/ReplicatedPG.cc", line=1332,
func=0xa36a20 "int ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp>&, ceph::bufferlist&)")
at ../../src/common/assert.cc:70
#11 0x000000000078ce93 in ReplicatedPG::do_osd_ops (this=0x7890000, ctx=0x35aab00, ops=..., odata=...)
at ../../src/osd/ReplicatedPG.cc:1332
#12 0x00000000007968d0 in ReplicatedPG::prepare_transaction (this=0x7890000, ctx=0x35aab00) at ../../src/osd/ReplicatedPG.cc:2472
#13 0x00000000007864c0 in ReplicatedPG::do_op (this=0x7890000, op=0x3245000) at ../../src/osd/ReplicatedPG.cc:595
#14 0x0000000000846f5d in OSD::dequeue_op (this=0x2169000, pg=0x7890000) at ../../src/osd/OSD.cc:5146
#15 0x00000000008526cd in OSD::OpWQ::_process (this=0x2169a48, pg=0x7890000) at ../../src/osd/OSD.h:342
#16 0x0000000000899e46 in WorkQueue<PG>::_void_process (this=0x2169a48, p=0x7890000) at ../../src/common/WorkQueue.h:62
#17 0x00000000008b4d02 in ThreadPool::worker (this=0x2169400) at ../../src/common/WorkQueue.cc:54
#18 0x0000000000851168 in ThreadPool::WorkThread::entry (this=0x2131ca0) at ../../src/common/WorkQueue.h:119
#19 0x00000000008a0e21 in Thread::_entry_func (arg=0x2131ca0) at ../../src/common/Thread.cc:45
#20 0x00007fa034be5d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007fa03342704d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#22 0x0000000000000000 in ?? ()

History

#1 Updated by Sam Lang almost 9 years ago

If I comment out the if(1) check that is being done (and causing the assert failure), I'm able to restart the osds that have died, but (2-3 seconds after restart) I get this crash on one of the osds that had previously crashed.

2011-08-31 15:15:49.018859 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6809/10204 not 192.168.101.11:6809/7922 - wrong node!
2011-08-31 15:15:49.018890 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=28 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.018971 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.019016 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.019159 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=29 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/10755 not 192.168.101.111:6801/8386 - wrong node!
2011-08-31 15:15:49.019173 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=29 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.019377 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6809/10204 not 192.168.101.11:6809/7922 - wrong node!
2011-08-31 15:15:49.019632 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=29 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/10755 not 192.168.101.111:6801/8386 - wrong node!
2011-08-31 15:15:49.019652 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.020096 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6813/10429 not 192.168.101.11:6813/8154 - wrong node!
2011-08-31 15:15:49.020120 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=30 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.020479 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6813/10881 not 192.168.101.114:6813/2714 - wrong node!
2011-08-31 15:15:49.020519 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.020561 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6809/12553 not 192.168.101.115:6809/27652 - wrong node!
2011-08-31 15:15:49.020598 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=35 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6813/10429 not 192.168.101.11:6813/8154 - wrong node!
2011-08-31 15:15:49.020621 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=37 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.020651 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6813/12675 not 192.168.101.115:6813/27765 - wrong node!
2011-08-31 15:15:49.020680 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=29 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/10755 not 192.168.101.111:6801/8386 - wrong node!
2011-08-31 15:15:49.020752 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6809/10204 not 192.168.101.11:6809/7922 - wrong node!
2011-08-31 15:15:49.020800 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6814/24193 not 192.168.101.12:6814/24832 - wrong node!
2011-08-31 15:15:49.020820 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.020837 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.020898 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6813/10881 not 192.168.101.114:6813/2714 - wrong node!
2011-08-31 15:15:49.020935 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6810/23976 not 192.168.101.12:6810/24750 - wrong node!
2011-08-31 15:15:49.020956 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.021007 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.021038 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6809/12553 not 192.168.101.115:6809/27652 - wrong node!
2011-08-31 15:15:49.021096 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6814/24193 not 192.168.101.12:6814/24832 - wrong node!
2011-08-31 15:15:49.021219 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6810/23976 not 192.168.101.12:6810/24750 - wrong node!
2011-08-31 15:15:49.021391 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6813/12675 not 192.168.101.115:6813/27765 - wrong node!
2011-08-31 15:15:49.021961 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.021976 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.021991 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6813/12675 not 192.168.101.115:6813/27765 - wrong node!
2011-08-31 15:15:49.022032 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=42 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6809/12553 not 192.168.101.115:6809/27652 - wrong node!
2011-08-31 15:15:49.022278 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.022848 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.022885 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.022939 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.022967 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023158 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.023194 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023223 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.023291 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:49.023339 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023375 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.023471 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:49.023504 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023527 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.023642 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.023675 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).fault first fault
2011-08-31 15:15:49.023773 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:49.023908 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:49.023982 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.221244 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=35 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6813/10429 not 192.168.101.11:6813/8154 - wrong node!
2011-08-31 15:15:49.221517 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6813/10881 not 192.168.101.114:6813/2714 - wrong node!
2011-08-31 15:15:49.221605 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6814/24193 not 192.168.101.12:6814/24832 - wrong node!
2011-08-31 15:15:49.221811 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6810/23976 not 192.168.101.12:6810/24750 - wrong node!
2011-08-31 15:15:49.222744 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.223818 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.224003 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.224094 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.224215 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:49.224556 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:49.224701 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.421342 7fae44c5f700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6801/8386 pipe(0x3f53500 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6801/10755 not 192.168.101.111:6801/8386 - wrong node!
2011-08-31 15:15:49.421391 7fae44e61700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6809/7922 pipe(0x3f20c80 sd=29 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6809/10204 not 192.168.101.11:6809/7922 - wrong node!
2011-08-31 15:15:49.421442 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=32 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.422568 7fae44255700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6813/27765 pipe(0x3f60780 sd=36 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6813/12675 not 192.168.101.115:6813/27765 - wrong node!
2011-08-31 15:15:49.422600 7fae44356700 -- 192.168.101.112:6813/23380 >> 192.168.101.115:6809/27652 pipe(0x3f60a00 sd=42 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6809/12553 not 192.168.101.115:6809/27652 - wrong node!
2011-08-31 15:15:49.621915 7fae44d60700 -- 192.168.101.112:6813/23380 >> 192.168.101.11:6813/8154 pipe(0x3f20a00 sd=35 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6813/10429 not 192.168.101.11:6813/8154 - wrong node!
2011-08-31 15:15:49.621996 7fae44053700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6814/24832 pipe(0x3f60280 sd=39 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6814/24193 not 192.168.101.12:6814/24832 - wrong node!
2011-08-31 15:15:49.622111 7fae4475a700 -- 192.168.101.112:6813/23380 >> 192.168.101.114:6813/2714 pipe(0x3f53780 sd=30 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6813/10881 not 192.168.101.114:6813/2714 - wrong node!
2011-08-31 15:15:49.622154 7fae44154700 -- 192.168.101.112:6813/23380 >> 192.168.101.12:6810/24750 pipe(0x3f60500 sd=38 pgs=0 cs=0 l=0).connect claims to be 192.168.101.12:6810/23976 not 192.168.101.12:6810/24750 - wrong node!
2011-08-31 15:15:49.623330 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.624586 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=0).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.624718 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.624817 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=0).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.624858 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:49.625154 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:49.625300 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=0).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.661046 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.662473 7fae44b5e700 -- 192.168.101.112:6813/23380 >> 192.168.101.111:6805/8478 pipe(0x3f53280 sd=28 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6805/10832 not 192.168.101.111:6805/8478 - wrong node!
2011-08-31 15:15:49.665093 7fae42b3e700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6810/27652 pipe(0x3f89500 sd=55 pgs=0 cs=0 l=1).connect claims to be 192.168.101.115:6810/12553 not 192.168.101.115:6810/27652 - wrong node!
2011-08-31 15:15:49.665132 7fae43346700 -- 192.168.101.112:6815/23381 >> 192.168.101.115:6814/27765 pipe(0x3f58500 sd=50 pgs=0 cs=0 l=1).connect claims to be 192.168.101.115:6814/12675 not 192.168.101.115:6814/27765 - wrong node!
2011-08-31 15:15:49.665158 7fae43447700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6814/8154 pipe(0x3f58780 sd=49 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6814/10429 not 192.168.101.11:6814/8154 - wrong node!
2011-08-31 15:15:49.665209 7fae43144700 -- 192.168.101.112:6815/23381 >> 192.168.101.114:6814/2714 pipe(0x3f89c80 sd=52 pgs=0 cs=0 l=1).connect claims to be 192.168.101.114:6814/10881 not 192.168.101.114:6814/2714 - wrong node!
2011-08-31 15:15:49.665238 7fae43b4e700 -- 192.168.101.112:6815/23381 >> 192.168.101.11:6810/7922 pipe(0x3f68a00 sd=37 pgs=0 cs=0 l=1).connect claims to be 192.168.101.11:6810/10204 not 192.168.101.11:6810/7922 - wrong node!
2011-08-31 15:15:49.665264 7fae43649700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6802/8386 pipe(0x3f58c80 sd=47 pgs=0 cs=0 l=1).connect claims to be 192.168.101.111:6802/10755 not 192.168.101.111:6802/8386 - wrong node!
2011-08-31 15:15:50.425556 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=0).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
2011-08-31 15:15:50.749669 7fae43043700 -- 192.168.101.112:6815/23381 >> 192.168.101.111:6806/8478 pipe(0x3f89a00 sd=53 pgs=0 cs=0 l=1).connect claims to be 192.168.101.111:6806/10832 not 192.168.101.111:6806/8478 - wrong node!
../../src/osd/PG.cc: In function 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)', in thread '0x7fae4ad81700'
../../src/osd/PG.cc: 3892: FAILED assert(0 == "we got a bad state machine event")
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91b861]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x93e87c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x977d03]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x977735]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9771fd]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x976d02]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x97669e]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97628c]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x975c77]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x9757dd]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x975180]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x974afc]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x9741a6]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x9734ef]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x971d44]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97000b]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96d262]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ccc7]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c8ff]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c40b]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7fd91a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f62dc]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7dfed3]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x952157]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x947f6e]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x850714]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x83e139]
28: (OSD::_dispatch(Message*)+0x737) [0x82ff93]
29: (OSD::ms_dispatch(Message*)+0x146) [0x82ecb4]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8d93a6]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c32c8]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x775f6c]
33: (Thread::_entry_func(void*)+0x23) [0x8a0c29]
34: (()+0x6d8c) [0x7fae57855d8c]
35: (clone()+0x6d) [0x7fae5609704d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x91b861]
2: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x93e87c]
3: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x977d03]
4: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x977735]
5: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9771fd]
6: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x976d02]
7: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x97669e]
8: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97628c]
9: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x975c77]
10: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x9757dd]
11: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x975180]
12: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x974afc]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x9741a6]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x9734ef]
15: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x971d44]
16: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97000b]
17: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96d262]
18: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ccc7]
19: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c8ff]
20: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c40b]
21: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7fd91a]
22: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f62dc]
23: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7dfed3]
24: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x952157]
25: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x947f6e]
26: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x850714]
27: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x83e139]
28: (OSD::_dispatch(Message*)+0x737) [0x82ff93]
29: (OSD::ms_dispatch(Message*)+0x146) [0x82ecb4]
30: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8d93a6]
31: (SimpleMessenger::dispatch_entry()+0x810) [0x8c32c8]
32: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x775f6c]
33: (Thread::_entry_func(void*)+0x23) [0x8a0c29]
34: (()+0x6d8c) [0x7fae57855d8c]
35: (clone()+0x6d) [0x7fae5609704d]
  • Caught signal (Aborted) *
    in thread 0x7fae4ad81700
    ceph version (commit:)
    1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x8f2e31]
    2: /usr/ceph/bin/cosd() [0x91bed3]
    3: (()+0xfc60) [0x7fae5785ec60]
    4: (gsignal()+0x35) [0x7fae55fe4d05]
    5: (abort()+0x186) [0x7fae55fe8ab6]
    6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7fae5689b6dd]
    7: (()+0xb9926) [0x7fae56899926]
    8: (()+0xb9953) [0x7fae56899953]
    9: (()+0xb9a5e) [0x7fae56899a5e]
    10: (ceph::__ceph_assert_fail(char const
    , char const*, int, char const*)+0x1f3) [0x91b9cb]
    11: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xa6) [0x93e87c]
    12: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x4b) [0x977d03]
    13: (boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x28) [0x977735]
    14: (boost::statechart::detail::inner_constructor<boost::mpl::l_item<mpl_::long_<1l>, PG::RecoveryState::Crashed, boost::mpl::l_end>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator> >::construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>&)+0x23) [0x9771fd]
    15: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit_impl<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::statechart::detail::no_transition_function>(boost::statechart::detail::no_transition_function const&)+0x80) [0x976d02]
    16: (boost::statechart::detail::safe_reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::transit<PG::RecoveryState::Crashed>()+0x26) [0x97669e]
    17: (boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>::react_without_action(PG::RecoveryState::Started&)+0x22) [0x97628c]
    18: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::without_action::react(PG::RecoveryState::Started&, boost::statechart::event_base const&)+0x26) [0x975c77]
    19: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::base::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2e) [0x9757dd]
    20: (boost::statechart::detail::reaction_dispatcher<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::reactions<PG::RecoveryState::Started>, PG::RecoveryState::Started, boost::statechart::event_base, boost::statechart::event_base, boost::statechart::detail::no_context<boost::statechart::event_base>, void const*>::react(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2d) [0x975180]
    21: (boost::statechart::detail::reaction_result boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>::react<PG::RecoveryState::Started, boost::statechart::event_base, void const*>(PG::RecoveryState::Started&, boost::statechart::event_base const&, void const* const&)+0x2b) [0x974afc]
    22: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x33) [0x9741a6]
    23: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list1<boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))> > >(boost::statechart::event_base const&, void const*)+0x2b) [0x9734ef]
    24: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x53) [0x971d44]
    25: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::statechart::custom_reaction<PG::RecoveryState::AdvMap>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &(boost::statechart::detail::no_context<boost::statechart::event_base>::no_function(boost::statechart::event_base const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base const&, void const*)+0x2b) [0x97000b]
    26: (boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x2c) [0x96d262]
    27: (boost::statechart::simple_state<PG::RecoveryState::Primary, PG::RecoveryState::Started, PG::RecoveryState::Peering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96ccc7]
    28: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c8ff]
    29: (boost::statechart::simple_state<PG::RecoveryState::GetInfo, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x69) [0x96c40b]
    30: (boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>::operator()()+0x46) [0x7fd91a]
    31: (boost::statechart::detail::safe_reaction_result boost::statechart::null_exception_translator::operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler>(boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::exception_event_handler)+0x2a) [0x7f62dc]
    32: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x13f) [0x7dfed3]
    33: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x23) [0x952157]
    34: (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x14e) [0x947f6e]
    35: (PG::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x34) [0x850714]
    36: (OSD::handle_pg_log(MOSDPGLog*)+0x3d9) [0x83e139]
    37: (OSD::_dispatch(Message*)+0x737) [0x82ff93]
    38: (OSD::ms_dispatch(Message*)+0x146) [0x82ecb4]
    39: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x8d93a6]
    40: (SimpleMessenger::dispatch_entry()+0x810) [0x8c32c8]
    41: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x775f6c]
    42: (Thread::_entry_func(void*)+0x23) [0x8a0c29]
    43: (()+0x6d8c) [0x7fae57855d8c]
    44: (clone()+0x6d) [0x7fae5609704d]

#2 Updated by Sage Weil almost 9 years ago

  • Priority changed from Normal to High
  • Target version set to v0.35

Can you reproduce the second crash with the osd log enabled?

#3 Updated by Sage Weil almost 9 years ago

  • Target version changed from v0.35 to v0.36

#4 Updated by Sage Weil almost 9 years ago

  • translation missing: en.field_position set to 8

#5 Updated by Sage Weil almost 9 years ago

  • translation missing: en.field_position deleted (27)
  • translation missing: en.field_position set to 46

#6 Updated by Sage Weil almost 9 years ago

  • Target version changed from v0.36 to v0.37

#7 Updated by Sage Weil almost 9 years ago

  • Target version changed from v0.37 to v0.38

#8 Updated by Sage Weil almost 9 years ago

  • Description updated (diff)

Just saw this on osd.11 on alexandria (see #1612):

osd/ReplicatedPG.cc: In function 'int ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::bufferlist&)', in thread '0x7fec8ef3b700'
osd/ReplicatedPG.cc: 1386: FAILED assert(0 == "oi disagrees with stat, or error code on stat")
 ceph version 0.36-277-g3f619dd (commit:3f619ddb55a7add161ab4fcee2ccb28bc60cb734)
 1: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x81e0) [0x4d4510]
 2: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0x5b) [0x4e44eb]
 3: (ReplicatedPG::do_op(MOSDOp*)+0x14cd) [0x4e6f2d]
 4: (OSD::dequeue_op(PG*)+0x583) [0x535953]
 5: (ThreadPool::worker()+0xa28) [0x5d03d8]
 6: (ThreadPool::WorkThread::entry()+0xd) [0x55ad7d]
 7: (()+0x68ba) [0x7fec9d6368ba]
 8: (clone()+0x6d) [0x7fec9c12602d]
 ceph version 0.36-277-g3f619dd (commit:3f619ddb55a7add161ab4fcee2ccb28bc60cb734)
 1: (ReplicatedPG::do_osd_ops(ReplicatedPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&, ceph::buffer::list&)+0x81e0) [0x4d4510]
 2: (ReplicatedPG::prepare_transaction(ReplicatedPG::OpContext*)+0x5b) [0x4e44eb]
 3: (ReplicatedPG::do_op(MOSDOp*)+0x14cd) [0x4e6f2d]
 4: (OSD::dequeue_op(PG*)+0x583) [0x535953]
 5: (ThreadPool::worker()+0xa28) [0x5d03d8]
 6: (ThreadPool::WorkThread::entry()+0xd) [0x55ad7d]
 7: (()+0x68ba) [0x7fec9d6368ba]
 8: (clone()+0x6d) [0x7fec9c12602d]

#9 Updated by Samuel Just almost 9 years ago

  • Description updated (diff)

At least the recent instances of this were probably caused by the btrfs xattr bug (#1612).

#10 Updated by Samuel Just almost 9 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF