Bug #2646
mon:update_from_paxos: error parsing incremental update: buffer::end_of_buffer
Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Monitor
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
-7> 2012-06-22 00:36:45.912018 7f0cbe00a700 1 -- [2607:f298:4:2243::5754]:6789/0 >> :/0 pipe(0x5147500 sd=17 pgs=0 cs=0 l=0).accept sd=17 -6> 2012-06-22 00:39:08.677789 7f0c9d3d1700 1 -- [2607:f298:4:2243::5754]:6789/0 --> mon.1 [2607:f298:4:2243::5753]:6789/0 -- mon_probe(slurp 31b8be2f-ac05-4e56-96b7-e702df166e29 name peon5754 machine_name pgmap 3116382-3116545) v1 -- ?+0 0x15a13900 -5> 2012-06-22 00:39:09.044030 7f0c9d3d1700 1 -- [2607:f298:4:2243::5754]:6789/0 <== mon.1 [2607:f298:4:2243::5753]:6789/0 3092 ==== mon_probe(data 31b8be2f-ac05-4e56-96b7-e702df166e29 name peon5753 machine_name pgmap 3116591-3117092) v1 ==== 274258+0+0 (3632809312 0 0) 0x15a13900 con 0x9066c80 -4> 2012-06-22 00:39:42.918877 7f0cbe00a700 1 -- [2607:f298:4:2243::5754]:6789/0 >> :/0 pipe(0x5e8f500 sd=18 pgs=0 cs=0 l=0).accept sd=18 -3> 2012-06-22 00:39:44.523942 7f0c9d3d1700 1 -- [2607:f298:4:2243::5754]:6789/0 --> mon.1 [2607:f298:4:2243::5753]:6789/0 -- mon_probe(slurp 31b8be2f-ac05-4e56-96b7-e702df166e29 name peon5754 machine_name pgmap 3116591-3116632) v1 -- ?+0 0x15a13900 -2> 2012-06-22 00:39:44.534313 7f0c9d3d1700 1 -- [2607:f298:4:2243::5754]:6789/0 <== osd.0 [2607:f298:4:2243::6236]:6800/3992 1 ==== auth(proto 0 26 bytes epoch 3) v1 ==== 56+0+0 (452516168 0 0) 0x62fa200 con 0x54c0b40 -1> 2012-06-22 00:39:44.812095 7f0c9cbd0700 0 mon.peon5754@2(slurping).pg v3116545 update_from_paxos: error parsing incremental update: buffer::end_of_buffer 0> 2012-06-22 00:39:44.875370 7f0c9cbd0700 -1 mon/PGMonitor.cc: In function 'virtual void PGMonitor::update_from_paxos()' thread 7f0c9cbd0700 time 2012-06-22 00:39:44.832068 mon/PGMonitor.cc: 181: FAILED assert(0 == "update_from_paxos: error parsing incremental update") ceph version 0.46-95-g66b2bb3 (commit:66b2bb339ed59198dfd622279c3671f7304329c1) 1: (PGMonitor::update_from_paxos()+0xdd4) [0x4ef274] 2: (PGMonitor::tick()+0x56) [0x4e7896] 3: (Monitor::tick()+0x5d) [0x481c7d] 4: (SafeTimer::timer_thread()+0x34a) [0x5cdb2a] 5: (SafeTimerThread::entry()+0xd) [0x5ce6fd] 6: (()+0x7e9a) [0x7f0cbdbe6e9a] 7: (clone()+0x6d) [0x7f0cbc39f4bd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- end dump of recent events --- 2012-06-22 00:39:50.365160 7f0c9cbd0700 -1 *** Caught signal (Aborted) ** in thread 7f0c9cbd0700 ceph version 0.46-95-g66b2bb3 (commit:66b2bb339ed59198dfd622279c3671f7304329c1) 1: /usr/bin/ceph-mon() [0x61cd4a] 2: (()+0xfcb0) [0x7f0cbdbeecb0] 3: (gsignal()+0x35) [0x7f0cbc2e3445] 4: (abort()+0x17b) [0x7f0cbc2e6bab] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f0cbcc3169d] 6: (()+0xb5846) [0x7f0cbcc2f846] 7: (()+0xb5873) [0x7f0cbcc2f873] 8: (()+0xb596e) [0x7f0cbcc2f96e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x282) [0x5d5ed2] 10: (PGMonitor::update_from_paxos()+0xdd4) [0x4ef274] 11: (PGMonitor::tick()+0x56) [0x4e7896] 12: (Monitor::tick()+0x5d) [0x481c7d] 13: (SafeTimer::timer_thread()+0x34a) [0x5cdb2a] 14: (SafeTimerThread::entry()+0xd) [0x5ce6fd] 15: (()+0x7e9a) [0x7f0cbdbe6e9a] 16: (clone()+0x6d) [0x7f0cbc39f4bd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2012-06-22 00:39:50.365160 7f0c9cbd0700 -1 *** Caught signal (Aborted) ** in thread 7f0c9cbd0700 ceph version 0.46-95-g66b2bb3 (commit:66b2bb339ed59198dfd622279c3671f7304329c1) 1: /usr/bin/ceph-mon() [0x61cd4a] 2: (()+0xfcb0) [0x7f0cbdbeecb0] 3: (gsignal()+0x35) [0x7f0cbc2e3445] 4: (abort()+0x17b) [0x7f0cbc2e6bab] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f0cbcc3169d] 6: (()+0xb5846) [0x7f0cbcc2f846] 7: (()+0xb5873) [0x7f0cbcc2f873] 8: (()+0xb596e) [0x7f0cbcc2f96e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x282) [0x5d5ed2] 10: (PGMonitor::update_from_paxos()+0xdd4) [0x4ef274] 11: (PGMonitor::tick()+0x56) [0x4e7896] 12: (Monitor::tick()+0x5d) [0x481c7d] 13: (SafeTimer::timer_thread()+0x34a) [0x5cdb2a] 14: (SafeTimerThread::entry()+0xd) [0x5ce6fd] 15: (()+0x7e9a) [0x7f0cbdbe6e9a] 16: (clone()+0x6d) [0x7f0cbc39f4bd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- end dump of recent events ---
Related issues
History
#1 Updated by Sage Weil about 11 years ago
- Status changed from New to Resolved
#2 Updated by Greg Farnum about 11 years ago
840ae244499496d543d634713bdee7c7884ce527
The tick happened at the same time as slurping, which meant the disk was inconsistent and the pg_map it was trying to read didn't exist on the disk.