Project

General

Profile

Actions
Copy link

Bug #10422

closed

osdmap full crc mismatch due to divergent cache_target_{dirty,full}_ratio_micro

Added by Yuri Weinstein over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Sage Weil
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

All test failed in run http://pulpito.front.sepia.ceph.com/teuthology-2014-12-22_09:28:15-upgrade:dumpling-firefly-x:stress-split-next-distro-basic-multi/

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-12-22_09:28:15-upgrade:dumpling-firefly-x:stress-split-next-distro-basic-multi/672412/

2014-12-22T11:28:03.278 INFO:tasks.ceph.mon.c.plana58.stderr:    -4> 2014-12-22 11:28:03.139305 7f4b30faa700 -1 mon.c@2(peon).osd e1679 previous osdmap reload, raw crc 3487354266
2014-12-22T11:28:03.278 INFO:tasks.ceph.mon.c.plana58.stderr:    -3> 2014-12-22 11:28:03.139316 7f4b30faa700 -1 mon.c@2(peon).osd e1679 reencode of that is 0
2014-12-22T11:28:03.278 INFO:tasks.ceph.mon.c.plana58.stderr:    -2> 2014-12-22 11:28:03.139372 7f4b30faa700 -1 mon.c@2(peon).osd e1679 again raw crc is 0
2014-12-22T11:28:03.278 INFO:tasks.ceph.mon.c.plana58.stderr:    -1> 2014-12-22 11:28:03.139376 7f4b30faa700 -1 mon.c@2(peon).osd e1679  full_crc 818170385
2014-12-22T11:28:03.278 INFO:tasks.ceph.mon.c.plana58.stderr:     0> 2014-12-22 11:28:03.175230 7f4b30faa700 -1 mon/OSDMonitor.cc: In function 'virtual void OSDMonitor::update_from_paxos(bool*)' thread 7f4b30faa700 time 2014-12-22 11:28:03.139384
2014-12-22T11:28:03.279 INFO:tasks.ceph.mon.c.plana58.stderr:mon/OSDMonitor.cc: 251: FAILED assert(0 == "got mismatched crc encoding full map")
2014-12-22T11:28:03.279 INFO:tasks.ceph.mon.c.plana58.stderr:
2014-12-22T11:28:03.279 INFO:tasks.ceph.mon.c.plana58.stderr: ceph version 0.90-586-g4047f2d (4047f2ddef22794fad1e97bb852151e4c6b3cf82)
2014-12-22T11:28:03.279 INFO:tasks.ceph.mon.c.plana58.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0x79831f]
2014-12-22T11:28:03.279 INFO:tasks.ceph.mon.c.plana58.stderr: 2: (OSDMonitor::update_from_paxos(bool*)+0x2dcb) [0x5eab0b]
2014-12-22T11:28:03.279 INFO:tasks.ceph.mon.c.plana58.stderr: 3: (PaxosService::refresh(bool*)+0x3c6) [0x5cba06]
2014-12-22T11:28:03.279 INFO:tasks.ceph.mon.c.plana58.stderr: 4: (Monitor::refresh_from_paxos(bool*)+0x34b) [0x576fab]
2014-12-22T11:28:03.279 INFO:tasks.ceph.mon.c.plana58.stderr: 5: (Paxos::do_refresh()+0x4c) [0x5b386c]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 6: (Paxos::handle_commit(MMonPaxos*)+0x243) [0x5ba7d3]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 7: (Paxos::dispatch(PaxosServiceMessage*)+0x22b) [0x5c30ab]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 8: (Monitor::dispatch(MonSession*, Message*, bool)+0x8c3) [0x598443]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x2a4) [0x598884]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 10: (Monitor::ms_dispatch(Message*)+0x32) [0x5b2b62]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 11: (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x88a087]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 12: (DispatchQueue::entry()+0x44a) [0x88722a]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x780dad]
2014-12-22T11:28:03.280 INFO:tasks.ceph.mon.c.plana58.stderr: 14: (()+0x7e9a) [0x7f4b375a2e9a]
2014-12-22T11:28:03.281 INFO:tasks.ceph.mon.c.plana58.stderr: 15: (clone()+0x6d) [0x7f4b35d4c31d]
2014-12-22T11:28:03.281 INFO:tasks.ceph.mon.c.plana58.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-12-22T11:28:03.281 INFO:tasks.ceph.mon.c.plana58.stderr:
2014-12-22T11:28:03.281 INFO:tasks.ceph.mon.c.plana58.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2014-12-22T11:28:03.281 INFO:tasks.ceph.mon.c.plana58.stderr:*** Caught signal (Aborted) **
2014-12-22T11:28:03.281 INFO:tasks.ceph.mon.c.plana58.stderr: in thread 7f4b30faa700
2014-12-22T11:28:03.281 INFO:tasks.ceph.mon.c.plana58.stderr: ceph version 0.90-586-g4047f2d (4047f2ddef22794fad1e97bb852151e4c6b3cf82)
2014-12-22T11:28:03.282 INFO:tasks.ceph.mon.c.plana58.stderr: 1: ceph-mon() [0x8df645]
2014-12-22T11:28:03.282 INFO:tasks.ceph.mon.c.plana58.stderr: 2: (()+0xfcb0) [0x7f4b375aacb0]
2014-12-22T11:28:03.282 INFO:tasks.ceph.mon.c.plana58.stderr: 3: (gsignal()+0x35) [0x7f4b35c8e0d5]
2014-12-22T11:28:03.282 INFO:tasks.ceph.mon.c.plana58.stderr: 4: (abort()+0x17b) [0x7f4b35c9183b]
2014-12-22T11:28:03.282 INFO:tasks.ceph.mon.c.plana58.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4b365e069d]
2014-12-22T11:28:03.282 INFO:tasks.ceph.mon.c.plana58.stderr: 6: (()+0xb5846) [0x7f4b365de846]
2014-12-22T11:28:03.282 INFO:tasks.ceph.mon.c.plana58.stderr: 7: (()+0xb5873) [0x7f4b365de873]
2014-12-22T11:28:03.282 INFO:tasks.ceph.mon.c.plana58.stderr: 8: (()+0xb596e) [0x7f4b365de96e]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0x7984f9]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 10: (OSDMonitor::update_from_paxos(bool*)+0x2dcb) [0x5eab0b]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 11: (PaxosService::refresh(bool*)+0x3c6) [0x5cba06]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 12: (Monitor::refresh_from_paxos(bool*)+0x34b) [0x576fab]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 13: (Paxos::do_refresh()+0x4c) [0x5b386c]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 14: (Paxos::handle_commit(MMonPaxos*)+0x243) [0x5ba7d3]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 15: (Paxos::dispatch(PaxosServiceMessage*)+0x22b) [0x5c30ab]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 16: (Monitor::dispatch(MonSession*, Message*, bool)+0x8c3) [0x598443]
2014-12-22T11:28:03.283 INFO:tasks.ceph.mon.c.plana58.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x2a4) [0x598884]
2014-12-22T11:28:03.284 INFO:tasks.ceph.mon.c.plana58.stderr: 18: (Monitor::ms_dispatch(Message*)+0x32) [0x5b2b62]
2014-12-22T11:28:03.284 INFO:tasks.ceph.mon.c.plana58.stderr: 19: (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x88a087]
2014-12-22T11:28:03.284 INFO:tasks.ceph.mon.c.plana58.stderr: 20: (DispatchQueue::entry()+0x44a) [0x88722a]
2014-12-22T11:28:03.284 INFO:tasks.ceph.mon.c.plana58.stderr: 21: (DispatchQueue::DispatchThread::entry()+0xd) [0x780dad]
2014-12-22T11:28:03.284 INFO:tasks.ceph.mon.c.plana58.stderr: 22: (()+0x7e9a) [0x7f4b375a2e9a]
2014-12-22T11:28:03.284 INFO:tasks.ceph.mon.c.plana58.stderr: 23: (clone()+0x6d) [0x7f4b35d4c31d]
2014-12-22T11:28:03.284 INFO:tasks.ceph.mon.c.plana58.stderr:2014-12-22 11:28:03.183647 7f4b30faa700 -1 *** Caught signal (Aborted) **
2014-12-22T11:28:03.284 INFO:tasks.ceph.mon.c.plana58.stderr: in thread 7f4b30faa700
2014-12-22T11:28:03.285 INFO:tasks.ceph.mon.c.plana58.stderr:
2014-12-22T11:28:03.285 INFO:tasks.ceph.mon.c.plana58.stderr: ceph version 0.90-586-g4047f2d (4047f2ddef22794fad1e97bb852151e4c6b3cf82)
2014-12-22T11:28:03.285 INFO:tasks.ceph.mon.c.plana58.stderr: 1: ceph-mon() [0x8df645]
2014-12-22T11:28:03.285 INFO:tasks.ceph.mon.c.plana58.stderr: 2: (()+0xfcb0) [0x7f4b375aacb0]
2014-12-22T11:28:03.285 INFO:tasks.ceph.mon.c.plana58.stderr: 3: (gsignal()+0x35) [0x7f4b35c8e0d5]
2014-12-22T11:28:03.285 INFO:tasks.ceph.mon.c.plana58.stderr: 4: (abort()+0x17b) [0x7f4b35c9183b]
2014-12-22T11:28:03.285 INFO:tasks.ceph.mon.c.plana58.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4b365e069d]
2014-12-22T11:28:03.286 INFO:tasks.ceph.mon.c.plana58.stderr: 6: (()+0xb5846) [0x7f4b365de846]
2014-12-22T11:28:03.286 INFO:tasks.ceph.mon.c.plana58.stderr: 7: (()+0xb5873) [0x7f4b365de873]
2014-12-22T11:28:03.286 INFO:tasks.ceph.mon.c.plana58.stderr: 8: (()+0xb596e) [0x7f4b365de96e]
2014-12-22T11:28:03.286 INFO:tasks.ceph.mon.c.plana58.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0x7984f9]
2014-12-22T11:28:03.286 INFO:tasks.ceph.mon.c.plana58.stderr: 10: (OSDMonitor::update_from_paxos(bool*)+0x2dcb) [0x5eab0b]
2014-12-22T11:28:03.286 INFO:tasks.ceph.mon.c.plana58.stderr: 11: (PaxosService::refresh(bool*)+0x3c6) [0x5cba06]
2014-12-22T11:28:03.286 INFO:tasks.ceph.mon.c.plana58.stderr: 12: (Monitor::refresh_from_paxos(bool*)+0x34b) [0x576fab]
2014-12-22T11:28:03.286 INFO:tasks.ceph.mon.c.plana58.stderr: 13: (Paxos::do_refresh()+0x4c) [0x5b386c]
2014-12-22T11:28:03.287 INFO:tasks.ceph.mon.c.plana58.stderr: 14: (Paxos::handle_commit(MMonPaxos*)+0x243) [0x5ba7d3]
2014-12-22T11:28:03.287 INFO:tasks.ceph.mon.c.plana58.stderr: 15: (Paxos::dispatch(PaxosServiceMessage*)+0x22b) [0x5c30ab]
2014-12-22T11:28:03.287 INFO:tasks.ceph.mon.c.plana58.stderr: 16: (Monitor::dispatch(MonSession*, Message*, bool)+0x8c3) [0x598443]
2014-12-22T11:28:03.287 INFO:tasks.ceph.mon.c.plana58.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x2a4) [0x598884]
2014-12-22T11:28:03.287 INFO:tasks.ceph.mon.c.plana58.stderr: 18: (Monitor::ms_dispatch(Message*)+0x32) [0x5b2b62]
2014-12-22T11:28:03.287 INFO:tasks.ceph.mon.c.plana58.stderr: 19: (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x88a087]
2014-12-22T11:28:03.287 INFO:tasks.ceph.mon.c.plana58.stderr: 20: (DispatchQueue::entry()+0x44a) [0x88722a]
2014-12-22T11:28:03.287 INFO:tasks.ceph.mon.c.plana58.stderr: 21: (DispatchQueue::DispatchThread::entry()+0xd) [0x780dad]
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr: 22: (()+0x7e9a) [0x7f4b375a2e9a]
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr: 23: (clone()+0x6d) [0x7f4b35d4c31d]
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr:
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr:     0> 2014-12-22 11:28:03.183647 7f4b30faa700 -1 *** Caught signal (Aborted) **
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr: in thread 7f4b30faa700
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr:
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr: ceph version 0.90-586-g4047f2d (4047f2ddef22794fad1e97bb852151e4c6b3cf82)
2014-12-22T11:28:03.288 INFO:tasks.ceph.mon.c.plana58.stderr: 1: ceph-mon() [0x8df645]
2014-12-22T11:28:03.289 INFO:tasks.ceph.mon.c.plana58.stderr: 2: (()+0xfcb0) [0x7f4b375aacb0]
2014-12-22T11:28:03.289 INFO:tasks.ceph.mon.c.plana58.stderr: 3: (gsignal()+0x35) [0x7f4b35c8e0d5]
2014-12-22T11:28:03.289 INFO:tasks.ceph.mon.c.plana58.stderr: 4: (abort()+0x17b) [0x7f4b35c9183b]
2014-12-22T11:28:03.289 INFO:tasks.ceph.mon.c.plana58.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4b365e069d]
2014-12-22T11:28:03.289 INFO:tasks.ceph.mon.c.plana58.stderr: 6: (()+0xb5846) [0x7f4b365de846]
2014-12-22T11:28:03.289 INFO:tasks.ceph.mon.c.plana58.stderr: 7: (()+0xb5873) [0x7f4b365de873]
2014-12-22T11:28:03.289 INFO:tasks.ceph.mon.c.plana58.stderr: 8: (()+0xb596e) [0x7f4b365de96e]
2014-12-22T11:28:03.289 INFO:tasks.ceph.mon.c.plana58.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0x7984f9]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 10: (OSDMonitor::update_from_paxos(bool*)+0x2dcb) [0x5eab0b]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 11: (PaxosService::refresh(bool*)+0x3c6) [0x5cba06]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 12: (Monitor::refresh_from_paxos(bool*)+0x34b) [0x576fab]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 13: (Paxos::do_refresh()+0x4c) [0x5b386c]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 14: (Paxos::handle_commit(MMonPaxos*)+0x243) [0x5ba7d3]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 15: (Paxos::dispatch(PaxosServiceMessage*)+0x22b) [0x5c30ab]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 16: (Monitor::dispatch(MonSession*, Message*, bool)+0x8c3) [0x598443]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x2a4) [0x598884]
2014-12-22T11:28:03.290 INFO:tasks.ceph.mon.c.plana58.stderr: 18: (Monitor::ms_dispatch(Message*)+0x32) [0x5b2b62]
2014-12-22T11:28:03.291 INFO:tasks.ceph.mon.c.plana58.stderr: 19: (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x88a087]
2014-12-22T11:28:03.291 INFO:tasks.ceph.mon.c.plana58.stderr: 20: (DispatchQueue::entry()+0x44a) [0x88722a]
2014-12-22T11:28:03.291 INFO:tasks.ceph.mon.c.plana58.stderr: 21: (DispatchQueue::DispatchThread::entry()+0xd) [0x780dad]
2014-12-22T11:28:03.291 INFO:tasks.ceph.mon.c.plana58.stderr: 22: (()+0x7e9a) [0x7f4b375a2e9a]
2014-12-22T11:28:03.291 INFO:tasks.ceph.mon.c.plana58.stderr: 23: (clone()+0x6d) [0x7f4b35d4c31d]
2014-12-22T11:28:03.291 INFO:tasks.ceph.mon.c.plana58.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Actions
Copy link

Also available in: Atom PDF