Actions
Bug #1440
closedmon: irtual void PGMonitor::encode_pending(ceph::bufferlist&)
% Done:
0%
Spent time:
Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
After upgrading to v0.33 I'm encountering:
mon/PGMonitor.cc: In function 'virtual void PGMonitor::encode_pending(ceph::bufferlist&)', in thread '0x7f7ce1f5b760' mon/PGMonitor.cc: 210: FAILED assert(paxos->get_version() + 1 == pending_inc.version) ceph version 0.33 (commit:340b4348bf620fe81f4c0d3a685fdc832663a4cf) 1: (PGMonitor::encode_pending(ceph::buffer::list&)+0x108) [0x4bbec8] 2: (PaxosService::propose_pending()+0xc6) [0x483326] 3: (PGMonitor::check_osd_map(unsigned int)+0xca9) [0x4c60d9] 4: (OSDMonitor::update_from_paxos()+0xb0b) [0x49412b] 5: (PaxosService::_active()+0x39) [0x484319] 6: (PaxosService::election_finished()+0x1b6) [0x484676] 7: (Monitor::win_election(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&)+0x234) [0x469b24] 8: (Monitor::win_standalone_election()+0x142) [0x469cb2] 9: (Monitor::init()+0x2a0) [0x469fc0] 10: (main()+0x1eda) [0x45affa] 11: (__libc_start_main()+0xed) [0x7f7ce04af30d] 12: /usr/bin/cmon() [0x45ca89]
I've raised the debug level (debug mon = 20) and this is showing:
2011-08-24 15:54:08.033926 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs scanning pg_pool(rep pg_size 3 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 lpg_num 0 lpgp_num 0 last_change 44 owner 0) 2011-08-24 15:54:08.033945 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 12.0 2011-08-24 15:54:08.033959 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 12.1 2011-08-24 15:54:08.033971 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 12.2 2011-08-24 15:54:08.033984 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 12.3 2011-08-24 15:54:08.033997 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 12.4 2011-08-24 15:54:08.034010 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 12.5 2011-08-24 15:54:08.034022 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 12.6 2011-08-24 15:54:08.034035 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 12.7 2011-08-24 15:54:08.034048 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs scanning pg_pool(rep pg_size 3 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 lpg_num 0 lpgp_num 0 last_change 45 owner 0) 2011-08-24 15:54:08.034061 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 13.0 2011-08-24 15:54:08.034074 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 13.1 2011-08-24 15:54:08.034091 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 13.2 2011-08-24 15:54:08.034104 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 13.3 2011-08-24 15:54:08.034117 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 13.4 2011-08-24 15:54:08.034130 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 13.5 2011-08-24 15:54:08.034142 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 13.6 2011-08-24 15:54:08.034155 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs will create 13.7 2011-08-24 15:54:08.034167 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs registered 8008 new pgs, removed 0 uncreated pgs 2011-08-24 15:54:08.034182 7f7ce1f5b760 mon.dga@0(leader).pg v0 encode_pending v 1 mon/PGMonitor.cc: In function 'virtual void PGMonitor::encode_pending(ceph::bufferlist&)', in thread '0x7f7ce1f5b760' mon/PGMonitor.cc: 210: FAILED assert(paxos->get_version() + 1 == pending_inc.version) ceph version 0.33 (commit:340b4348bf620fe81f4c0d3a685fdc832663a4cf) 1: (PGMonitor::encode_pending(ceph::buffer::list&)+0x108) [0x4bbec8] 2: (PaxosService::propose_pending()+0xc6) [0x483326] 3: (PGMonitor::check_osd_map(unsigned int)+0xca9) [0x4c60d9] 4: (OSDMonitor::update_from_paxos()+0xb0b) [0x49412b] 5: (PaxosService::_active()+0x39) [0x484319] 6: (PaxosService::election_finished()+0x1b6) [0x484676] 7: (Monitor::win_election(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&)+0x234) [0x469b24] 8: (Monitor::win_standalone_election()+0x142) [0x469cb2] 9: (Monitor::init()+0x2a0) [0x469fc0] 10: (main()+0x1eda) [0x45affa] 11: (__libc_start_main()+0xed) [0x7f7ce04af30d] 12: /usr/bin/cmon() [0x45ca89] ceph version 0.33 (commit:340b4348bf620fe81f4c0d3a685fdc832663a4cf) 1: (PGMonitor::encode_pending(ceph::buffer::list&)+0x108) [0x4bbec8] 2: (PaxosService::propose_pending()+0xc6) [0x483326] 3: (PGMonitor::check_osd_map(unsigned int)+0xca9) [0x4c60d9] 4: (OSDMonitor::update_from_paxos()+0xb0b) [0x49412b] 5: (PaxosService::_active()+0x39) [0x484319] 6: (PaxosService::election_finished()+0x1b6) [0x484676] 7: (Monitor::win_election(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&)+0x234) [0x469b24] 8: (Monitor::win_standalone_election()+0x142) [0x469cb2] 9: (Monitor::init()+0x2a0) [0x469fc0] 10: (main()+0x1eda) [0x45affa] 11: (__libc_start_main()+0xed) [0x7f7ce04af30d] 12: /usr/bin/cmon() [0x45ca89] *** Caught signal (Aborted) ** in thread 0x7f7ce1f5b760 ceph version 0.33 (commit:340b4348bf620fe81f4c0d3a685fdc832663a4cf) 1: /usr/bin/cmon() [0x58dbc6] 2: (()+0x10040) [0x7f7ce1b3e040] 3: (gsignal()+0x35) [0x7f7ce04c43a5] 4: (abort()+0x17b) [0x7f7ce04c7b0b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f7ce0d82e6d] 6: (()+0xba016) [0x7f7ce0d81016] 7: (()+0xba043) [0x7f7ce0d81043] 8: (()+0xba13e) [0x7f7ce0d8113e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x193) [0x56c2d3] 10: (PGMonitor::encode_pending(ceph::buffer::list&)+0x108) [0x4bbec8] 11: (PaxosService::propose_pending()+0xc6) [0x483326] 12: (PGMonitor::check_osd_map(unsigned int)+0xca9) [0x4c60d9] 13: (OSDMonitor::update_from_paxos()+0xb0b) [0x49412b] 14: (PaxosService::_active()+0x39) [0x484319] 15: (PaxosService::election_finished()+0x1b6) [0x484676] 16: (Monitor::win_election(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&)+0x234) [0x469b24] 17: (Monitor::win_standalone_election()+0x142) [0x469cb2] 18: (Monitor::init()+0x2a0) [0x469fc0] 19: (main()+0x1eda) [0x45affa] 20: (__libc_start_main()+0xed) [0x7f7ce04af30d] 21: /usr/bin/cmon() [0x45ca89]
At the moment of starting the monitor there were no online OSDs.
This cluster has one monitor.
Updated by Sage Weil over 12 years ago
- Status changed from New to Resolved
- Assignee set to Sage Weil
- Target version set to v0.34
Tracked this down last night.. should be fixed by eaaefece778b042b91d7c7e7fd8e00d418d7e93a
Actions