Project

General

Profile

Bug #1440

mon: irtual void PGMonitor::encode_pending(ceph::bufferlist&)

Added by Wido den Hollander over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Monitor
Target version:
% Done:

0%

Spent time:
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

After upgrading to v0.33 I'm encountering:

mon/PGMonitor.cc: In function 'virtual void PGMonitor::encode_pending(ceph::bufferlist&)', in thread '0x7f7ce1f5b760'
mon/PGMonitor.cc: 210: FAILED assert(paxos->get_version() + 1 == pending_inc.version)
 ceph version 0.33 (commit:340b4348bf620fe81f4c0d3a685fdc832663a4cf)
 1: (PGMonitor::encode_pending(ceph::buffer::list&)+0x108) [0x4bbec8]
 2: (PaxosService::propose_pending()+0xc6) [0x483326]
 3: (PGMonitor::check_osd_map(unsigned int)+0xca9) [0x4c60d9]
 4: (OSDMonitor::update_from_paxos()+0xb0b) [0x49412b]
 5: (PaxosService::_active()+0x39) [0x484319]
 6: (PaxosService::election_finished()+0x1b6) [0x484676]
 7: (Monitor::win_election(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&)+0x234) [0x469b24]
 8: (Monitor::win_standalone_election()+0x142) [0x469cb2]
 9: (Monitor::init()+0x2a0) [0x469fc0]
 10: (main()+0x1eda) [0x45affa]
 11: (__libc_start_main()+0xed) [0x7f7ce04af30d]
 12: /usr/bin/cmon() [0x45ca89]

I've raised the debug level (debug mon = 20) and this is showing:

2011-08-24 15:54:08.033926 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs scanning pg_pool(rep pg_size 3 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 lpg_num 0 lpgp_num 0 last_change 44 owner 0)
2011-08-24 15:54:08.033945 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 12.0
2011-08-24 15:54:08.033959 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 12.1
2011-08-24 15:54:08.033971 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 12.2
2011-08-24 15:54:08.033984 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 12.3
2011-08-24 15:54:08.033997 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 12.4
2011-08-24 15:54:08.034010 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 12.5
2011-08-24 15:54:08.034022 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 12.6
2011-08-24 15:54:08.034035 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 12.7
2011-08-24 15:54:08.034048 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs scanning pg_pool(rep pg_size 3 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 lpg_num 0 lpgp_num 0 last_change 45 owner 0)
2011-08-24 15:54:08.034061 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 13.0
2011-08-24 15:54:08.034074 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 13.1
2011-08-24 15:54:08.034091 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 13.2
2011-08-24 15:54:08.034104 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 13.3
2011-08-24 15:54:08.034117 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 13.4
2011-08-24 15:54:08.034130 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 13.5
2011-08-24 15:54:08.034142 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 13.6
2011-08-24 15:54:08.034155 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs  will create 13.7
2011-08-24 15:54:08.034167 7f7ce1f5b760 mon.dga@0(leader).pg v0 register_new_pgs registered 8008 new pgs, removed 0 uncreated pgs
2011-08-24 15:54:08.034182 7f7ce1f5b760 mon.dga@0(leader).pg v0 encode_pending v 1
mon/PGMonitor.cc: In function 'virtual void PGMonitor::encode_pending(ceph::bufferlist&)', in thread '0x7f7ce1f5b760'
mon/PGMonitor.cc: 210: FAILED assert(paxos->get_version() + 1 == pending_inc.version)
 ceph version 0.33 (commit:340b4348bf620fe81f4c0d3a685fdc832663a4cf)
 1: (PGMonitor::encode_pending(ceph::buffer::list&)+0x108) [0x4bbec8]
 2: (PaxosService::propose_pending()+0xc6) [0x483326]
 3: (PGMonitor::check_osd_map(unsigned int)+0xca9) [0x4c60d9]
 4: (OSDMonitor::update_from_paxos()+0xb0b) [0x49412b]
 5: (PaxosService::_active()+0x39) [0x484319]
 6: (PaxosService::election_finished()+0x1b6) [0x484676]
 7: (Monitor::win_election(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&)+0x234) [0x469b24]
 8: (Monitor::win_standalone_election()+0x142) [0x469cb2]
 9: (Monitor::init()+0x2a0) [0x469fc0]
 10: (main()+0x1eda) [0x45affa]
 11: (__libc_start_main()+0xed) [0x7f7ce04af30d]
 12: /usr/bin/cmon() [0x45ca89]
 ceph version 0.33 (commit:340b4348bf620fe81f4c0d3a685fdc832663a4cf)
 1: (PGMonitor::encode_pending(ceph::buffer::list&)+0x108) [0x4bbec8]
 2: (PaxosService::propose_pending()+0xc6) [0x483326]
 3: (PGMonitor::check_osd_map(unsigned int)+0xca9) [0x4c60d9]
 4: (OSDMonitor::update_from_paxos()+0xb0b) [0x49412b]
 5: (PaxosService::_active()+0x39) [0x484319]
 6: (PaxosService::election_finished()+0x1b6) [0x484676]
 7: (Monitor::win_election(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&)+0x234) [0x469b24]
 8: (Monitor::win_standalone_election()+0x142) [0x469cb2]
 9: (Monitor::init()+0x2a0) [0x469fc0]
 10: (main()+0x1eda) [0x45affa]
 11: (__libc_start_main()+0xed) [0x7f7ce04af30d]
 12: /usr/bin/cmon() [0x45ca89]
*** Caught signal (Aborted) **
 in thread 0x7f7ce1f5b760
 ceph version 0.33 (commit:340b4348bf620fe81f4c0d3a685fdc832663a4cf)
 1: /usr/bin/cmon() [0x58dbc6]
 2: (()+0x10040) [0x7f7ce1b3e040]
 3: (gsignal()+0x35) [0x7f7ce04c43a5]
 4: (abort()+0x17b) [0x7f7ce04c7b0b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f7ce0d82e6d]
 6: (()+0xba016) [0x7f7ce0d81016]
 7: (()+0xba043) [0x7f7ce0d81043]
 8: (()+0xba13e) [0x7f7ce0d8113e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x193) [0x56c2d3]
 10: (PGMonitor::encode_pending(ceph::buffer::list&)+0x108) [0x4bbec8]
 11: (PaxosService::propose_pending()+0xc6) [0x483326]
 12: (PGMonitor::check_osd_map(unsigned int)+0xca9) [0x4c60d9]
 13: (OSDMonitor::update_from_paxos()+0xb0b) [0x49412b]
 14: (PaxosService::_active()+0x39) [0x484319]
 15: (PaxosService::election_finished()+0x1b6) [0x484676]
 16: (Monitor::win_election(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&)+0x234) [0x469b24]
 17: (Monitor::win_standalone_election()+0x142) [0x469cb2]
 18: (Monitor::init()+0x2a0) [0x469fc0]
 19: (main()+0x1eda) [0x45affa]
 20: (__libc_start_main()+0xed) [0x7f7ce04af30d]
 21: /usr/bin/cmon() [0x45ca89]

At the moment of starting the monitor there were no online OSDs.

This cluster has one monitor.

History

#1 Updated by Sage Weil over 8 years ago

  • Status changed from New to Resolved
  • Assignee set to Sage Weil
  • Target version set to v0.34

Tracked this down last night.. should be fixed by eaaefece778b042b91d7c7e7fd8e00d418d7e93a

Also available in: Atom PDF