Ceph

Executed the following:

ubuntu@burnupi30:~$ sudo ceph health detail

Wait awhile and ceph will start to output the following:

2012-08-01 19:05:21.473145 7f8b0b5d3700 0 monclient: hunting for new mon

The backtrace from to machine (burnupi32) whose monitor died can be found here:

ceph version 0.48argonaut-49-g05c5b84 (commit:05c5b8407a880dc95f8a8917ac2105e5198268dd)
 1: /usr/bin/ceph-mon() [0x530aea]
 2: (()+0x10060) [0x7f6d1e0ae060]
 3: (PGMonitor::check_full_osd_health(std::list<std::pair<health_status_t, std::string>, std::allocator<std::pair<health_status_t, std::string> > >&, std::list<std::pair<health_status_t, std::string>, std::allocator<std::pair<health_status_t, std::string> > >, std::set<int, std::less<int>, std::allocator<int> > const&, char const, health_status_t) const+0x41d) [0x4e563d]
 4: (PGMonitor::get_health(std::list<std::pair<health_status_t, std::string>, std::allocator<std::pair<health_status_t, std::string> > >&, std::list<std::pair<health_status_t, std::string>, std::allocator<std::pair<health_status_t, std::string> > >) const+0x13cf) [0x4e72bf]
 5: (Monitor::get_health(std::string&, ceph::buffer::list)+0x76) [0x476e06]
 6: (Monitor::handle_command(MMonCommand*)+0xe2e) [0x478c5e]
 7: (Monitor::_ms_dispatch(Message*)+0x111b) [0x48805b]
 8: (Monitor::ms_dispatch(Message*)+0x32) [0x495aa2]
 9: (SimpleMessenger::dispatch_entry()+0x92b) [0x5cdbcb]
 10: (SimpleMessenger::DispatchThread::entry()+0xd) [0x59c70d]
 11: (()+0x7efc) [0x7f6d1e0a5efc]
 12: (clone()+0x6d) [0x7f6d1c8da59d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2012-08-01 18:55:37.303799 7f6d091e2700  5 throttle(msgr_dispatch_throttler-mon 0x239f9d0) get 74 (763 > 837)
-- end dump of recent events ---