Ceph

Run: http://pulpito.ceph.com/teuthology-2015-10-06_17:02:02-upgrade:hammer-x-infernalis-distro-basic-multi/
Job: 1092041
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-06_17:02:02-upgrade:hammer-x-infernalis-distro-basic-multi/1092041/teuthology.log

2015-10-09T00:55:12.133 INFO:tasks.mon_thrash:joining mon_thrasher
2015-10-09T00:55:12.133 ERROR:teuthology.run_tasks:Manager failed: mon_thrash
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_infernalis/tasks/mon_thrash.py", line 341, in task
    thrash_proc.do_join()
  File "/var/lib/teuthworker/src/ceph-qa-suite_infernalis/tasks/mon_thrash.py", line 143, in do_join
    self.thread.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get
    raise self._exception
AssertionError: failed to reach quorum size 3 before timeout expired

Crash: failed to reach quorum size 3 before timeout expired
ceph version 9.0.3-2052-ge5a6529 (e5a6529afc62e27afd1a9fd2caae8910e3a4cd86)
 1: (()+0x68cdaa) [0x7f669ad5edaa]
 2: (()+0x10340) [0x7f669956e340]
 3: (gsignal()+0x39) [0x7f6697a0fcc9]
 4: (abort()+0x148) [0x7f6697a130d8]
 5: (()+0x2fb86) [0x7f6697a08b86]
 6: (()+0x2fc32) [0x7f6697a08c32]
 7: (()+0x20f3f3) [0x7f669a8e13f3]
 8: (Monitor::remove_session(MonSession*)+0x852) [0x7f669a900dd2]
 9: (Monitor::waitlist_or_zap_client(std::shared_ptr<MonOpRequest>)+0x137) [0x7f669a9016a7]
 10: (Monitor::_ms_dispatch(Message*)+0x4fb) [0x7f669a91604b]
 11: (Monitor::handle_forward(std::shared_ptr<MonOpRequest>)+0x863) [0x7f669a9146f3]
 12: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xc0e) [0x7f669a91579e]
 13: (Monitor::_ms_dispatch(Message*)+0x628) [0x7f669a916178]
 14: (Monitor::ms_dispatch(Message*)+0x23) [0x7f669a934ab3]
 15: (DispatchQueue::entry()+0x63a) [0x7f669acafbda]
 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f669ab17b7d]
 17: (()+0x8182) [0x7f6699566182]
 18: (clone()+0x6d) [0x7f6697ad347d]