Ceph

Run: http://pulpito.ceph.com/teuthology-2015-10-06_17:10:09-upgrade:hammer-x-infernalis-distro-basic-vps/
Jobs: ['1092051', '1092057','1092049']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-06_17:10:09-upgrade:hammer-x-infernalis-distro-basic-vps/1092051/teuthology.log

2015-10-07T00:37:59.777 INFO:teuthology.orchestra.run.vpm078.stderr:2015-10-07 07:37:59.769738 7fd58c3d8700  0 -- :/622582173 >> 10.214.130.130:6789/0 pipe(0x7fd580096010 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd580061080).fault
2015-10-07T00:38:02.907 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":306,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","monmap":{"epoch":1,"fsid":"339e3331-e181-4408-955e-e6ad52dd3bef","modified":"2015-10-07 03:35:30.560657","created":"2015-10-07 03:35:30.560657","mons":[{"rank":0,"name":"a","addr":"10.214.130.78:6789\/0"},{"rank":1,"name":"b","addr":"10.214.130.130:6789\/0"},{"rank":2,"name":"c","addr":"10.214.130.130:6790\/0"}]}}

2015-10-07T00:38:02.908 ERROR:teuthology.run_tasks:Manager failed: mon_thrash
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_infernalis/tasks/mon_thrash.py", line 341, in task
    thrash_proc.do_join()
  File "/var/lib/teuthworker/src/ceph-qa-suite_infernalis/tasks/mon_thrash.py", line 143, in do_join
    self.thread.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 331, in get
    raise self._exception
AssertionError: failed to reach quorum size 3 before timeout expired

Crash: failed to reach quorum size 3 before timeout expired
ceph version 9.0.3-2052-ge5a6529 (e5a6529afc62e27afd1a9fd2caae8910e3a4cd86)
 1: (()+0x68cdaa) [0x7fa19c169daa]
 2: (()+0x10340) [0x7fa19a979340]
 3: (gsignal()+0x39) [0x7fa198e1acc9]
 4: (abort()+0x148) [0x7fa198e1e0d8]
 5: (()+0x2fb86) [0x7fa198e13b86]
 6: (()+0x2fc32) [0x7fa198e13c32]
 7: (()+0x20f3f3) [0x7fa19bcec3f3]
 8: (Monitor::remove_session(MonSession*)+0x852) [0x7fa19bd0bdd2]
 9: (Monitor::waitlist_or_zap_client(std::shared_ptr<MonOpRequest>)+0x137) [0x7fa19bd0c6a7]
 10: (Monitor::_ms_dispatch(Message*)+0x4fb) [0x7fa19bd2104b]
 11: (Monitor::handle_forward(std::shared_ptr<MonOpRequest>)+0x863) [0x7fa19bd1f6f3]
 12: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xc0e) [0x7fa19bd2079e]
 13: (Monitor::_ms_dispatch(Message*)+0x628) [0x7fa19bd21178]
 14: (Monitor::ms_dispatch(Message*)+0x23) [0x7fa19bd3fab3]
 15: (DispatchQueue::entry()+0x63a) [0x7fa19c0babda]
 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fa19bf22b7d]
 17: (()+0x8182) [0x7fa19a971182]
 18: (clone()+0x6d) [0x7fa198ede47d]