-5> 2015-06-17 06:35:00.761215 7f1bff4cf700 0 log_channel(cluster) log [INF] : mon.vs64@0 won leader election with quorum 0,3,4 -4> 2015-06-17 06:35:00.767644 7f1bff4cf700 0 log_channel(cluster) log [INF] : HEALTH_WARN; 103 pgs degraded; 16 pgs down; 42 pgs peering; 223 pgs stale; 103 pgs stuck degraded; 60 pgs stuck inactive; 223 pgs stuck stale; 512 pgs stuck unclean; 103 pgs stuck undersized; 103 pgs undersized; recovery 1429/12927 objects degraded (11.054%); recovery 10423/12927 objects misplaced (80.630%); too few PGs per OSD (21 < min 30); 2 mons down, quorum 0,3,4 vs64,vs67,vs68 -3> 2015-06-17 06:35:00.775910 7f1bfecce700 0 log_channel(cluster) log [INF] : monmap e1: 5 mons at {vs64=10.1.128.6:6789/0,vs65=10.1.128.7:6789/0,vs66=10.1.128.8:6789/0,vs67=10.1.128.9:6789/0,vs68=10.1.128.10:6789/0} -2> 2015-06-17 06:35:00.776424 7f1bfecce700 0 log_channel(cluster) log [INF] : pgmap v41593: 512 pgs: 91 stale+active+remapped, 31 active+undersized+degraded+remapped, 26 stale+remapped+peering, 2 stale+down+peering, 258 active+remapped, 14 stale+active+undersized+degraded, 14 stale+down+remapped+peering, 18 stale+remapped, 58 stale+active+undersized+degraded+remapped; 4309 MB data, 14193 MB used, 2382 GB / 2396 GB avail; 1429/12927 objects degraded (11.054%); 10423/12927 objects misplaced (80.630%) -1> 2015-06-17 06:35:00.776520 7f1bfecce700 0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up 0> 2015-06-17 06:35:00.784275 7f1bfecce700 -1 *** Caught signal (Segmentation fault) ** in thread 7f1bfecce700 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: /usr/bin/ceph-mon() [0xa5c0fc] 2: (()+0xf0a0) [0x7f1c05bb50a0] 3: /usr/bin/ceph-mon() [0x901884] 4: (crush_do_rule()+0x390) [0x9024c0] 5: (CrushWrapper::do_rule(int, int, std::vector >&, int, std::vector > const&) const+0x8b) [0x862b6b] 6: (OSDMap::_pg_to_osds(pg_pool_t const&, pg_t, std::vector >*, int*, unsigned int*) const+0x7c) [0x8500ac] 7: (OSDMap::pg_to_raw_up(pg_t, std::vector >*, int*) const+0xa4) [0x850524] 8: (OSDMap::remove_redundant_temporaries(CephContext*, OSDMap const&, OSDMap::Incremental*)+0x29b) [0x85086b] 9: (OSDMonitor::create_pending()+0xca) [0x6e5e7a] 10: (PaxosService::_active()+0x619) [0x6c3f49] 11: (Context::complete(int)+0x9) [0x6902e9] 12: (void finish_contexts(CephContext*, std::list >&, int)+0xa4) [0x69be14] 13: (Paxos::finish_round()+0xbe) [0x6b8ace] 14: (Paxos::handle_last(MMonPaxos*)+0xbec) [0x6bbf0c] 15: (Paxos::dispatch(PaxosServiceMessage*)+0x1db) [0x6bd41b] 16: (Monitor::dispatch(MonSession*, Message*, bool)+0x82e) [0x68ed7e] 17: (Monitor::_ms_dispatch(Message*)+0x1a2) [0x68f092] 18: (Monitor::ms_dispatch(Message*)+0x32) [0x6ad2d2] 19: (DispatchQueue::entry()+0x45b) [0x9d4e2b] 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x86c05d] 21: (()+0x6b50) [0x7f1c05bacb50] 22: (clone()+0x6d) [0x7f1c045c895d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mon.vs64.log