Ceph

Tamil hit this last night.

Have been unable to reproduce it, but the only scenario in which this would happen is when somehow ::statfs() fails. My guess is that this was triggered during some weird behavior on teuthology's upgrade task, but that's just a theory.

Creating the ticket for future reference. Pushing a patch shortly, which will force the monitor to cleanly shutdown if it is unable to obtain stats from his own disk -- if something like that goes wrong, we sure want to shutdown the monitor asap before things get worse.

2013-03-19T14:10:22.109 INFO:teuthology.task.ceph.mon.c.err:mon/DataHealthService.cc: In function 'void DataHealthService:hare_stats()' thread 7f8310a33700 time 2013-03-19 14:10:22.238341
2013-03-19T14:10:22.109 INFO:teuthology.task.ceph.mon.c.err:mon/DataHealthService.cc: 134: FAILED assert(!stats.empty())
2013-03-19T14:10:22.110 INFO:teuthology.task.ceph.mon.c.err: ceph version 0.58-754-gb7e2a0d (b7e2a0d464d63ee00e3c39a1376fd7fc4894c938)
2013-03-19T14:10:22.110 INFO:teuthology.task.ceph.mon.c.err: 1: (DataHealthService:hare_stats()+0xacb) [0x57ad5b]
2013-03-19T14:10:22.110 INFO:teuthology.task.ceph.mon.c.err: 2: (DataHealthService:ervice_tick()+0x2b8) [0x57b4c8]
2013-03-19T14:10:22.110 INFO:teuthology.task.ceph.mon.c.err: 3: (QuorumService::C_Tick::finish(int)+0x17) [0x57bf87]
2013-03-19T14:10:22.110 INFO:teuthology.task.ceph.mon.c.err: 4: (Context::complete(int)+0xa) [0x4c1d9a]
2013-03-19T14:10:22.110 INFO:teuthology.task.ceph.mon.c.err: 5: (SafeTimer::timer_thread()+0x425) [0x633ff5]
2013-03-19T14:10:22.111 INFO:teuthology.task.ceph.mon.c.err: 6: (SafeTimerThread::entry()+0xd) [0x634c2d]
2013-03-19T14:10:22.111 INFO:teuthology.task.ceph.mon.c.err: 7: (()+0x7e9a) [0x7f8314f95e9a]
2013-03-19T14:10:22.111 INFO:teuthology.task.ceph.mon.c.err: 8: (clone()+0x6d) [0x7f8313545cbd]
2013-03-19T14:10:22.111 INFO:teuthology.task.ceph.mon.c.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2013-03-19T14:10:22.113 INFO:teuthology.task.ceph.mon.c.err:2013-03-19 14:10:22.239510 7f8310a33700 -1 mon/DataHealthService.cc: In function 'void DataHealthService:hare_stats()' thread 7f8310a33700 time 2013-03-19 14:10:22.238341