Ceph

It crashes after about 15-30 minutes of working. Reproduced many times.

ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)

Here is log:

2013-03-18 17:17:59.834820 7f5c239f8700 -1 *** Caught signal (Aborted) **
 in thread 7f5c239f8700

 ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
 1: /usr/bin/ceph-osd() [0x790e59]
 2: (()+0xf500) [0x7f5c45d54500]
 3: (gsignal()+0x35) [0x7f5c44a208a5]
 4: (abort()+0x175) [0x7f5c44a22085]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f5c452d9a5d]
 6: (()+0xbcbe6) [0x7f5c452d7be6]
 7: (()+0xbcc13) [0x7f5c452d7c13]
 8: (()+0xbcd0e) [0x7f5c452d7d0e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7c9) [0x83d699]
 10: (SyncEntryTimeout::finish(int)+0xbf) [0x7357bf]
 11: (SafeTimer::timer_thread()+0x453) [0x854f23]
 12: (SafeTimerThread::entry()+0xd) [0x8570ed]
 13: (()+0x7851) [0x7f5c45d4c851]
 14: (clone()+0x6d) [0x7f5c44ad611d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
    -3> 2013-03-18 17:17:59.322403 7f5c40a8a700  5 osd.2 241 tick
    -2> 2013-03-18 17:17:59.825744 7f5c1fdf2700  1 -- [2a01:4f8:190:73ae::2]:6803/54197 <== osd.3 [2a01:4f8:100:9363::2]:0/22824 734 ==== osd_ping(ping e241 stamp 2013-03-18 17:17:59.822076) v2 ==== 47+0+0 (2616145970 0 0) 0x7f5bbc000ea0 con 0x7f5bc4000b10
    -1> 2013-03-18 17:17:59.825796 7f5c1fdf2700  1 -- [2a01:4f8:190:73ae::2]:6803/54197 --> [2a01:4f8:100:9363::2]:0/22824 -- osd_ping(ping_reply e241 stamp 2013-03-18 17:17:59.822076) v2 -- ?+0 0x7f5b980008c0 con 0x7f5bc4000b10
     0> 2013-03-18 17:17:59.834820 7f5c239f8700 -1 *** Caught signal (Aborted) **
 in thread 7f5c239f8700

 ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
 1: /usr/bin/ceph-osd() [0x790e59]
 2: (()+0xf500) [0x7f5c45d54500]
 3: (gsignal()+0x35) [0x7f5c44a208a5]
 4: (abort()+0x175) [0x7f5c44a22085]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f5c452d9a5d]
 6: (()+0xbcbe6) [0x7f5c452d7be6]
 7: (()+0xbcc13) [0x7f5c452d7c13]
 8: (()+0xbcd0e) [0x7f5c452d7d0e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7c9) [0x83d699]
 10: (SyncEntryTimeout::finish(int)+0xbf) [0x7357bf]
 11: (SafeTimer::timer_thread()+0x453) [0x854f23]
 12: (SafeTimerThread::entry()+0xd) [0x8570ed]
 13: (()+0x7851) [0x7f5c45d4c851]
 14: (clone()+0x6d) [0x7f5c44ad611d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   0/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 hadoop
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent    100000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---