Actions
Bug #4488
closedceph-osd crash on server under heavy load
Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
It crashes after about 15-30 minutes of working. Reproduced many times.
ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
Here is log:
2013-03-18 17:17:59.834820 7f5c239f8700 -1 *** Caught signal (Aborted) ** in thread 7f5c239f8700 ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5) 1: /usr/bin/ceph-osd() [0x790e59] 2: (()+0xf500) [0x7f5c45d54500] 3: (gsignal()+0x35) [0x7f5c44a208a5] 4: (abort()+0x175) [0x7f5c44a22085] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f5c452d9a5d] 6: (()+0xbcbe6) [0x7f5c452d7be6] 7: (()+0xbcc13) [0x7f5c452d7c13] 8: (()+0xbcd0e) [0x7f5c452d7d0e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7c9) [0x83d699] 10: (SyncEntryTimeout::finish(int)+0xbf) [0x7357bf] 11: (SafeTimer::timer_thread()+0x453) [0x854f23] 12: (SafeTimerThread::entry()+0xd) [0x8570ed] 13: (()+0x7851) [0x7f5c45d4c851] 14: (clone()+0x6d) [0x7f5c44ad611d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -3> 2013-03-18 17:17:59.322403 7f5c40a8a700 5 osd.2 241 tick -2> 2013-03-18 17:17:59.825744 7f5c1fdf2700 1 -- [2a01:4f8:190:73ae::2]:6803/54197 <== osd.3 [2a01:4f8:100:9363::2]:0/22824 734 ==== osd_ping(ping e241 stamp 2013-03-18 17:17:59.822076) v2 ==== 47+0+0 (2616145970 0 0) 0x7f5bbc000ea0 con 0x7f5bc4000b10 -1> 2013-03-18 17:17:59.825796 7f5c1fdf2700 1 -- [2a01:4f8:190:73ae::2]:6803/54197 --> [2a01:4f8:100:9363::2]:0/22824 -- osd_ping(ping_reply e241 stamp 2013-03-18 17:17:59.822076) v2 -- ?+0 0x7f5b980008c0 con 0x7f5bc4000b10 0> 2013-03-18 17:17:59.834820 7f5c239f8700 -1 *** Caught signal (Aborted) ** in thread 7f5c239f8700 ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5) 1: /usr/bin/ceph-osd() [0x790e59] 2: (()+0xf500) [0x7f5c45d54500] 3: (gsignal()+0x35) [0x7f5c44a208a5] 4: (abort()+0x175) [0x7f5c44a22085] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f5c452d9a5d] 6: (()+0xbcbe6) [0x7f5c452d7be6] 7: (()+0xbcc13) [0x7f5c452d7c13] 8: (()+0xbcd0e) [0x7f5c452d7d0e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7c9) [0x83d699] 10: (SyncEntryTimeout::finish(int)+0xbf) [0x7357bf] 11: (SafeTimer::timer_thread()+0x453) [0x854f23] 12: (SafeTimerThread::entry()+0xd) [0x8570ed] 13: (()+0x7851) [0x7f5c45d4c851] 14: (clone()+0x6d) [0x7f5c44ad611d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 100000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events ---
Files
Actions