Project

General

Profile

mds.replay.fail.log

John Leach, 02/13/2011 12:49 PM

Download (7.91 KB)

 
1
2011-02-13 19:38:24.969592 7f06a6f0d700 mds0.snap check_osd_map - version unchanged
2
2011-02-13 19:38:24.969603 7f06a6f0d700 mds0.8 beacon_send up:active seq 2635 (currently up:active)
3
2011-02-13 19:38:24.969619 7f06a6f0d700 -- 10.202.105.222:6804/23070 --> mon0 10.135.211.78:6789/0 -- mdsbeacon(4308/srv-an56n up:active seq 2635 v41) v1 -- ?+0 0x43ce2500
4
2011-02-13 19:38:24.969637 7f06a6f0d700 -- 10.202.105.222:6804/23070 submit_message mdsbeacon(4308/srv-an56n up:active seq 2635 v41) v1 remote, 10.135.211.78:6789/0, have pipe.
5
2011-02-13 19:38:24.969672 7f06a6f0d700 mds0.8 beacon_kill last_acked_stamp 2011-02-13 19:37:18.905477, setting laggy flag.
6
2011-02-13 19:38:24.969756 7f06aaaad700 -- 10.202.105.222:6804/23070 >> 10.135.211.78:6789/0 pipe(0x223e280 sd=13 pgs=69 cs=1 l=1).writer: state = 2 policy.server=0
7
2011-02-13 19:38:24.969850 7f06aaaad700 -- 10.202.105.222:6804/23070 >> 10.135.211.78:6789/0 pipe(0x223e280 sd=13 pgs=69 cs=1 l=1).writer encoding 2730 0x43ce2500 mdsbeacon(4308/srv-an56n up:active seq 2635 v41) v1
8
2011-02-13 19:38:24.969934 7f06a8010700 mds0.8 handle_command args: [injectargs,--debug_ms 1 --debug_mds 10000]
9
2011-02-13 19:38:24.970029 7f06aaaad700 -- 10.202.105.222:6804/23070 >> 10.135.211.78:6789/0 pipe(0x223e280 sd=13 pgs=69 cs=1 l=1).writer sending 2730 0x43ce2500
10
2011-02-13 19:38:24.970122 7f06a8010700 -- 10.202.105.222:6804/23070 <== mon0 10.135.211.78:6789/0 2744 ==== mon_command(injectargs --debug_ms 1 --debug_mds 100 v 0) v1 ==== 84+0+0 (3487630235 0 0) 0x7ea0540 con 0x223f3c0
11
2011-02-13 19:38:24.970178 7f06a6f0d700 mds0.8  last tick was 39.299112 > 5 seconds ago, laggy_until 2011-02-13 17:37:11.364794, setting laggy flag
12
2011-02-13 19:38:24.970288 7f06a8010700 mds0.8 handle_command args: [injectargs,--debug_ms 1 --debug_mds 100]
13
2011-02-13 19:38:24.970332 7f06a8010700 -- 10.202.105.222:6804/23070 <== mon0 10.135.211.78:6789/0 2745 ==== mon_command(injectargs --debug_ms 1 --debug_mds 10 v 0) v1 ==== 83+0+0 (980379374 0 0) 0x7ea0700 con 0x223f3c0 
14
2011-02-13 19:38:24.970347 7f06a8010700 mds0.8 handle_command args: [injectargs,--debug_ms 1 --debug_mds 10]
15
2011-02-13 19:38:24.970374 7f06a8010700 -- 10.202.105.222:6804/23070 <== mon0 10.135.211.78:6789/0 2746 ==== osd_map(39,39) v1 ==== 284+0+0 (2678458743 0 0) 0x537ea00 con 0x223f3c0
16
2011-02-13 19:38:24.970389 7f06a8010700 mds0.8 laggy, deferring osd_map(39,39) v1
17
2011-02-13 19:38:24.970407 7f06a8010700 -- 10.202.105.222:6804/23070 <== mon0 10.135.211.78:6789/0 2747 ==== mdsmap(e 42) v1 ==== 530+0+0 (1817951688 0 0) 0x537e800 con 0x223f3c0
18
2011-02-13 19:38:24.976658 7f06a8010700 mds0.8 handle_mds_map epoch 42 from mon0
19
2011-02-13 19:38:24.991483 7f06a8010700 mds0.8      my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
20
2011-02-13 19:38:24.991517 7f06a8010700 mds0.8  mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate objec}
21
2011-02-13 19:38:24.991532 7f06a8010700 mds-1.8 map says i am 10.202.105.222:6804/23070 mds-1 state down:dne
22
2011-02-13 19:38:24.991543 7f06a8010700 mds-1.8 handle_mds_map i (10.202.105.222:6804/23070) dne in the mdsmap, respawning myself
23
2011-02-13 19:38:24.991554 7f06a8010700 mds-1.8 respawn
24
2011-02-13 19:38:24.991563 7f06a8010700 mds-1.8  e: '/usr/bin/cmds'
25
2011-02-13 19:38:24.991571 7f06a8010700 mds-1.8  0: '/usr/bin/cmds'
26
2011-02-13 19:38:24.991584 7f06a8010700 mds-1.8  1: '-i'
27
2011-02-13 19:38:24.991597 7f06a8010700 mds-1.8  2: 'srv-an56n'
28
2011-02-13 19:38:24.991609 7f06a8010700 mds-1.8  3: '-c'
29
2011-02-13 19:38:24.991618 7f06a8010700 mds-1.8  4: '/etc/ceph/ceph.conf'
30
2011-02-13 19:38:24.991660 7f06a8010700 mds-1.8  cwd /
31
2011-02-13 19:38:25.780830 7f31e5c28720 starting mds.srv-an56n at 0.0.0.0:6805/23353
32
2011-02-13 19:38:25.784792 7f31e5c28720 -- 0.0.0.0:6805/23353 messenger.start: error creating directory: '/': error 17: File exists
33
2011-02-13 19:38:25.791139 7f31e318a700 mds-1.0 ms_handle_connect on 10.202.105.222:6789/0
34
2011-02-13 19:38:29.801070 7f31e318a700 mds-1.0 handle_mds_map standby
35
2011-02-13 19:38:29.886928 7f31e318a700 mds0.10 handle_mds_map i am now mds0.10
36
2011-02-13 19:38:29.886962 7f31e318a700 mds0.10 handle_mds_map state change up:standby --> up:replay
37
2011-02-13 19:38:29.886974 7f31e318a700 mds0.10 replay_start
38
2011-02-13 19:38:29.886988 7f31e318a700 mds0.10  recovery set is 
39
2011-02-13 19:38:29.886997 7f31e318a700 mds0.10  need osdmap epoch 40, have 39
40
2011-02-13 19:38:29.887365 7f31e318a700 mds0.cache handle_mds_failure mds0 : recovery peers are 
41
2011-02-13 19:38:29.915904 7f31e318a700 mds0.10 ms_handle_connect on 10.202.105.222:6801/23135
42
2011-02-13 19:38:29.916525 7f31e318a700 mds0.10 ms_handle_connect on 10.67.65.62:6800/22211
43
2011-02-13 19:38:29.916896 7f31e318a700 mds0.10 ms_handle_connect on 10.61.136.222:6800/21630
44
2011-02-13 19:38:29.916921 7f31e318a700 mds0.10 ms_handle_connect on 10.135.211.78:6801/5657
45
2011-02-13 19:38:30.417439 7f31e318a700 mds0.cache creating system inode with ino:100
46
2011-02-13 19:38:30.420327 7f31e318a700 mds0.cache creating system inode with ino:1
47
2011-02-13 19:38:31.431785 7f31e087d700 mds0.journaler try_read_entry got 0 len entry at offset 4051698581
48
2011-02-13 19:38:31.431846 7f31e087d700 mds0.log _replay journaler got error -22, aborting
49
2011-02-13 19:38:31.445544 sdc/Journaler.h: In function 'void Journaler::init_headers(Journaler::Header&)', In thread 7f31e318a700
50
2011-02-13 19:38:31.445572 sdc/Journaler.h:225: FAILED assert(readonly || state == STATE_READHEAD)
51
2011-02-13 19:38:31.445584 ceph version 0.25~rc (commit:)
52
2011-02-13 19:38:31.451874 1: (Journaler::_finish_reread_head(int, ceph::buffer::list&, Context*)+0x1fd) [0x6baa1d]
53
2011-02-13 19:38:31.451898 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x951) [0x69dde1]
54
2011-02-13 19:38:31.451910 3: (MDS::_dispatch(Message*)+0x3004) [0x4bff34]
55
2011-02-13 19:38:31.451920 4: (MDS::ms_dispatch(Message*)+0x59) [0x4bff99]
56
2011-02-13 19:38:31.451929 5: (SimpleMessenger::dispatch_entry()+0x8a3) [0x488bf3]
57
2011-02-13 19:38:31.451937 6: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x483f3c]
58
2011-02-13 19:38:31.451946 7: (Thread::_entry_func(void*)+0xa) [0x49d1ea]
59
2011-02-13 19:38:31.451963 8: (()+0x69ca) [0x7f31e57ff9ca]
60
2011-02-13 19:38:31.451972 9: (clone()+0x6d) [0x7f31e448c70d]
61
2011-02-13 19:38:31.451979 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
62
2011-02-13 19:38:31.451985 2011-02-13 19:38:31.465711 ** Caught signal (Aborted) ***
63
2011-02-13 19:38:31.465736 n thread 7f31e318a700
64
2011-02-13 19:38:31.478160 ceph version 0.25~rc (commit:)
65
2011-02-13 19:38:31.478201 1: /usr/bin/cmds() [0x72185c]
66
2011-02-13 19:38:31.478222 2: (()+0xf8f0) [0x7f31e58088f0]
67
2011-02-13 19:38:31.478238 3: (gsignal()+0x35) [0x7f31e43d9a75]
68
2011-02-13 19:38:31.478261 4: (abort()+0x180) [0x7f31e43dd5c0]
69
2011-02-13 19:38:31.478279 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f31e4c8f8e5]
70
2011-02-13 19:38:31.478295 6: (()+0xcad16) [0x7f31e4c8dd16]
71
2011-02-13 19:38:31.478305 7: (()+0xcad43) [0x7f31e4c8dd43]
72
2011-02-13 19:38:31.478312 8: (()+0xcae3e) [0x7f31e4c8de3e]
73
2011-02-13 19:38:31.478324 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x2f2) [0x709c02]
74
2011-02-13 19:38:31.478334 a: (Journaler::_finish_reread_head(int, ceph::buffer::list&, Context*)+0x1fd) [0x6baa1d]
75
2011-02-13 19:38:31.478343 b: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x951) [0x69dde1]
76
2011-02-13 19:38:31.478351 c: (MDS::_dispatch(Message*)+0x3004) [0x4bff34]
77
2011-02-13 19:38:31.478359 d: (MDS::ms_dispatch(Message*)+0x59) [0x4bff99]
78
2011-02-13 19:38:31.478367 e: (SimpleMessenger::dispatch_entry()+0x8a3) [0x488bf3]
79
2011-02-13 19:38:31.478375 f: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x483f3c]
80
2011-02-13 19:38:31.478383 10: (Thread::_entry_func(void*)+0xa) [0x49d1ea]
81
2011-02-13 19:38:31.478392 11: (()+0x69ca) [0x7f31e57ff9ca]
82
2011-02-13 19:38:31.478399 12: (clone()+0x6d) [0x7f31e448c70d]
83
2011-02-13 19:38:31.478404 ^C
84

    
85