|
2011-02-13 19:38:24.969592 7f06a6f0d700 mds0.snap check_osd_map - version unchanged
|
|
2011-02-13 19:38:24.969603 7f06a6f0d700 mds0.8 beacon_send up:active seq 2635 (currently up:active)
|
|
2011-02-13 19:38:24.969619 7f06a6f0d700 -- 10.202.105.222:6804/23070 --> mon0 10.135.211.78:6789/0 -- mdsbeacon(4308/srv-an56n up:active seq 2635 v41) v1 -- ?+0 0x43ce2500
|
|
2011-02-13 19:38:24.969637 7f06a6f0d700 -- 10.202.105.222:6804/23070 submit_message mdsbeacon(4308/srv-an56n up:active seq 2635 v41) v1 remote, 10.135.211.78:6789/0, have pipe.
|
|
2011-02-13 19:38:24.969672 7f06a6f0d700 mds0.8 beacon_kill last_acked_stamp 2011-02-13 19:37:18.905477, setting laggy flag.
|
|
2011-02-13 19:38:24.969756 7f06aaaad700 -- 10.202.105.222:6804/23070 >> 10.135.211.78:6789/0 pipe(0x223e280 sd=13 pgs=69 cs=1 l=1).writer: state = 2 policy.server=0
|
|
2011-02-13 19:38:24.969850 7f06aaaad700 -- 10.202.105.222:6804/23070 >> 10.135.211.78:6789/0 pipe(0x223e280 sd=13 pgs=69 cs=1 l=1).writer encoding 2730 0x43ce2500 mdsbeacon(4308/srv-an56n up:active seq 2635 v41) v1
|
|
2011-02-13 19:38:24.969934 7f06a8010700 mds0.8 handle_command args: [injectargs,--debug_ms 1 --debug_mds 10000]
|
|
2011-02-13 19:38:24.970029 7f06aaaad700 -- 10.202.105.222:6804/23070 >> 10.135.211.78:6789/0 pipe(0x223e280 sd=13 pgs=69 cs=1 l=1).writer sending 2730 0x43ce2500
|
|
2011-02-13 19:38:24.970122 7f06a8010700 -- 10.202.105.222:6804/23070 <== mon0 10.135.211.78:6789/0 2744 ==== mon_command(injectargs --debug_ms 1 --debug_mds 100 v 0) v1 ==== 84+0+0 (3487630235 0 0) 0x7ea0540 con 0x223f3c0
|
|
2011-02-13 19:38:24.970178 7f06a6f0d700 mds0.8 last tick was 39.299112 > 5 seconds ago, laggy_until 2011-02-13 17:37:11.364794, setting laggy flag
|
|
2011-02-13 19:38:24.970288 7f06a8010700 mds0.8 handle_command args: [injectargs,--debug_ms 1 --debug_mds 100]
|
|
2011-02-13 19:38:24.970332 7f06a8010700 -- 10.202.105.222:6804/23070 <== mon0 10.135.211.78:6789/0 2745 ==== mon_command(injectargs --debug_ms 1 --debug_mds 10 v 0) v1 ==== 83+0+0 (980379374 0 0) 0x7ea0700 con 0x223f3c0
|
|
2011-02-13 19:38:24.970347 7f06a8010700 mds0.8 handle_command args: [injectargs,--debug_ms 1 --debug_mds 10]
|
|
2011-02-13 19:38:24.970374 7f06a8010700 -- 10.202.105.222:6804/23070 <== mon0 10.135.211.78:6789/0 2746 ==== osd_map(39,39) v1 ==== 284+0+0 (2678458743 0 0) 0x537ea00 con 0x223f3c0
|
|
2011-02-13 19:38:24.970389 7f06a8010700 mds0.8 laggy, deferring osd_map(39,39) v1
|
|
2011-02-13 19:38:24.970407 7f06a8010700 -- 10.202.105.222:6804/23070 <== mon0 10.135.211.78:6789/0 2747 ==== mdsmap(e 42) v1 ==== 530+0+0 (1817951688 0 0) 0x537e800 con 0x223f3c0
|
|
2011-02-13 19:38:24.976658 7f06a8010700 mds0.8 handle_mds_map epoch 42 from mon0
|
|
2011-02-13 19:38:24.991483 7f06a8010700 mds0.8 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
|
|
2011-02-13 19:38:24.991517 7f06a8010700 mds0.8 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate objec}
|
|
2011-02-13 19:38:24.991532 7f06a8010700 mds-1.8 map says i am 10.202.105.222:6804/23070 mds-1 state down:dne
|
|
2011-02-13 19:38:24.991543 7f06a8010700 mds-1.8 handle_mds_map i (10.202.105.222:6804/23070) dne in the mdsmap, respawning myself
|
|
2011-02-13 19:38:24.991554 7f06a8010700 mds-1.8 respawn
|
|
2011-02-13 19:38:24.991563 7f06a8010700 mds-1.8 e: '/usr/bin/cmds'
|
|
2011-02-13 19:38:24.991571 7f06a8010700 mds-1.8 0: '/usr/bin/cmds'
|
|
2011-02-13 19:38:24.991584 7f06a8010700 mds-1.8 1: '-i'
|
|
2011-02-13 19:38:24.991597 7f06a8010700 mds-1.8 2: 'srv-an56n'
|
|
2011-02-13 19:38:24.991609 7f06a8010700 mds-1.8 3: '-c'
|
|
2011-02-13 19:38:24.991618 7f06a8010700 mds-1.8 4: '/etc/ceph/ceph.conf'
|
|
2011-02-13 19:38:24.991660 7f06a8010700 mds-1.8 cwd /
|
|
2011-02-13 19:38:25.780830 7f31e5c28720 starting mds.srv-an56n at 0.0.0.0:6805/23353
|
|
2011-02-13 19:38:25.784792 7f31e5c28720 -- 0.0.0.0:6805/23353 messenger.start: error creating directory: '/': error 17: File exists
|
|
2011-02-13 19:38:25.791139 7f31e318a700 mds-1.0 ms_handle_connect on 10.202.105.222:6789/0
|
|
2011-02-13 19:38:29.801070 7f31e318a700 mds-1.0 handle_mds_map standby
|
|
2011-02-13 19:38:29.886928 7f31e318a700 mds0.10 handle_mds_map i am now mds0.10
|
|
2011-02-13 19:38:29.886962 7f31e318a700 mds0.10 handle_mds_map state change up:standby --> up:replay
|
|
2011-02-13 19:38:29.886974 7f31e318a700 mds0.10 replay_start
|
|
2011-02-13 19:38:29.886988 7f31e318a700 mds0.10 recovery set is
|
|
2011-02-13 19:38:29.886997 7f31e318a700 mds0.10 need osdmap epoch 40, have 39
|
|
2011-02-13 19:38:29.887365 7f31e318a700 mds0.cache handle_mds_failure mds0 : recovery peers are
|
|
2011-02-13 19:38:29.915904 7f31e318a700 mds0.10 ms_handle_connect on 10.202.105.222:6801/23135
|
|
2011-02-13 19:38:29.916525 7f31e318a700 mds0.10 ms_handle_connect on 10.67.65.62:6800/22211
|
|
2011-02-13 19:38:29.916896 7f31e318a700 mds0.10 ms_handle_connect on 10.61.136.222:6800/21630
|
|
2011-02-13 19:38:29.916921 7f31e318a700 mds0.10 ms_handle_connect on 10.135.211.78:6801/5657
|
|
2011-02-13 19:38:30.417439 7f31e318a700 mds0.cache creating system inode with ino:100
|
|
2011-02-13 19:38:30.420327 7f31e318a700 mds0.cache creating system inode with ino:1
|
|
2011-02-13 19:38:31.431785 7f31e087d700 mds0.journaler try_read_entry got 0 len entry at offset 4051698581
|
|
2011-02-13 19:38:31.431846 7f31e087d700 mds0.log _replay journaler got error -22, aborting
|
|
2011-02-13 19:38:31.445544 sdc/Journaler.h: In function 'void Journaler::init_headers(Journaler::Header&)', In thread 7f31e318a700
|
|
2011-02-13 19:38:31.445572 sdc/Journaler.h:225: FAILED assert(readonly || state == STATE_READHEAD)
|
|
2011-02-13 19:38:31.445584 ceph version 0.25~rc (commit:)
|
|
2011-02-13 19:38:31.451874 1: (Journaler::_finish_reread_head(int, ceph::buffer::list&, Context*)+0x1fd) [0x6baa1d]
|
|
2011-02-13 19:38:31.451898 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x951) [0x69dde1]
|
|
2011-02-13 19:38:31.451910 3: (MDS::_dispatch(Message*)+0x3004) [0x4bff34]
|
|
2011-02-13 19:38:31.451920 4: (MDS::ms_dispatch(Message*)+0x59) [0x4bff99]
|
|
2011-02-13 19:38:31.451929 5: (SimpleMessenger::dispatch_entry()+0x8a3) [0x488bf3]
|
|
2011-02-13 19:38:31.451937 6: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x483f3c]
|
|
2011-02-13 19:38:31.451946 7: (Thread::_entry_func(void*)+0xa) [0x49d1ea]
|
|
2011-02-13 19:38:31.451963 8: (()+0x69ca) [0x7f31e57ff9ca]
|
|
2011-02-13 19:38:31.451972 9: (clone()+0x6d) [0x7f31e448c70d]
|
|
2011-02-13 19:38:31.451979 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
2011-02-13 19:38:31.451985 2011-02-13 19:38:31.465711 ** Caught signal (Aborted) ***
|
|
2011-02-13 19:38:31.465736 n thread 7f31e318a700
|
|
2011-02-13 19:38:31.478160 ceph version 0.25~rc (commit:)
|
|
2011-02-13 19:38:31.478201 1: /usr/bin/cmds() [0x72185c]
|
|
2011-02-13 19:38:31.478222 2: (()+0xf8f0) [0x7f31e58088f0]
|
|
2011-02-13 19:38:31.478238 3: (gsignal()+0x35) [0x7f31e43d9a75]
|
|
2011-02-13 19:38:31.478261 4: (abort()+0x180) [0x7f31e43dd5c0]
|
|
2011-02-13 19:38:31.478279 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f31e4c8f8e5]
|
|
2011-02-13 19:38:31.478295 6: (()+0xcad16) [0x7f31e4c8dd16]
|
|
2011-02-13 19:38:31.478305 7: (()+0xcad43) [0x7f31e4c8dd43]
|
|
2011-02-13 19:38:31.478312 8: (()+0xcae3e) [0x7f31e4c8de3e]
|
|
2011-02-13 19:38:31.478324 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x2f2) [0x709c02]
|
|
2011-02-13 19:38:31.478334 a: (Journaler::_finish_reread_head(int, ceph::buffer::list&, Context*)+0x1fd) [0x6baa1d]
|
|
2011-02-13 19:38:31.478343 b: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x951) [0x69dde1]
|
|
2011-02-13 19:38:31.478351 c: (MDS::_dispatch(Message*)+0x3004) [0x4bff34]
|
|
2011-02-13 19:38:31.478359 d: (MDS::ms_dispatch(Message*)+0x59) [0x4bff99]
|
|
2011-02-13 19:38:31.478367 e: (SimpleMessenger::dispatch_entry()+0x8a3) [0x488bf3]
|
|
2011-02-13 19:38:31.478375 f: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x483f3c]
|
|
2011-02-13 19:38:31.478383 10: (Thread::_entry_func(void*)+0xa) [0x49d1ea]
|
|
2011-02-13 19:38:31.478392 11: (()+0x69ca) [0x7f31e57ff9ca]
|
|
2011-02-13 19:38:31.478399 12: (clone()+0x6d) [0x7f31e448c70d]
|
|
2011-02-13 19:38:31.478404 ^C
|
|
|
|
|