Project

General

Profile

Bug #1110 » mds.0.log

Andre Noll, 06/14/2011 01:41 AM

 
2011-06-08 17:59:32.799894 7f51223b8720 ceph version 0.28.2-200-g350e650.commit: 350e65035b4a7d485fe62a48cc11b782ddbe5127. process: cmds. pid: 1894
2011-06-08 17:59:32.800873 7f51223b8720 write_pid_file: failed to open pid file '/var/run/ceph/mds.0.pid': error 2: No such file or directory
2011-06-08 17:59:32.802036 7f511f5ad700 mds-1.0 ms_handle_connect on 192.168.3.25:6789/0
2011-06-08 17:59:37.461396 7f511f5ad700 mds-1.0 handle_mds_map standby
2011-06-08 17:59:37.544795 7f511f5ad700 mds0.1 handle_mds_map i am now mds0.1
2011-06-08 17:59:37.544819 7f511f5ad700 mds0.1 handle_mds_map state change up:standby --> up:creating
2011-06-08 17:59:37.545141 7f511f5ad700 mds0.cache creating system inode with ino:1
2011-06-08 17:59:37.545406 7f511f5ad700 mds0.cache creating system inode with ino:2
2011-06-08 17:59:37.545822 7f511f5ad700 mds0.cache creating system inode with ino:100
2011-06-08 17:59:37.545900 7f511f5ad700 mds0.cache creating system inode with ino:600
2011-06-08 17:59:37.546084 7f511f5ad700 mds0.cache creating system inode with ino:601
2011-06-08 17:59:37.546248 7f511f5ad700 mds0.cache creating system inode with ino:602
2011-06-08 17:59:37.546409 7f511f5ad700 mds0.cache creating system inode with ino:603
2011-06-08 17:59:37.546586 7f511f5ad700 mds0.cache creating system inode with ino:604
2011-06-08 17:59:37.546754 7f511f5ad700 mds0.cache creating system inode with ino:605
2011-06-08 17:59:37.546919 7f511f5ad700 mds0.cache creating system inode with ino:606
2011-06-08 17:59:37.547085 7f511f5ad700 mds0.cache creating system inode with ino:607
2011-06-08 17:59:37.547250 7f511f5ad700 mds0.cache creating system inode with ino:608
2011-06-08 17:59:37.547417 7f511f5ad700 mds0.cache creating system inode with ino:609
2011-06-08 17:59:37.547583 7f511f5ad700 mds0.cache creating system inode with ino:200
2011-06-08 17:59:37.548977 7f511f5ad700 mds0.1 ms_handle_connect on 192.168.3.28:6800/1494
2011-06-08 17:59:37.549001 7f511f5ad700 mds0.1 ms_handle_connect on 192.168.3.27:6800/1492
2011-06-08 17:59:37.549279 7f511f5ad700 mds0.1 ms_handle_connect on 192.168.3.26:6800/1805
2011-06-08 17:59:37.549433 7f511f5ad700 mds0.1 ms_handle_connect on 192.168.3.29:6800/1689
2011-06-08 17:59:37.549976 7f511f5ad700 mds0.1 ms_handle_connect on 192.168.3.30:6800/1696
2011-06-08 17:59:39.095684 7f511f5ad700 mds0.1 creating_done
2011-06-08 17:59:39.170246 7f511f5ad700 mds0.1 handle_mds_map i am now mds0.1
2011-06-08 17:59:39.170268 7f511f5ad700 mds0.1 handle_mds_map state change up:creating --> up:active
2011-06-08 17:59:39.170281 7f511f5ad700 mds0.1 active_start
2011-06-08 17:59:50.036877 7f511ca9e700 -- 192.168.3.25:6800/1894 >> 192.168.3.25:0/1201428746 pipe(0xa12b60 sd=11 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/1201428746 (socket is 192.168.3.25:55770/0)
2011-06-08 17:59:50.038303 7f511c89c700 -- 192.168.3.25:6800/1894 >> 192.168.3.29:0/3833052252 pipe(0xa13200 sd=12 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3833052252 (socket is 192.168.3.29:56696/0)
2011-06-08 17:59:50.038360 7f511c79b700 -- 192.168.3.25:6800/1894 >> 192.168.3.28:0/1232072976 pipe(0xa13720 sd=13 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/1232072976 (socket is 192.168.3.28:49069/0)
2011-06-08 17:59:50.038462 7f511c69a700 -- 192.168.3.25:6800/1894 >> 192.168.3.26:0/552781789 pipe(0x7f511807a050 sd=14 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/552781789 (socket is 192.168.3.26:60302/0)
2011-06-08 17:59:50.038548 7f511c599700 -- 192.168.3.25:6800/1894 >> 192.168.3.30:0/603305768 pipe(0x7f5118062310 sd=15 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/603305768 (socket is 192.168.3.30:46297/0)
2011-06-08 17:59:50.039249 7f5117fff700 -- 192.168.3.25:6800/1894 >> 192.168.3.27:0/512535685 pipe(0x7f51180022f0 sd=16 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/512535685 (socket is 192.168.3.27:57591/0)
mds/Locker.cc: In function 'void Locker::file_xsyn(SimpleLock*, bool*)', in thread '0x7f511f5ad700'
mds/Locker.cc: 3894: FAILED assert(in->get_loner() >= 0 && in->mds_caps_wanted.empty())
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7f5121fa49ca]
13: (clone()+0x6d) [0x7f5120e9770d]
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7f5121fa49ca]
13: (clone()+0x6d) [0x7f5120e9770d]
*** Caught signal (Aborted) **
in thread 0x7f511f5ad700
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: cmds() [0x7025ee]
2: (()+0xf8f0) [0x7f5121fad8f0]
3: (gsignal()+0x35) [0x7f5120de4a75]
4: (abort()+0x180) [0x7f5120de85c0]
5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f512169a8e5]
6: (()+0xcad16) [0x7f5121698d16]
7: (()+0xcad43) [0x7f5121698d43]
8: (()+0xcae3e) [0x7f5121698e3e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x36a) [0x6cb77a]
10: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
11: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
12: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
13: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
14: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
15: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
16: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
17: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
18: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
19: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
20: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
21: (()+0x69ca) [0x7f5121fa49ca]
22: (clone()+0x6d) [0x7f5120e9770d]
2011-06-09 12:38:04.823664 7f45d4884720 ceph version 0.28.2-200-g350e650.commit: 350e65035b4a7d485fe62a48cc11b782ddbe5127. process: cmds. pid: 2011
2011-06-09 12:38:04.826613 7f45d4884720 write_pid_file: failed to open pid file '/var/run/ceph/mds.0.pid': error 2: No such file or directory
2011-06-09 12:38:04.827808 7f45d1a79700 mds-1.0 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 12:38:08.981081 7f45d1a79700 mds-1.0 handle_mds_map standby
2011-06-09 12:38:16.773507 7f45d1a79700 mds0.2 handle_mds_map i am now mds0.2
2011-06-09 12:38:16.773591 7f45d1a79700 mds0.2 handle_mds_map state change up:standby --> up:replay
2011-06-09 12:38:16.773607 7f45d1a79700 mds0.2 replay_start
2011-06-09 12:38:16.773625 7f45d1a79700 mds0.2 recovery set is
2011-06-09 12:38:16.773639 7f45d1a79700 mds0.2 need osdmap epoch 5, have 4
2011-06-09 12:38:16.773653 7f45d1a79700 mds0.2 waiting for osdmap 5 (which blacklists prior instance)
2011-06-09 12:38:16.773699 7f45d1a79700 mds0.cache handle_mds_failure mds0 : recovery peers are
2011-06-09 12:38:16.774784 7f45d1a79700 mds0.2 ms_handle_connect on 192.168.3.30:6800/1696
2011-06-09 12:38:16.774881 7f45d1a79700 mds0.2 ms_handle_connect on 192.168.3.28:6800/1494
2011-06-09 12:38:16.775805 7f45d1a79700 mds0.2 ms_handle_connect on 192.168.3.29:6800/1689
2011-06-09 12:38:17.612834 7f45d1a79700 mds0.2 ms_handle_connect on 192.168.3.27:6800/1492
2011-06-09 12:38:17.613420 7f45d1a79700 mds0.cache creating system inode with ino:100
2011-06-09 12:38:17.613601 7f45d1a79700 mds0.cache creating system inode with ino:1
2011-06-09 12:38:17.643659 7f45cb0f7700 mds0.2 replay_done
2011-06-09 12:38:17.643747 7f45cb0f7700 mds0.2 making mds journal writeable
2011-06-09 12:38:17.644704 7f45d1a79700 mds0.2 ms_handle_connect on 192.168.3.26:6800/1805
2011-06-09 12:38:18.048340 7f45d1a79700 mds0.2 handle_mds_map i am now mds0.2
2011-06-09 12:38:18.048379 7f45d1a79700 mds0.2 handle_mds_map state change up:replay --> up:reconnect
2011-06-09 12:38:18.048396 7f45d1a79700 mds0.2 reconnect_start
2011-06-09 12:38:18.048410 7f45d1a79700 mds0.2 reopen_log
2011-06-09 12:38:18.048443 7f45d1a79700 mds0.server reconnect_clients -- 6 sessions
2011-06-09 12:38:18.048562 7f45ca6f4700 -- 192.168.3.25:6800/2011 >> 192.168.3.25:0/1201428746 pipe(0xa0e990 sd=11 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/1201428746 (socket is 192.168.3.25:34884/0)
2011-06-09 12:38:18.048966 7f45ca4f2700 -- 192.168.3.25:6800/2011 >> 192.168.3.29:0/3833052252 pipe(0xa10270 sd=12 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3833052252 (socket is 192.168.3.29:43189/0)
2011-06-09 12:38:18.049011 7f45ca3f1700 -- 192.168.3.25:6800/2011 >> 192.168.3.27:0/512535685 pipe(0xa0f000 sd=13 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/512535685 (socket is 192.168.3.27:56385/0)
2011-06-09 12:38:18.049111 7f45ca1ef700 -- 192.168.3.25:6800/2011 >> 192.168.3.26:0/552781789 pipe(0x7f45cc0342f0 sd=15 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/552781789 (socket is 192.168.3.26:39925/0)
2011-06-09 12:38:18.049198 7f45ca2f0700 -- 192.168.3.25:6800/2011 >> 192.168.3.28:0/1232072976 pipe(0x7f45cc033f50 sd=14 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/1232072976 (socket is 192.168.3.28:48292/0)
2011-06-09 12:38:18.049229 7f45d1a79700 log [DBG] : reconnect by client4103 192.168.3.25:0/1201428746 after 0.000493
2011-06-09 12:38:18.049291 7f45ca0ee700 -- 192.168.3.25:6800/2011 >> 192.168.3.30:0/603305768 pipe(0x7f45cc034960 sd=16 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/603305768 (socket is 192.168.3.30:38301/0)
2011-06-09 12:38:18.049691 7f45d1a79700 log [DBG] : reconnect by client4107 192.168.3.27:0/512535685 after 0.001228
2011-06-09 12:38:18.049806 7f45d1a79700 log [DBG] : reconnect by client4106 192.168.3.29:0/3833052252 after 0.001357
2011-06-09 12:38:18.049841 7f45d1a79700 log [DBG] : reconnect by client4105 192.168.3.28:0/1232072976 after 0.001396
2011-06-09 12:38:18.049880 7f45d1a79700 log [DBG] : reconnect by client4108 192.168.3.30:0/603305768 after 0.001429
2011-06-09 12:38:18.049918 7f45d1a79700 log [DBG] : reconnect by client4104 192.168.3.26:0/552781789 after 0.001468
2011-06-09 12:38:18.049978 7f45d1a79700 mds0.2 reconnect_done
2011-06-09 12:38:18.189784 7f45d1a79700 mds0.2 handle_mds_map i am now mds0.2
2011-06-09 12:38:18.189817 7f45d1a79700 mds0.2 handle_mds_map state change up:reconnect --> up:rejoin
2011-06-09 12:38:18.189832 7f45d1a79700 mds0.2 rejoin_joint_start
2011-06-09 12:38:18.190419 7f45d1a79700 mds0.2 rejoin_done
2011-06-09 12:38:18.694617 7f45d1a79700 mds0.2 handle_mds_map i am now mds0.2
2011-06-09 12:38:18.694657 7f45d1a79700 mds0.2 handle_mds_map state change up:rejoin --> up:active
2011-06-09 12:38:18.694671 7f45d1a79700 mds0.2 recovery_done -- successful recovery!
2011-06-09 12:38:18.694816 7f45d1a79700 mds0.2 active_start
2011-06-09 12:38:18.695798 7f45d1a79700 mds0.2 cluster recovered.
mds/Locker.cc: In function 'void Locker::file_xsyn(SimpleLock*, bool*)', in thread '0x7f45d1a79700'
mds/Locker.cc: 3894: FAILED assert(in->get_loner() >= 0 && in->mds_caps_wanted.empty())
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7f45d44709ca]
13: (clone()+0x6d) [0x7f45d336370d]
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7f45d44709ca]
13: (clone()+0x6d) [0x7f45d336370d]
*** Caught signal (Aborted) **
in thread 0x7f45d1a79700
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: cmds() [0x7025ee]
2: (()+0xf8f0) [0x7f45d44798f0]
3: (gsignal()+0x35) [0x7f45d32b0a75]
4: (abort()+0x180) [0x7f45d32b45c0]
5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f45d3b668e5]
6: (()+0xcad16) [0x7f45d3b64d16]
7: (()+0xcad43) [0x7f45d3b64d43]
8: (()+0xcae3e) [0x7f45d3b64e3e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x36a) [0x6cb77a]
10: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
11: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
12: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
13: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
14: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
15: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
16: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
17: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
18: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
19: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
20: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
21: (()+0x69ca) [0x7f45d44709ca]
22: (clone()+0x6d) [0x7f45d336370d]
2011-06-09 12:51:15.382827 7ffff7fd9720 ceph version 0.28.2-200-g350e650.commit: 350e65035b4a7d485fe62a48cc11b782ddbe5127. process: cmds. pid: 2059
2011-06-09 12:51:15.384353 7ffff7fd9720 write_pid_file: failed to open pid file '/var/run/ceph/mds.0.pid': error 2: No such file or directory
2011-06-09 12:51:15.385417 7ffff51cf700 mds-1.0 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 12:51:19.471696 7ffff51cf700 mds-1.0 handle_mds_map standby
2011-06-09 12:51:29.486455 7ffff7fd9720 ceph version 0.28.2-200-g350e650.commit: 350e65035b4a7d485fe62a48cc11b782ddbe5127. process: cmds. pid: 2076
2011-06-09 12:51:29.494283 7ffff51cf700 mds-1.0 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 12:51:33.436483 7ffff51cf700 mds0.3 handle_mds_map i am now mds0.3
2011-06-09 12:51:33.436526 7ffff51cf700 mds0.3 handle_mds_map state change up:standby --> up:replay
2011-06-09 12:51:33.436540 7ffff51cf700 mds0.3 replay_start
2011-06-09 12:51:33.436556 7ffff51cf700 mds0.3 recovery set is
2011-06-09 12:51:33.436569 7ffff51cf700 mds0.3 need osdmap epoch 6, have 5
2011-06-09 12:51:33.436581 7ffff51cf700 mds0.3 waiting for osdmap 6 (which blacklists prior instance)
2011-06-09 12:51:33.436621 7ffff51cf700 mds0.cache handle_mds_failure mds0 : recovery peers are
2011-06-09 12:51:33.437662 7ffff51cf700 mds0.3 ms_handle_connect on 192.168.3.29:6800/1689
2011-06-09 12:51:33.437715 7ffff51cf700 mds0.3 ms_handle_connect on 192.168.3.30:6800/1696
2011-06-09 12:51:33.437878 7ffff51cf700 mds0.3 ms_handle_connect on 192.168.3.28:6800/1494
2011-06-09 12:51:33.641470 7ffff51cf700 mds0.3 ms_handle_connect on 192.168.3.27:6800/1492
2011-06-09 12:51:33.647132 7ffff51cf700 mds-1.0 handle_mds_map standby
2011-06-09 12:51:34.211537 7ffff51cf700 mds0.cache creating system inode with ino:100
2011-06-09 12:51:34.211731 7ffff51cf700 mds0.cache creating system inode with ino:1
2011-06-09 12:51:34.258112 7fffee7f5700 mds0.3 replay_done
2011-06-09 12:51:34.258201 7fffee7f5700 mds0.3 making mds journal writeable
2011-06-09 12:51:34.259456 7ffff51cf700 mds0.3 ms_handle_connect on 192.168.3.26:6800/1805
2011-06-09 12:51:34.770096 7ffff51cf700 mds0.3 handle_mds_map i am now mds0.3
2011-06-09 12:51:34.770129 7ffff51cf700 mds-1.0 handle_mds_map standby
2011-06-09 12:51:34.770168 7ffff51cf700 mds0.3 handle_mds_map state change up:replay --> up:reconnect
2011-06-09 12:51:34.770239 7ffff51cf700 mds0.3 reconnect_start
2011-06-09 12:51:34.770256 7ffff51cf700 mds0.3 reopen_log
2011-06-09 12:51:34.770320 7fffedef3700 -- 192.168.3.25:6800/2059 >> 192.168.3.25:0/1201428746 pipe(0xa2b880 sd=13 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/1201428746 (socket is 192.168.3.25:38260/0)
2011-06-09 12:51:34.770363 7ffff51cf700 mds0.server reconnect_clients -- 6 sessions
2011-06-09 12:51:34.770759 7ffff51cf700 log [DBG] : reconnect by client4103 192.168.3.25:0/1201428746 after 0.000394
2011-06-09 12:51:34.771006 7fffedcf1700 -- 192.168.3.25:6800/2059 >> 192.168.3.30:0/603305768 pipe(0xa2b5a0 sd=14 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/603305768 (socket is 192.168.3.30:47072/0)
2011-06-09 12:51:34.771059 7fffedbf0700 -- 192.168.3.25:6800/2059 >> 192.168.3.28:0/1232072976 pipe(0xa2d010 sd=15 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/1232072976 (socket is 192.168.3.28:59152/0)
2011-06-09 12:51:34.771126 7fffedaef700 -- 192.168.3.25:6800/2059 >> 192.168.3.26:0/552781789 pipe(0xa2d580 sd=16 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/552781789 (socket is 192.168.3.26:47793/0)
2011-06-09 12:51:34.771194 7fffed9ee700 -- 192.168.3.25:6800/2059 >> 192.168.3.29:0/3833052252 pipe(0x7ffff0018f60 sd=17 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3833052252 (socket is 192.168.3.29:44468/0)
2011-06-09 12:51:34.771527 7ffff51cf700 log [DBG] : reconnect by client4108 192.168.3.30:0/603305768 after 0.001216
2011-06-09 12:51:34.771576 7fffed6eb700 -- 192.168.3.25:6800/2059 >> 192.168.3.27:0/512535685 pipe(0x7ffff0007730 sd=18 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/512535685 (socket is 192.168.3.27:36733/0)
2011-06-09 12:51:34.771618 7ffff51cf700 log [DBG] : reconnect by client4104 192.168.3.26:0/552781789 after 0.001278
2011-06-09 12:51:34.771822 7ffff51cf700 log [DBG] : reconnect by client4106 192.168.3.29:0/3833052252 after 0.001518
2011-06-09 12:51:34.771932 7ffff51cf700 log [DBG] : reconnect by client4105 192.168.3.28:0/1232072976 after 0.001615
2011-06-09 12:51:34.772086 7ffff51cf700 log [DBG] : reconnect by client4107 192.168.3.27:0/512535685 after 0.001789
2011-06-09 12:51:34.772189 7ffff51cf700 mds0.3 reconnect_done
2011-06-09 12:51:35.252821 7ffff51cf700 mds0.3 handle_mds_map i am now mds0.3
2011-06-09 12:51:35.252898 7ffff51cf700 mds0.3 handle_mds_map state change up:reconnect --> up:rejoin
2011-06-09 12:51:35.252915 7ffff51cf700 mds0.3 rejoin_joint_start
2011-06-09 12:51:35.252950 7ffff51cf700 mds-1.0 handle_mds_map standby
2011-06-09 12:51:35.253606 7ffff51cf700 mds0.3 rejoin_done
2011-06-09 12:51:35.755125 7ffff51cf700 mds0.3 handle_mds_map i am now mds0.3
2011-06-09 12:51:35.755185 7ffff51cf700 mds0.3 handle_mds_map state change up:rejoin --> up:active
2011-06-09 12:51:35.755198 7ffff51cf700 mds0.3 recovery_done -- successful recovery!
2011-06-09 12:51:35.755334 7ffff51cf700 mds0.3 active_start
2011-06-09 12:51:35.755808 7ffff51cf700 mds-1.0 handle_mds_map standby
2011-06-09 12:51:35.756389 7ffff51cf700 mds0.3 cluster recovered.
2011-06-09 13:06:34.363337 7ffff51cf700 mds0.3 ms_handle_reset on 192.168.3.30:6800/1696
2011-06-09 13:06:34.363572 7ffff51cf700 mds0.3 ms_handle_reset on 192.168.3.26:6800/1805
2011-06-09 13:06:34.364160 7ffff51cf700 mds0.3 ms_handle_connect on 192.168.3.30:6800/1696
2011-06-09 13:06:34.364220 7ffff51cf700 mds0.3 ms_handle_connect on 192.168.3.26:6800/1805
2011-06-09 13:06:35.859660 7ffff51cf700 mds0.3 ms_handle_reset on 192.168.3.29:6800/1689
2011-06-09 13:06:35.860349 7ffff51cf700 mds0.3 ms_handle_connect on 192.168.3.29:6800/1689
mds/Locker.cc: In function 'void Locker::file_xsyn(SimpleLock*, bool*)', in thread '0x7ffff51cf700'
mds/Locker.cc: 3894: FAILED assert(in->get_loner() >= 0 && in->mds_caps_wanted.empty())
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7ffff7bc69ca]
13: (clone()+0x6d) [0x7ffff6ab970d]
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7ffff7bc69ca]
13: (clone()+0x6d) [0x7ffff6ab970d]
*** Caught signal (Aborted) **
in thread 0x7ffff51cf700
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: /usr/local/bin/cmds() [0x7025ee]
2: (()+0xf8f0) [0x7ffff7bcf8f0]
3: (gsignal()+0x35) [0x7ffff6a06a75]
4: (abort()+0x180) [0x7ffff6a0a5c0]
5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7ffff72bc8e5]
6: (()+0xcad16) [0x7ffff72bad16]
7: (()+0xcad43) [0x7ffff72bad43]
8: (()+0xcae3e) [0x7ffff72bae3e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x36a) [0x6cb77a]
10: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
11: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
12: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
13: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
14: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
15: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
16: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
17: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
18: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
19: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
20: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
21: (()+0x69ca) [0x7ffff7bc69ca]
22: (clone()+0x6d) [0x7ffff6ab970d]
2011-06-09 13:13:36.176157 7ffff51cf700 mds0.4 handle_mds_map i am now mds0.4
2011-06-09 13:13:36.176240 7ffff51cf700 mds0.4 handle_mds_map state change up:standby --> up:replay
2011-06-09 13:13:36.176256 7ffff51cf700 mds0.4 replay_start
2011-06-09 13:13:36.176278 7ffff51cf700 mds0.4 recovery set is
2011-06-09 13:13:36.176292 7ffff51cf700 mds0.4 need osdmap epoch 8, have 5
2011-06-09 13:13:36.176305 7ffff51cf700 mds0.4 waiting for osdmap 8 (which blacklists prior instance)
2011-06-09 13:13:36.176344 7ffff51cf700 mds0.cache handle_mds_failure mds0 : recovery peers are
2011-06-09 13:13:36.179397 7ffff51cf700 mds0.4 ms_handle_connect on 192.168.3.29:6800/1689
2011-06-09 13:13:36.179942 7ffff51cf700 mds0.4 ms_handle_connect on 192.168.3.30:6800/1696
2011-06-09 13:13:36.180549 7ffff51cf700 mds0.4 ms_handle_connect on 192.168.3.28:6800/1494
2011-06-09 13:13:45.049043 7ffff51cf700 mds0.4 ms_handle_connect on 192.168.3.26:6800/1805
2011-06-09 13:13:51.599168 7ffff51cf700 mds0.cache creating system inode with ino:100
2011-06-09 13:13:51.600092 7ffff51cf700 mds0.cache creating system inode with ino:1
2011-06-09 13:13:51.600872 7ffff51cf700 mds0.4 ms_handle_connect on 192.168.3.27:6800/1492
2011-06-09 13:13:52.170205 7ffff26c0700 mds0.4 replay_done
2011-06-09 13:13:52.170296 7ffff26c0700 mds0.4 making mds journal writeable
2011-06-09 13:13:52.308955 7ffff51cf700 mds0.4 handle_mds_map i am now mds0.4
2011-06-09 13:13:52.308995 7ffff51cf700 mds0.4 handle_mds_map state change up:replay --> up:reconnect
2011-06-09 13:13:52.309010 7ffff51cf700 mds0.4 reconnect_start
2011-06-09 13:13:52.309024 7ffff51cf700 mds0.4 reopen_log
2011-06-09 13:13:52.309051 7ffff51cf700 mds0.server reconnect_clients -- 6 sessions
2011-06-09 13:13:52.310696 7ffff1ebf700 -- 192.168.3.25:6801/2076 >> 192.168.3.25:0/1201428746 pipe(0xa55f80 sd=13 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/1201428746 (socket is 192.168.3.25:34757/0)
2011-06-09 13:13:52.310934 7ffff1dbe700 -- 192.168.3.25:6801/2076 >> 192.168.3.28:0/1232072976 pipe(0xa55a70 sd=14 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/1232072976 (socket is 192.168.3.28:55677/0)
2011-06-09 13:13:52.311651 7ffff1cbd700 -- 192.168.3.25:6801/2076 >> 192.168.3.29:0/3833052252 pipe(0xa10160 sd=15 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3833052252 (socket is 192.168.3.29:35332/0)
2011-06-09 13:13:52.313556 7ffff1bbc700 -- 192.168.3.25:6801/2076 >> 192.168.3.30:0/603305768 pipe(0xa566c0 sd=16 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/603305768 (socket is 192.168.3.30:33612/0)
2011-06-09 13:13:52.314334 7ffff19ba700 -- 192.168.3.25:6801/2076 >> 192.168.3.27:0/512535685 pipe(0x7fffec002140 sd=17 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/512535685 (socket is 192.168.3.27:34983/0)
2011-06-09 13:13:52.315194 7ffff18b9700 -- 192.168.3.25:6801/2076 >> 192.168.3.26:0/552781789 pipe(0x7fffec002760 sd=18 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/552781789 (socket is 192.168.3.26:46437/0)
2011-06-09 13:13:52.315249 7ffff51cf700 log [DBG] : reconnect by client4103 192.168.3.25:0/1201428746 after 0.004611
2011-06-09 13:13:52.317672 7ffff51cf700 log [DBG] : reconnect by client4108 192.168.3.30:0/603305768 after 0.008595
2011-06-09 13:13:52.319087 7ffff51cf700 log [DBG] : reconnect by client4106 192.168.3.29:0/3833052252 after 0.010006
2011-06-09 13:13:52.319219 7ffff51cf700 log [DBG] : reconnect by client4107 192.168.3.27:0/512535685 after 0.010157
2011-06-09 13:13:52.320267 7ffff51cf700 log [DBG] : reconnect by client4104 192.168.3.26:0/552781789 after 0.011204
2011-06-09 13:13:52.320541 7ffff51cf700 log [DBG] : reconnect by client4105 192.168.3.28:0/1232072976 after 0.011459
2011-06-09 13:13:52.320592 7ffff51cf700 mds0.4 reconnect_done
2011-06-09 13:13:52.475742 7ffff51cf700 mds0.4 handle_mds_map i am now mds0.4
2011-06-09 13:13:52.475779 7ffff51cf700 mds0.4 handle_mds_map state change up:reconnect --> up:rejoin
2011-06-09 13:13:52.475833 7ffff51cf700 mds0.4 rejoin_joint_start
2011-06-09 13:13:52.476430 7ffff51cf700 mds0.4 rejoin_done
2011-06-09 13:13:53.023951 7ffff51cf700 mds0.4 handle_mds_map i am now mds0.4
2011-06-09 13:13:53.023984 7ffff51cf700 mds0.4 handle_mds_map state change up:rejoin --> up:active
2011-06-09 13:13:53.024000 7ffff51cf700 mds0.4 recovery_done -- successful recovery!
2011-06-09 13:13:53.024236 7ffff51cf700 mds0.4 active_start
2011-06-09 13:13:53.025116 7ffff51cf700 mds0.4 cluster recovered.
2011-06-09 13:18:18.854168 7f5fbbd9f720 ceph version 0.28.2-200-g350e650.commit: 350e65035b4a7d485fe62a48cc11b782ddbe5127. process: cmds. pid: 2150
2011-06-09 13:18:18.855498 7f5fbbd9f720 write_pid_file: failed to open pid file '/var/run/ceph/mds.0.pid': error 2: No such file or directory
2011-06-09 13:18:18.856570 7f5fb8f94700 mds-1.0 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 13:18:23.087178 7f5fb8f94700 mds-1.0 handle_mds_map standby
2011-06-09 13:18:33.712472 7f5fb8f94700 mds0.5 handle_mds_map i am now mds0.5
2011-06-09 13:18:33.712539 7f5fb8f94700 mds0.5 handle_mds_map state change up:standby --> up:replay
2011-06-09 13:18:33.712553 7f5fb8f94700 mds0.5 replay_start
2011-06-09 13:18:33.712569 7f5fb8f94700 mds0.5 recovery set is
2011-06-09 13:18:33.712582 7f5fb8f94700 mds0.5 need osdmap epoch 10, have 9
2011-06-09 13:18:33.712594 7f5fb8f94700 mds0.5 waiting for osdmap 10 (which blacklists prior instance)
2011-06-09 13:18:33.712638 7f5fb8f94700 mds0.cache handle_mds_failure mds0 : recovery peers are
2011-06-09 13:18:33.713622 7f5fb8f94700 mds0.5 ms_handle_connect on 192.168.3.29:6800/1689
2011-06-09 13:18:33.713737 7f5fb8f94700 mds0.5 ms_handle_connect on 192.168.3.28:6800/1494
2011-06-09 13:18:33.714310 7f5fb8f94700 mds0.5 ms_handle_connect on 192.168.3.30:6800/1696
2011-06-09 13:18:36.687028 7f5fb8f94700 mds0.5 ms_handle_connect on 192.168.3.26:6800/1805
2011-06-09 13:18:40.697958 7f5fb8f94700 mds0.cache creating system inode with ino:100
2011-06-09 13:18:40.698314 7f5fb8f94700 mds0.cache creating system inode with ino:1
2011-06-09 13:18:40.721461 7f5fb8f94700 mds0.5 ms_handle_connect on 192.168.3.27:6800/1492
2011-06-09 13:18:42.318093 7f5fb6485700 mds0.5 replay_done
2011-06-09 13:18:42.318358 7f5fb6485700 mds0.5 making mds journal writeable
2011-06-09 13:18:42.570429 7f5fb5c84700 -- 192.168.3.25:6800/2150 >> 192.168.3.25:0/1201428746 pipe(0xa879d0 sd=11 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/1201428746 (socket is 192.168.3.25:46663/0)
2011-06-09 13:18:42.572784 7f5fb8f94700 mds0.5 handle_mds_map i am now mds0.5
2011-06-09 13:18:42.573040 7f5fb8f94700 mds0.5 handle_mds_map state change up:replay --> up:reconnect
2011-06-09 13:18:42.573250 7f5fb8f94700 mds0.5 reconnect_start
2011-06-09 13:18:42.573494 7f5fb8f94700 mds0.5 reopen_log
2011-06-09 13:18:42.573853 7f5fb8f94700 mds0.server reconnect_clients -- 6 sessions
2011-06-09 13:18:42.574438 7f5fb5a82700 -- 192.168.3.25:6800/2150 >> 192.168.3.29:0/3833052252 pipe(0xa10010 sd=12 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3833052252 (socket is 192.168.3.29:48329/0)
2011-06-09 13:18:42.575361 7f5fb8f94700 log [DBG] : reconnect by client4103 192.168.3.25:0/1201428746 after 0.000678
2011-06-09 13:18:42.576766 7f5fb5981700 -- 192.168.3.25:6800/2150 >> 192.168.3.28:0/1232072976 pipe(0xa08560 sd=13 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/1232072976 (socket is 192.168.3.28:39842/0)
2011-06-09 13:18:42.581297 7f5fb567e700 -- 192.168.3.25:6800/2150 >> 192.168.3.26:0/552781789 pipe(0xa8b010 sd=14 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/552781789 (socket is 192.168.3.26:54339/0)
2011-06-09 13:18:42.581584 7f5fb8f94700 log [DBG] : reconnect by client4105 192.168.3.28:0/1232072976 after 0.006281
2011-06-09 13:18:42.583508 7f5fb557d700 -- 192.168.3.25:6800/2150 >> 192.168.3.30:0/603305768 pipe(0xa10a80 sd=15 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/603305768 (socket is 192.168.3.30:46274/0)
2011-06-09 13:18:42.585090 7f5fb8f94700 log [DBG] : reconnect by client4106 192.168.3.29:0/3833052252 after 0.008516
2011-06-09 13:18:42.587163 7f5fb537b700 -- 192.168.3.25:6800/2150 >> 192.168.3.27:0/512535685 pipe(0xa955f0 sd=16 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/512535685 (socket is 192.168.3.27:39811/0)
2011-06-09 13:18:42.588520 7f5fb8f94700 log [DBG] : reconnect by client4104 192.168.3.26:0/552781789 after 0.012169
2011-06-09 13:18:42.589789 7f5fb8f94700 log [DBG] : reconnect by client4108 192.168.3.30:0/603305768 after 0.015799
2011-06-09 13:18:42.592787 7f5fb8f94700 log [DBG] : reconnect by client4107 192.168.3.27:0/512535685 after 0.018818
2011-06-09 13:18:42.593328 7f5fb8f94700 mds0.5 reconnect_done
2011-06-09 13:18:42.746111 7f5fb8f94700 mds0.5 handle_mds_map i am now mds0.5
2011-06-09 13:18:42.746335 7f5fb8f94700 mds0.5 handle_mds_map state change up:reconnect --> up:rejoin
2011-06-09 13:18:42.746437 7f5fb8f94700 mds0.5 rejoin_joint_start
2011-06-09 13:18:42.747045 7f5fb8f94700 mds0.5 rejoin_done
2011-06-09 13:18:43.292590 7f5fb8f94700 mds0.5 handle_mds_map i am now mds0.5
2011-06-09 13:18:43.292832 7f5fb8f94700 mds0.5 handle_mds_map state change up:rejoin --> up:active
2011-06-09 13:18:43.293039 7f5fb8f94700 mds0.5 recovery_done -- successful recovery!
2011-06-09 13:18:43.305009 7f5fb8f94700 mds0.5 active_start
2011-06-09 13:18:43.305957 7f5fb8f94700 mds0.5 cluster recovered.
2011-06-09 13:33:42.426795 7f5fb8f94700 mds0.5 ms_handle_reset on 192.168.3.26:6800/1805
2011-06-09 13:33:42.436434 7f5fb8f94700 mds0.5 ms_handle_connect on 192.168.3.26:6800/1805
2011-06-09 13:33:43.511029 7f5fb8f94700 mds0.5 ms_handle_reset on 192.168.3.27:6800/1492
2011-06-09 13:33:43.511852 7f5fb8f94700 mds0.5 ms_handle_connect on 192.168.3.27:6800/1492
2011-06-09 13:33:53.809539 7f5fb8f94700 mds0.5 ms_handle_reset on 192.168.3.30:6800/1696
2011-06-09 13:33:53.810361 7f5fb8f94700 mds0.5 ms_handle_connect on 192.168.3.30:6800/1696
mds/Locker.cc: In function 'void Locker::file_xsyn(SimpleLock*, bool*)', in thread '0x7f5fb8f94700'
mds/Locker.cc: 3894: FAILED assert(in->get_loner() >= 0 && in->mds_caps_wanted.empty())
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7f5fbb98b9ca]
13: (clone()+0x6d) [0x7f5fba87e70d]
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7f5fbb98b9ca]
13: (clone()+0x6d) [0x7f5fba87e70d]
*** Caught signal (Aborted) **
in thread 0x7f5fb8f94700
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: cmds() [0x7025ee]
2: (()+0xf8f0) [0x7f5fbb9948f0]
3: (gsignal()+0x35) [0x7f5fba7cba75]
4: (abort()+0x180) [0x7f5fba7cf5c0]
5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f5fbb0818e5]
6: (()+0xcad16) [0x7f5fbb07fd16]
7: (()+0xcad43) [0x7f5fbb07fd43]
8: (()+0xcae3e) [0x7f5fbb07fe3e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x36a) [0x6cb77a]
10: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
11: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
12: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
13: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
14: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
15: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
16: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
17: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
18: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
19: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
20: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
21: (()+0x69ca) [0x7f5fbb98b9ca]
22: (clone()+0x6d) [0x7f5fba87e70d]
2011-06-09 13:44:36.325427 7ffff7fd9720 ceph version 0.28.2-200-g350e650.commit: 350e65035b4a7d485fe62a48cc11b782ddbe5127. process: cmds. pid: 2227
2011-06-09 13:44:36.325785 7ffff7fd9720 unrecognized arg -D
2011-06-09 13:44:36.325815 7ffff7fd9720 usage: cmds -i name [flags] [[--journal_check rank]|[--hot-standby][rank]]
-m monitorip:port
connect to monitor at given address
--debug_mds n
debug MDS level (e.g. 10)
--dump-journal rank filename
dump the MDS journal for rank. Defaults to mds.journal.dump
--journal-check rank
replay the journal for rank, then exit
--hot-standby rank
stat up as a hot standby for rank
--reset-journal rank
discard the MDS journal for rank, and replace it with a single
event that updates/resets inotable and sessionmap on replay.

2011-06-09 13:45:04.435110 7ffff7fd9720 ceph version 0.28.2-200-g350e650.commit: 350e65035b4a7d485fe62a48cc11b782ddbe5127. process: cmds. pid: 2230
2011-06-09 13:45:04.444217 7ffff51cf700 mds-1.0 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 13:45:08.596476 7ffff51cf700 mds-1.0 handle_mds_map standby
2011-06-09 13:45:15.532188 7ffff51cf700 mds0.6 handle_mds_map i am now mds0.6
2011-06-09 13:45:15.532224 7ffff51cf700 mds0.6 handle_mds_map state change up:standby --> up:replay
2011-06-09 13:45:15.532237 7ffff51cf700 mds0.6 replay_start
2011-06-09 13:45:15.532256 7ffff51cf700 mds0.6 recovery set is
2011-06-09 13:45:15.532268 7ffff51cf700 mds0.6 need osdmap epoch 13, have 12
2011-06-09 13:45:15.532280 7ffff51cf700 mds0.6 waiting for osdmap 13 (which blacklists prior instance)
2011-06-09 13:45:15.532316 7ffff51cf700 mds0.cache handle_mds_failure mds0 : recovery peers are
2011-06-09 13:45:15.535692 7ffff51cf700 mds0.6 ms_handle_connect on 192.168.3.30:6800/1696
2011-06-09 13:45:15.536607 7ffff51cf700 mds0.6 ms_handle_connect on 192.168.3.28:6800/1494
2011-06-09 13:45:15.538503 7ffff51cf700 mds0.6 ms_handle_connect on 192.168.3.29:6800/1689
2011-06-09 13:45:16.137679 7ffff51cf700 mds0.6 ms_handle_connect on 192.168.3.26:6800/1805
2011-06-09 13:45:16.566342 7ffff51cf700 mds0.cache creating system inode with ino:100
2011-06-09 13:45:16.567215 7ffff51cf700 mds0.cache creating system inode with ino:1
2011-06-09 13:45:16.567774 7ffff51cf700 mds0.6 ms_handle_connect on 192.168.3.27:6800/1492
2011-06-09 13:45:17.275535 7ffff26c0700 mds0.6 replay_done
2011-06-09 13:45:17.275618 7ffff26c0700 mds0.6 making mds journal writeable
2011-06-09 13:45:17.420603 7ffff51cf700 mds0.6 handle_mds_map i am now mds0.6
2011-06-09 13:45:17.420663 7ffff51cf700 mds0.6 handle_mds_map state change up:replay --> up:reconnect
2011-06-09 13:45:17.420677 7ffff51cf700 mds0.6 reconnect_start
2011-06-09 13:45:17.420689 7ffff51cf700 mds0.6 reopen_log
2011-06-09 13:45:17.420717 7ffff51cf700 mds0.server reconnect_clients -- 6 sessions
2011-06-09 13:45:17.421630 7ffff1ebf700 -- 192.168.3.25:6800/2230 >> 192.168.3.25:0/1201428746 pipe(0xa104e0 sd=13 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/1201428746 (socket is 192.168.3.25:54537/0)
2011-06-09 13:45:17.422472 7ffff1dbe700 -- 192.168.3.25:6800/2230 >> 192.168.3.29:0/3833052252 pipe(0xa109c0 sd=14 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3833052252 (socket is 192.168.3.29:40731/0)
2011-06-09 13:45:17.423178 7ffff1cbd700 -- 192.168.3.25:6800/2230 >> 192.168.3.30:0/603305768 pipe(0xa8bf00 sd=15 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/603305768 (socket is 192.168.3.30:53870/0)
2011-06-09 13:45:17.425265 7ffff1bbc700 -- 192.168.3.25:6800/2230 >> 192.168.3.28:0/1232072976 pipe(0xa12f90 sd=16 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/1232072976 (socket is 192.168.3.28:35779/0)
2011-06-09 13:45:17.425387 7ffff19ba700 -- 192.168.3.25:6800/2230 >> 192.168.3.26:0/552781789 pipe(0x7fffe4001060 sd=17 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/552781789 (socket is 192.168.3.26:45379/0)
2011-06-09 13:45:17.426216 7ffff18b9700 -- 192.168.3.25:6800/2230 >> 192.168.3.27:0/512535685 pipe(0x7fffe40015b0 sd=18 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/512535685 (socket is 192.168.3.27:44100/0)
2011-06-09 13:45:17.426263 7ffff51cf700 log [DBG] : reconnect by client4103 192.168.3.25:0/1201428746 after 0.003815
2011-06-09 13:45:17.428966 7ffff51cf700 log [DBG] : reconnect by client4106 192.168.3.29:0/3833052252 after 0.008217
2011-06-09 13:45:17.430397 7ffff51cf700 log [DBG] : reconnect by client4108 192.168.3.30:0/603305768 after 0.009662
2011-06-09 13:45:17.431070 7ffff51cf700 log [DBG] : reconnect by client4105 192.168.3.28:0/1232072976 after 0.010338
2011-06-09 13:45:17.431120 7ffff51cf700 log [DBG] : reconnect by client4104 192.168.3.26:0/552781789 after 0.010395
2011-06-09 13:45:17.431610 7ffff51cf700 log [DBG] : reconnect by client4107 192.168.3.27:0/512535685 after 0.010876
2011-06-09 13:45:17.431834 7ffff51cf700 mds0.6 reconnect_done
2011-06-09 13:45:17.613620 7ffff51cf700 mds0.6 handle_mds_map i am now mds0.6
2011-06-09 13:45:17.613662 7ffff51cf700 mds0.6 handle_mds_map state change up:reconnect --> up:rejoin
2011-06-09 13:45:17.613677 7ffff51cf700 mds0.6 rejoin_joint_start
2011-06-09 13:45:17.614514 7ffff51cf700 mds0.6 rejoin_done
2011-06-09 13:45:18.093133 7ffff51cf700 mds0.6 handle_mds_map i am now mds0.6
2011-06-09 13:45:18.093159 7ffff51cf700 mds0.6 handle_mds_map state change up:rejoin --> up:active
2011-06-09 13:45:18.093171 7ffff51cf700 mds0.6 recovery_done -- successful recovery!
2011-06-09 13:45:18.093538 7ffff51cf700 mds0.6 active_start
2011-06-09 13:45:18.093666 7ffff51cf700 mds0.6 cluster recovered.
mds/Locker.cc: In function 'void Locker::file_xsyn(SimpleLock*, bool*)', in thread '0x7ffff51cf700'
mds/Locker.cc: 3894: FAILED assert(in->get_loner() >= 0 && in->mds_caps_wanted.empty())
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7ffff7bc69ca]
13: (clone()+0x6d) [0x7ffff6ab970d]
ceph version 0.28.2-200-g350e650 (commit:350e65035b4a7d485fe62a48cc11b782ddbe5127)
1: (Locker::file_xsyn(SimpleLock*, bool*)+0x767) [0x5b5c87]
2: (Locker::_rdlock_kick(SimpleLock*)+0x16f) [0x5befcf]
3: (Locker::rdlock_start(SimpleLock*, MDRequest*, bool)+0x91) [0x5cb991]
4: (Locker::acquire_locks(MDRequest*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x28f3) [0x5d1113]
5: (Server::handle_client_stat(MDRequest*)+0x254) [0x506d24]
6: (Server::handle_client_request(MClientRequest*)+0x524) [0x516694]
7: (MDS::handle_deferrable_message(Message*)+0x9df) [0x497d7f]
8: (MDS::_dispatch(Message*)+0x1766) [0x4ac2c6]
9: (MDS::ms_dispatch(Message*)+0x59) [0x4ac819]
10: (SimpleMessenger::dispatch_entry()+0x803) [0x6d79d3]
11: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4893fc]
12: (()+0x69ca) [0x7ffff7bc69ca]
13: (clone()+0x6d) [0x7ffff6ab970d]
2011-06-09 18:40:28.159910 7fb2379c4720 ceph version 0.28.2-201-g5922de2.commit: 5922de29e7414be3aaa6eb04be8bb0ae5c943ccf. process: cmds. pid: 2292
2011-06-09 18:40:28.161513 7fb2379c4720 write_pid_file: failed to open pid file '/var/run/ceph/mds.0.pid': error 2: No such file or directory
2011-06-09 18:41:23.745374 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.26:0/3060205489 pipe(0xa097c0 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/3060205489 (socket is 192.168.3.26:46181/0)
2011-06-09 18:41:23.745568 7faa34bb9700 -- 0.0.0.0:6800/2292 >> 192.168.3.26:0/3060205489 pipe(0xa09d00 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/3060205489 (socket is 192.168.3.26:46182/0)
2011-06-09 18:41:24.683453 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.26:0/3060205489 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/3060205489 (socket is 192.168.3.26:46183/0)
2011-06-09 18:41:26.688161 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.26:0/3060205489 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/3060205489 (socket is 192.168.3.26:46184/0)
2011-06-09 18:41:30.688635 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.26:0/3060205489 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/3060205489 (socket is 192.168.3.26:46185/0)
2011-06-09 18:41:38.690202 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.26:0/3060205489 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/3060205489 (socket is 192.168.3.26:46186/0)
2011-06-09 18:41:54.818796 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.26:0/3060205489 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/3060205489 (socket is 192.168.3.26:46187/0)
2011-06-09 18:42:04.871778 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.25:0/1201428746 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/1201428746 (socket is 192.168.3.25:44217/0)
2011-06-09 18:42:04.874566 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.25:0/1201428746 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 1), sending RESETSESSION
2011-06-09 18:42:06.140134 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.29:0/3833052252 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3833052252 (socket is 192.168.3.29:35888/0)
2011-06-09 18:42:08.752087 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.28:0/1232072976 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/1232072976 (socket is 192.168.3.28:41459/0)
2011-06-09 18:42:09.271045 7fb2379c3700 -- 0.0.0.0:6800/2292 >> 192.168.3.30:0/603305768 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/603305768 (socket is 192.168.3.30:41582/0)
2011-06-09 18:42:10.352107 7faa34bb9700 -- 0.0.0.0:6800/2292 >> 192.168.3.27:0/512535685 pipe(0xa097c0 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/512535685 (socket is 192.168.3.27:57374/0)
2011-06-09 18:42:26.729372 7faa34bb9700 -- 0.0.0.0:6800/2292 >> 192.168.3.26:0/3060205489 pipe(0xa09d00 sd=5 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/3060205489 (socket is 192.168.3.26:46188/0)
*** Caught signal (Terminated) **
in thread 0x7fb2379c4720. Shutting down.
2011-06-09 20:53:34.735772 7fdc0f291720 ceph version 0.28.2-201-g5922de2.commit: 5922de29e7414be3aaa6eb04be8bb0ae5c943ccf. process: cmds. pid: 1145
2011-06-09 20:53:34.763016 7fdc0f291720 write_pid_file: failed to open pid file '/var/run/ceph/mds.0.pid': error 2: No such file or directory
2011-06-09 20:53:34.767106 7fdc0c486700 mds-1.0 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 20:53:40.229324 7fdc0c486700 mds-1.0 handle_mds_map standby
2011-06-09 20:53:57.323412 7fdc0c486700 mds0.7 handle_mds_map i am now mds0.7
2011-06-09 20:53:57.323492 7fdc0c486700 mds0.7 handle_mds_map state change up:standby --> up:replay
2011-06-09 20:53:57.323505 7fdc0c486700 mds0.7 replay_start
2011-06-09 20:53:57.323522 7fdc0c486700 mds0.7 recovery set is
2011-06-09 20:53:57.323534 7fdc0c486700 mds0.7 need osdmap epoch 40, have 28
2011-06-09 20:53:57.323545 7fdc0c486700 mds0.7 waiting for osdmap 40 (which blacklists prior instance)
2011-06-09 20:53:57.323586 7fdc0c486700 mds0.cache handle_mds_failure mds0 : recovery peers are
2011-06-09 20:53:57.325129 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 20:53:57.325176 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 20:53:57.325292 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 20:53:59.472054 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 20:54:00.342008 7fdc0c486700 mds0.cache creating system inode with ino:100
2011-06-09 20:54:00.342202 7fdc0c486700 mds0.cache creating system inode with ino:1
2011-06-09 20:54:00.342788 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 20:54:03.908847 7fdc09977700 mds0.7 replay_done
2011-06-09 20:54:03.908928 7fdc09977700 mds0.7 making mds journal writeable
2011-06-09 20:54:04.537689 7fdc0c486700 mds0.7 handle_mds_map i am now mds0.7
2011-06-09 20:54:04.537762 7fdc0c486700 mds0.7 handle_mds_map state change up:replay --> up:reconnect
2011-06-09 20:54:04.537775 7fdc0c486700 mds0.7 reconnect_start
2011-06-09 20:54:04.537801 7fdc0c486700 mds0.7 reopen_log
2011-06-09 20:54:04.537834 7fdc0c486700 mds0.server reconnect_clients -- 6 sessions
2011-06-09 20:54:50.377576 7fdc0b383700 mds0.server reconnect gave up on client4103 192.168.3.25:0/1201428746
2011-06-09 20:54:50.377679 7fdc0b383700 mds0.server reconnect gave up on client4104 192.168.3.26:0/552781789
2011-06-09 20:54:50.377694 7fdc0b383700 mds0.server reconnect gave up on client4105 192.168.3.28:0/1232072976
2011-06-09 20:54:50.377707 7fdc0b383700 mds0.server reconnect gave up on client4106 192.168.3.29:0/3833052252
2011-06-09 20:54:50.377720 7fdc0b383700 mds0.server reconnect gave up on client4107 192.168.3.27:0/512535685
2011-06-09 20:54:50.377733 7fdc0b383700 mds0.server reconnect gave up on client4108 192.168.3.30:0/603305768
2011-06-09 20:54:50.377746 7fdc0b383700 mds0.7 reconnect_done
2011-06-09 20:54:50.525535 7fdc0c486700 mds0.7 handle_mds_map i am now mds0.7
2011-06-09 20:54:50.525612 7fdc0c486700 mds0.7 handle_mds_map state change up:reconnect --> up:rejoin
2011-06-09 20:54:50.525627 7fdc0c486700 mds0.7 rejoin_joint_start
2011-06-09 20:54:50.526072 7fdc0c486700 mds0.7 rejoin_done
2011-06-09 20:54:50.643433 7fdc0c486700 mds0.7 handle_mds_map i am now mds0.7
2011-06-09 20:54:50.643459 7fdc0c486700 mds0.7 handle_mds_map state change up:rejoin --> up:active
2011-06-09 20:54:50.643471 7fdc0c486700 mds0.7 recovery_done -- successful recovery!
2011-06-09 20:54:50.643722 7fdc0c486700 mds0.7 active_start
2011-06-09 20:54:50.643816 7fdc0c486700 mds0.7 cluster recovered.
2011-06-09 20:54:50.644151 7fdc09176700 -- 192.168.3.25:6800/1145 >> 192.168.3.26:0/4176601959 pipe(0xa12c80 sd=11 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/4176601959 (socket is 192.168.3.26:50566/0)
2011-06-09 18:59:01.209355 7fdc0896e700 -- 192.168.3.25:6800/1145 >> 192.168.3.27:0/2510318079 pipe(0xac9d30 sd=12 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/2510318079 (socket is 192.168.3.27:54319/0)
2011-06-09 18:59:01.219327 7fdc0876c700 -- 192.168.3.25:6800/1145 >> 192.168.3.28:0/932047072 pipe(0xaa9540 sd=13 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/932047072 (socket is 192.168.3.28:33138/0)
2011-06-09 18:59:01.219860 7fdc0856a700 -- 192.168.3.25:6800/1145 >> 192.168.3.25:0/436025089 pipe(0xa6d330 sd=14 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/436025089 (socket is 192.168.3.25:34691/0)
2011-06-09 18:59:01.221034 7fdc08368700 -- 192.168.3.25:6800/1145 >> 192.168.3.29:0/3089883130 pipe(0xac5050 sd=15 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3089883130 (socket is 192.168.3.29:45283/0)
2011-06-09 18:59:01.225363 7fdc08166700 -- 192.168.3.25:6800/1145 >> 192.168.3.30:0/3450827354 pipe(0xac5990 sd=16 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/3450827354 (socket is 192.168.3.30:55461/0)
2011-06-09 19:09:53.252279 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 19:09:53.252927 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 19:09:53.268342 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.27:6800/1135
2011-06-09 19:09:53.268955 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 19:11:02.446408 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.28:6800/1136
2011-06-09 19:11:02.447005 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 19:11:02.666311 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 19:11:02.666863 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 19:14:02.326818 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 19:14:02.327510 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 19:24:53.359169 7fdc0c486700 monclient: hunting for new mon
2011-06-09 19:24:53.359301 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.25:6789/0
2011-06-09 19:24:53.359321 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 19:24:53.359593 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 19:24:53.360008 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 19:24:53.375011 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.27:6800/1135
2011-06-09 19:24:53.375598 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 19:26:02.553843 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.28:6800/1136
2011-06-09 19:26:02.554440 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 19:26:02.769860 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 19:26:02.770410 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 19:39:53.463743 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 19:39:53.463862 7fdc0c486700 monclient: hunting for new mon
2011-06-09 19:39:53.463919 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.25:6789/0
2011-06-09 19:39:53.464240 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 19:39:53.464399 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 19:39:53.479588 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.27:6800/1135
2011-06-09 19:39:53.480169 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 19:40:30.125084 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 19:40:30.125769 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 19:41:02.658167 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.28:6800/1136
2011-06-09 19:41:02.658740 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 19:41:02.874192 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 19:41:02.874752 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 19:54:53.565837 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 19:54:53.565940 7fdc0c486700 monclient: hunting for new mon
2011-06-09 19:54:53.565995 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.25:6789/0
2011-06-09 19:54:53.566354 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 19:54:53.566452 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 19:54:53.581539 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.27:6800/1135
2011-06-09 19:54:53.582086 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 19:55:30.230492 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 19:55:30.231182 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 19:56:02.759264 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.28:6800/1136
2011-06-09 19:56:02.759876 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 19:56:02.980144 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 19:56:02.980724 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 20:09:53.670476 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 20:09:53.670617 7fdc0c486700 monclient: hunting for new mon
2011-06-09 20:09:53.670727 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.25:6789/0
2011-06-09 20:09:53.671075 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 20:09:53.671154 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 20:09:53.682476 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.27:6800/1135
2011-06-09 20:09:53.683042 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 20:10:30.336484 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 20:10:30.337149 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 20:11:02.861532 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.28:6800/1136
2011-06-09 20:11:02.862102 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 20:11:03.081566 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 20:11:03.082141 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 20:24:53.764773 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 20:24:53.765405 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 20:24:53.773025 7fdc0c486700 monclient: hunting for new mon
2011-06-09 20:24:53.773155 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.25:6789/0
2011-06-09 20:24:53.773406 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 20:24:53.778148 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.27:6800/1135
2011-06-09 20:24:53.778752 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 20:25:30.431179 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 20:25:30.431839 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 20:26:02.949994 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.28:6800/1136
2011-06-09 20:26:02.950549 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 20:26:03.177658 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 20:26:03.178205 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 20:39:53.868545 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 20:39:53.869178 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 20:39:53.880544 7fdc0c486700 monclient: hunting for new mon
2011-06-09 20:39:53.880636 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.25:6789/0
2011-06-09 20:39:53.880965 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 20:39:53.882578 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.27:6800/1135
2011-06-09 20:39:53.883136 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 20:40:30.534042 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 20:40:30.534720 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 20:41:03.055357 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.28:6800/1136
2011-06-09 20:41:03.055939 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 20:41:03.283388 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 20:41:03.283928 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 20:54:53.972326 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 20:54:53.972951 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 20:54:53.986915 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.27:6800/1135
2011-06-09 20:54:53.987154 7fdc0c486700 monclient: hunting for new mon
2011-06-09 20:54:53.987333 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.25:6789/0
2011-06-09 20:54:53.987516 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-09 20:54:53.987643 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.25:6789/0
2011-06-09 20:55:30.638559 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 20:55:30.639261 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 20:55:48.320591 7fdc09176700 -- 192.168.3.25:6800/1145 >> 192.168.3.26:0/4176601959 pipe(0xa12c80 sd=11 pgs=2 cs=1 l=0).fault initiating reconnect
2011-06-09 20:56:03.385272 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 20:56:03.385915 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 20:59:05.592327 7fdc0b383700 log [INF] : closing stale session client4103 192.168.3.25:0/1201428746 after 302.664960
2011-06-09 20:59:05.592531 7fdc0b383700 log [INF] : closing stale session client4104 192.168.3.26:0/552781789 after 302.664952
2011-06-09 20:59:05.592597 7fdc0b383700 log [INF] : closing stale session client4108 192.168.3.30:0/603305768 after 302.664949
2011-06-09 20:59:05.592651 7fdc0b383700 log [INF] : closing stale session client4105 192.168.3.28:0/1232072976 after 302.664945
2011-06-09 20:59:05.592703 7fdc0b383700 log [INF] : closing stale session client4106 192.168.3.29:0/3833052252 after 302.664942
2011-06-09 20:59:05.592754 7fdc0b383700 log [INF] : closing stale session client4107 192.168.3.27:0/512535685 after 302.664938
2011-06-09 20:59:15.602390 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-09 21:09:54.077183 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 21:09:54.077920 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 21:11:03.490576 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 21:11:03.491343 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 21:24:54.183793 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 21:24:54.229305 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 21:26:03.595390 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 21:26:03.805507 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 21:39:54.332626 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 21:39:54.333358 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 21:41:03.910043 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 21:41:03.910753 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 21:55:42.379449 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 21:55:42.380296 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 21:55:49.968682 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 21:55:50.094757 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 21:56:04.013987 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 21:56:04.014714 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 22:10:42.482155 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-09 22:10:42.482941 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-09 22:10:50.199868 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 22:10:50.201360 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 22:11:04.118918 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 22:11:04.132452 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 22:25:50.304998 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 22:25:50.305801 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 22:26:04.233756 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 22:26:04.234476 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 22:40:50.410953 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 22:40:50.420997 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 22:41:04.337455 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 22:41:04.338203 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 22:55:50.522921 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 22:55:50.524044 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 22:56:04.441731 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 22:56:04.442404 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 23:10:50.625760 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 23:10:50.626598 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 23:11:04.543486 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 23:11:04.544214 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 23:25:50.730314 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 23:25:50.731096 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 23:26:04.648991 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 23:26:04.650064 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 23:40:50.833989 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 23:40:50.834723 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 23:41:04.752809 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 23:41:04.753515 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-09 23:55:50.936824 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-09 23:55:50.937922 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-09 23:56:04.857180 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-09 23:56:04.879764 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 00:10:51.039990 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 00:10:51.040774 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 00:11:04.975808 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 00:11:04.976520 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 00:25:51.142858 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 00:25:51.143810 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 00:26:05.080496 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 00:26:05.081490 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 00:40:51.245331 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 00:40:51.246261 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 00:41:05.183986 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 00:41:05.184704 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 00:55:51.350549 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 00:55:51.351498 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 00:56:05.289045 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 00:56:05.289787 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 00:59:37.411206 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-10 01:11:05.391930 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 01:11:05.392672 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 01:14:41.452486 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 01:14:41.456537 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 01:15:55.685328 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 01:15:55.686084 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 01:26:05.491035 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 01:26:05.491954 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 01:29:41.556719 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 01:29:41.558012 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 01:30:55.785719 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 01:30:55.786520 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 01:41:05.594797 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 01:41:05.595542 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 01:44:41.659308 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 01:44:41.660224 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 01:45:55.887746 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 01:45:55.888466 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 01:56:05.695207 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 01:56:05.695951 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 01:59:41.761326 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 01:59:41.762103 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 02:00:55.986235 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 02:00:55.986939 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 02:11:05.797401 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 02:11:05.798233 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 02:14:41.863273 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 02:14:41.864029 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 02:15:56.088722 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 02:15:56.089466 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 02:26:05.897422 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 02:26:05.898326 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 02:29:41.964497 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 02:29:41.965688 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 02:30:56.189762 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 02:30:56.190465 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 02:41:05.998955 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 02:41:05.999692 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 02:44:42.066084 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 02:44:42.066921 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 02:45:56.289987 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 02:45:56.291037 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 02:56:06.101992 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 02:56:06.102801 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 02:59:42.168487 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 02:59:42.169287 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 03:00:56.390713 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 03:00:56.391429 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 03:11:06.208435 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 03:11:06.209276 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 03:14:42.271442 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 03:14:42.272212 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 03:15:56.492703 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 03:15:56.493471 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 03:26:06.312464 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 03:26:06.313292 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 03:29:42.375108 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 03:29:42.376060 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 03:30:56.594641 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 03:30:56.595361 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 03:41:06.416271 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 03:41:06.417167 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 03:44:42.479234 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 03:44:42.480223 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 03:45:56.697178 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 03:45:56.698349 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 03:56:06.520377 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 03:56:06.521143 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 03:59:42.580666 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 03:59:42.581424 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 04:00:56.802134 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 04:00:56.803350 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 04:11:06.622216 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 04:11:06.623009 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 04:14:42.683943 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 04:14:42.684747 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 04:15:56.904518 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 04:15:56.905363 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 04:26:06.726032 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 04:26:06.726784 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 04:29:42.788809 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 04:29:42.789784 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 04:30:57.007173 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 04:30:57.007893 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 04:41:06.831288 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 04:41:06.832216 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 04:44:42.892665 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 04:44:42.893403 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 04:45:57.111876 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 04:45:57.112794 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 04:56:06.935479 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 04:56:06.936441 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 04:59:42.998521 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 04:59:42.999707 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 05:00:57.216882 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 05:00:57.217630 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 05:11:07.040497 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 05:11:07.055836 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 05:14:43.101202 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 05:14:43.102162 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 05:15:57.321157 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 05:15:57.322171 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 05:26:07.158455 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 05:26:07.159181 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 05:29:43.205394 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 05:29:43.206163 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 05:30:57.425841 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 05:30:57.427061 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 05:34:15.849670 7fdc0b383700 mds0.objecter FULL, paused modify 0x7fdbf8009770 tid 7707
2011-06-10 05:34:15.849885 7fdc0b383700 mds0.objecter FULL, paused modify 0x7fdbf80054c0 tid 7708
2011-06-10 05:34:30.863567 7fdc0b383700 mds0.objecter FULL, paused modify 0x7fdbfc0042a0 tid 7709
2011-06-10 05:41:07.261087 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 05:41:07.261812 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 05:44:43.311783 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 05:44:43.312525 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 05:45:57.529922 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 05:45:57.530635 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 05:56:07.364606 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 05:56:07.365320 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 05:59:43.416046 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 05:59:43.416780 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 06:00:57.633988 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 06:00:57.634691 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 06:11:07.469994 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 06:11:07.470703 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 06:14:43.520901 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 06:14:43.521643 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 06:15:57.739239 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 06:15:57.739918 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 06:26:07.572325 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 06:26:07.573038 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 06:29:43.625628 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 06:29:43.626357 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 06:30:57.843667 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 06:30:57.844362 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 06:41:07.678127 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 06:41:07.678851 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 06:44:43.729051 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 06:44:43.729784 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 06:45:57.945511 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 06:45:57.946186 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 06:56:07.781485 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 06:56:07.782176 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 06:59:43.834612 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 06:59:43.835369 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 07:00:58.049891 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 07:00:58.050570 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 07:11:07.887058 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 07:11:07.887767 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 07:14:43.938493 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 07:14:43.939234 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 07:15:58.154626 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 07:15:58.155333 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 07:26:07.991918 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 07:26:07.992628 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 07:29:44.043281 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 07:29:44.044026 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 07:30:58.260532 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 07:30:58.261234 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 07:41:08.097011 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 07:41:08.097724 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 07:44:44.149500 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 07:44:44.150262 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 07:45:58.363445 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 07:45:58.364147 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 07:56:08.200688 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 07:56:08.201393 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 07:59:44.250648 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 07:59:44.251412 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 08:00:58.468823 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 08:00:58.469512 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 08:11:08.303386 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 08:11:08.304099 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 08:14:44.355205 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 08:14:44.355959 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 08:15:58.574150 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 08:15:58.574825 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 08:26:08.404889 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 08:26:08.405599 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 08:29:44.458068 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 08:29:44.458800 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 08:30:58.679580 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 08:30:58.680273 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 08:41:08.510194 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 08:41:08.510893 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 08:44:44.561746 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 08:44:44.562495 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 08:45:58.781425 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 08:45:58.782115 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 08:56:08.613198 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 08:56:08.613919 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 08:59:44.665098 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 08:59:44.665846 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 09:00:58.888078 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 09:00:58.888785 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 09:11:08.720176 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 09:11:08.720871 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 09:14:44.768356 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 09:14:44.769136 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 09:15:58.990003 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 09:15:58.990705 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 09:26:08.822590 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 09:26:08.823304 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 09:29:44.870981 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 09:29:44.871726 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 09:30:59.092012 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 09:30:59.092689 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 09:41:08.925387 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 09:41:08.926085 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 09:44:44.977735 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 09:44:44.978488 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 09:45:59.196412 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 09:45:59.197088 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 09:56:09.028175 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 09:56:09.028877 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 09:59:45.083003 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 09:59:45.083780 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 10:00:59.298494 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 10:00:59.299193 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 10:11:09.133794 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 10:11:09.134477 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 10:14:45.187773 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 10:14:45.188530 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 10:15:59.403184 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 10:15:59.403857 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 10:26:09.238106 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 10:26:09.238797 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 10:29:45.293595 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 10:29:45.294338 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 10:30:59.507960 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 10:30:59.508652 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 10:41:09.345326 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 10:41:09.346023 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 10:44:45.396721 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 10:44:45.397481 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 10:45:59.611668 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 10:45:59.612334 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 10:50:58.562364 7fdc08e73700 -- 192.168.3.25:6800/1145 >> 192.168.3.26:0/4176601959 pipe(0xa43c20 sd=17 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/4176601959 (socket is 192.168.3.26:39132/0)
2011-06-10 10:50:58.562516 7fdc08e73700 -- 192.168.3.25:6800/1145 >> 192.168.3.26:0/4176601959 pipe(0xa43c20 sd=17 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 2 state 3
2011-06-10 10:50:58.562537 7fdc08e73700 -- 192.168.3.25:6800/1145 >> 192.168.3.26:0/4176601959 pipe(0xa43c20 sd=17 pgs=0 cs=0 l=0).accept peer reset, then tried to connect to us, replacing
2011-06-10 10:50:58.562597 7fdc0c486700 mds0.7 ms_handle_remote_reset on 192.168.3.26:0/4176601959
2011-06-10 10:50:58.562827 7fdc0c486700 mds0.objecter FULL, paused modify 0x7fdbf800f320 tid 7710
2011-06-10 10:50:58.562862 7fdc0c486700 mds0.objecter FULL, paused modify 0x7fdbf8005620 tid 7711
2011-06-10 10:52:13.022504 7fdc0c486700 mds0.objecter FULL, paused modify 0x7fdbf80579f0 tid 7712
2011-06-10 10:52:13.022656 7fdc0c486700 mds0.objecter FULL, paused modify 0x7fdbf8057890 tid 7713
2011-06-10 10:56:09.447830 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 10:56:09.448531 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 10:59:45.499458 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 10:59:45.500195 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 11:00:59.715146 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 11:00:59.715811 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 11:05:58.664307 7fdc08e73700 -- 192.168.3.25:6800/1145 >> 192.168.3.26:0/4176601959 pipe(0xa43c20 sd=17 pgs=136 cs=1 l=0).fault with nothing to send, going to standby
2011-06-10 11:11:09.551407 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 11:11:09.552131 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 11:14:45.605702 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 11:14:45.606451 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 11:15:59.819070 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 11:15:59.819757 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 11:26:09.657253 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 11:26:09.657948 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 11:29:45.709217 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.26:6800/1172
2011-06-10 11:29:45.709985 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 11:30:59.924197 7fdc0c486700 mds0.7 ms_handle_reset on 192.168.3.30:6800/1135
2011-06-10 11:30:59.924880 7fdc0c486700 mds0.7 ms_handle_connect on 192.168.3.30:6800/1135
*** Caught signal (Terminated) **
in thread 0x7fdc0f291720. Shutting down.
2011-06-10 11:40:30.308054 7f931e792720 ceph version 0.28.2-201-g5922de2.commit: 5922de29e7414be3aaa6eb04be8bb0ae5c943ccf. process: cmds. pid: 1823
2011-06-10 11:40:30.309450 7f931e792720 write_pid_file: failed to open pid file '/var/run/ceph/mds.0.pid': error 2: No such file or directory
2011-06-10 11:40:30.310533 7f931b987700 mds-1.0 ms_handle_connect on 192.168.3.25:6789/0
2011-06-10 11:40:30.734152 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.29:0/3089883130 pipe(0xa0f480 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3089883130 (socket is 192.168.3.29:58182/0)
2011-06-10 11:40:30.734233 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.29:0/3089883130 pipe(0xa0f480 sd=6 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 2), sending RESETSESSION
2011-06-10 11:40:31.194357 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.30:0/3450827354 pipe(0xa0f480 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/3450827354 (socket is 192.168.3.30:40703/0)
2011-06-10 11:40:31.194397 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.30:0/3450827354 pipe(0xa0f480 sd=6 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 2), sending RESETSESSION
2011-06-10 11:40:34.504649 7f931b987700 mds-1.0 handle_mds_map standby
2011-06-10 11:40:36.492885 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.27:0/2510318079 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/2510318079 (socket is 192.168.3.27:59374/0)
2011-06-10 11:40:36.492974 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.27:0/2510318079 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 2), sending RESETSESSION
2011-06-10 11:40:36.932726 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.28:0/932047072 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/932047072 (socket is 192.168.3.28:38187/0)
2011-06-10 11:40:36.932766 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.28:0/932047072 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 2), sending RESETSESSION
2011-06-10 11:40:37.063012 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.25:0/436025089 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/436025089 (socket is 192.168.3.25:46913/0)
2011-06-10 11:40:37.063133 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.25:0/436025089 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 2), sending RESETSESSION
2011-06-10 11:40:38.751151 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.29:0/3089883130 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3089883130 (socket is 192.168.3.29:58183/0)
2011-06-10 11:40:38.751204 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.29:0/3089883130 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 2), sending RESETSESSION
2011-06-10 11:40:39.212735 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.30:0/3450827354 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/3450827354 (socket is 192.168.3.30:40704/0)
2011-06-10 11:40:39.212776 7f9319882700 -- 192.168.3.25:6800/1823 >> 192.168.3.30:0/3450827354 pipe(0xa0fb90 sd=6 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 2), sending RESETSESSION
2011-06-10 11:40:49.706307 7f931b987700 mds0.8 handle_mds_map i am now mds0.8
2011-06-10 11:40:49.706384 7f931b987700 mds0.8 handle_mds_map state change up:standby --> up:replay
2011-06-10 11:40:49.706400 7f931b987700 mds0.8 replay_start
2011-06-10 11:40:49.706422 7f931b987700 mds0.8 recovery set is
2011-06-10 11:40:49.706434 7f931b987700 mds0.8 need osdmap epoch 50, have 50
2011-06-10 11:40:49.706756 7f931b987700 mds0.cache handle_mds_failure mds0 : recovery peers are
2011-06-10 11:40:49.707250 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.30:6800/1135
2011-06-10 11:40:49.707268 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.28:6800/1136
2011-06-10 11:40:49.707283 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 11:40:57.672256 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.27:6800/1135
2011-06-10 11:41:00.370344 7f931b987700 mds0.cache creating system inode with ino:100
2011-06-10 11:41:00.370595 7f931b987700 mds0.cache creating system inode with ino:1
2011-06-10 11:41:00.372181 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.26:6800/1172
2011-06-10 11:41:03.311093 7f931907a700 mds0.8 replay_done
2011-06-10 11:41:03.311179 7f931907a700 mds0.8 making mds journal writeable
2011-06-10 11:41:03.311231 7f931907a700 mds0.objecter FULL, paused modify 0xa9f440 tid 18
2011-06-10 11:41:03.311256 7f931907a700 mds0.objecter FULL, paused modify 0xaa35f0 tid 19
2011-06-10 11:41:03.311278 7f931907a700 mds0.objecter FULL, paused modify 0xaa3a70 tid 20
2011-06-10 11:41:03.311298 7f931907a700 mds0.objecter FULL, paused modify 0xac1fb0 tid 21
2011-06-10 11:41:03.311319 7f931907a700 mds0.objecter FULL, paused modify 0xac20d0 tid 22
2011-06-10 11:41:03.311339 7f931907a700 mds0.objecter FULL, paused modify 0xac21f0 tid 23
2011-06-10 11:41:03.518814 7f931b987700 mds0.8 handle_mds_map i am now mds0.8
2011-06-10 11:41:03.518875 7f931b987700 mds0.8 handle_mds_map state change up:replay --> up:reconnect
2011-06-10 11:41:03.518912 7f931b987700 mds0.8 reconnect_start
2011-06-10 11:41:03.518925 7f931b987700 mds0.8 reopen_log
2011-06-10 11:41:03.518956 7f931b987700 mds0.server reconnect_clients -- 5 sessions
2011-06-10 11:41:03.519373 7f9318677700 -- 192.168.3.25:6800/1823 >> 192.168.3.29:0/3089883130 pipe(0xac2310 sd=11 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.29:0/3089883130 (socket is 192.168.3.29:58188/0)
2011-06-10 11:41:03.519448 7f9318576700 -- 192.168.3.25:6800/1823 >> 192.168.3.28:0/932047072 pipe(0xac25c0 sd=12 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.28:0/932047072 (socket is 192.168.3.28:38192/0)
2011-06-10 11:41:03.519567 7f9318475700 -- 192.168.3.25:6800/1823 >> 192.168.3.27:0/2510318079 pipe(0xac2aa0 sd=13 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.27:0/2510318079 (socket is 192.168.3.27:59379/0)
2011-06-10 11:41:03.519632 7f9318273700 -- 192.168.3.25:6800/1823 >> 192.168.3.30:0/3450827354 pipe(0xac4610 sd=15 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.30:0/3450827354 (socket is 192.168.3.30:40712/0)
2011-06-10 11:41:03.519745 7f9318374700 -- 192.168.3.25:6800/1823 >> 192.168.3.25:0/436025089 pipe(0xac4010 sd=14 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.25:0/436025089 (socket is 192.168.3.25:46919/0)
2011-06-10 11:41:03.520172 7f931b987700 log [DBG] : reconnect by client4206 192.168.3.28:0/932047072 after 0.001118
2011-06-10 11:41:03.520330 7f931b987700 log [DBG] : reconnect by client4208 192.168.3.29:0/3089883130 after 0.001334
2011-06-10 11:41:03.520385 7f931b987700 log [DBG] : reconnect by client4205 192.168.3.27:0/2510318079 after 0.001418
2011-06-10 11:41:03.520634 7f931b987700 log [DBG] : reconnect by client4209 192.168.3.30:0/3450827354 after 0.001659
2011-06-10 11:41:03.520885 7f931b987700 log [DBG] : reconnect by client4207 192.168.3.25:0/436025089 after 0.001918
2011-06-10 11:41:03.521330 7f931b987700 mds0.8 reconnect_done
2011-06-10 11:41:03.694303 7f931b987700 mds0.8 handle_mds_map i am now mds0.8
2011-06-10 11:41:03.694340 7f931b987700 mds0.8 handle_mds_map state change up:reconnect --> up:rejoin
2011-06-10 11:41:03.694367 7f931b987700 mds0.8 rejoin_joint_start
2011-06-10 11:41:03.695027 7f931b987700 mds0.8 rejoin_done
2011-06-10 11:41:04.324669 7f931b987700 mds0.8 handle_mds_map i am now mds0.8
2011-06-10 11:41:04.324707 7f931b987700 mds0.8 handle_mds_map state change up:rejoin --> up:active
2011-06-10 11:41:04.324721 7f931b987700 mds0.8 recovery_done -- successful recovery!
2011-06-10 11:41:04.325513 7f931b987700 mds0.8 active_start
2011-06-10 11:41:04.325719 7f9313bfb700 -- 192.168.3.25:6800/1823 >> 192.168.3.26:0/4176601959 pipe(0x7f931400c7a0 sd=16 pgs=0 cs=0 l=0).accept peer addr is really 192.168.3.26:0/4176601959 (socket is 192.168.3.26:38232/0)
2011-06-10 11:41:04.325925 7f931b987700 mds0.objecter FULL, paused modify 0x7f931402cee0 tid 25
2011-06-10 11:41:04.325963 7f931b987700 mds0.8 cluster recovered.
2011-06-10 11:56:04.432160 7f9313bfb700 -- 192.168.3.25:6800/1823 >> 192.168.3.26:0/4176601959 pipe(0x7f931400c7a0 sd=16 pgs=137 cs=1 l=0).fault with nothing to send, going to standby
2011-06-10 11:56:05.148414 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 11:56:05.149067 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 12:11:05.250096 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 12:11:05.250795 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 12:26:05.355931 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 12:26:05.356646 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 12:41:05.462276 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 12:41:05.462979 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 12:56:05.566899 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 12:56:05.567599 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 13:11:05.671587 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 13:11:05.672286 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 13:26:05.775185 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 13:26:05.775891 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 13:41:05.878036 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 13:41:05.878752 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 13:56:05.980694 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 13:56:05.981406 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 14:11:06.084240 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 14:11:06.084936 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 14:26:06.188832 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 14:26:06.189552 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 14:41:06.292555 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 14:41:06.293262 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 14:56:06.396414 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 14:56:06.397108 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 15:11:06.503481 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 15:11:06.504193 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 15:26:06.608636 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 15:26:06.609335 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 15:41:06.711079 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 15:41:06.711788 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 15:56:06.812315 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 15:56:06.813019 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 16:11:06.916410 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 16:11:06.917119 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 16:26:07.021680 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 16:26:07.022401 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 16:41:07.126253 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 16:41:07.126936 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 16:56:07.230220 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 16:56:07.230983 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 17:11:07.336259 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 17:11:07.336990 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 17:26:07.443332 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 17:26:07.444021 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 17:41:07.546622 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 17:41:07.547303 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 17:56:07.649938 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 17:56:07.650653 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 18:11:07.754523 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 18:11:07.755271 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 18:26:07.857820 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 18:26:07.858541 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 18:41:07.962017 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 18:41:07.962718 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 18:56:08.064481 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 18:56:08.065167 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 19:11:08.166451 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 19:11:08.167157 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 19:26:08.271773 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 19:26:08.272485 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 19:41:08.376852 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 19:41:08.377653 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 19:56:08.478405 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 19:56:08.479080 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 20:11:08.584373 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 20:11:08.585116 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 20:26:08.687567 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 20:26:08.688247 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 20:41:08.793618 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 20:41:08.794344 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 20:56:08.897606 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 20:56:08.898290 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 21:11:08.999153 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 21:11:08.999871 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 21:26:09.104768 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 21:26:09.105455 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 21:41:09.206874 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 21:41:09.207582 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 21:56:09.309420 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 21:56:09.310163 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 22:11:09.415272 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 22:11:09.415985 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 22:26:09.520917 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 22:26:09.521639 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 22:41:09.626124 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 22:41:09.626830 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 22:56:09.730410 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 22:56:09.731112 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 23:11:09.834737 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 23:11:09.835453 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 23:26:09.938361 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 23:26:09.939067 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 23:41:10.041071 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 23:41:10.041772 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-10 23:56:10.146517 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-10 23:56:10.147246 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 00:11:10.250440 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 00:11:10.251138 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 00:26:10.352946 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 00:26:10.353643 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 00:41:10.455968 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 00:41:10.456680 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 00:56:10.560872 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 00:56:10.561593 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 01:11:10.665581 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 01:11:10.666290 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 01:26:10.767306 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 01:26:10.768001 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 01:41:10.874590 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 01:41:10.875291 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 01:56:10.979513 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 01:56:10.980230 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 02:11:11.083547 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 02:11:11.084277 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 02:26:11.187581 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 02:26:11.188309 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 02:41:11.291633 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 02:41:11.292344 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 02:56:11.393601 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 02:56:11.394309 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 03:11:11.496261 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 03:11:11.496970 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 03:26:11.600361 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 03:26:11.601078 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 03:41:11.704779 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 03:41:11.705494 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 03:56:11.806797 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 03:56:11.807508 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 04:11:11.911617 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 04:11:11.912335 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 04:26:12.015583 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 04:26:12.016283 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 04:41:12.119969 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 04:41:12.120681 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 04:56:12.224233 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 04:56:12.224955 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 05:11:12.330256 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 05:11:12.330973 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 05:26:12.434761 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 05:26:12.435470 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 05:41:12.539539 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 05:41:12.540239 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 05:56:12.641509 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 05:56:12.642207 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 06:11:12.745448 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 06:11:12.746144 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 06:26:12.848200 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 06:26:12.848890 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 06:41:12.954998 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 06:41:12.955704 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 06:56:13.056863 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 06:56:13.057563 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 07:11:13.161980 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 07:11:13.162682 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 07:26:13.265163 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 07:26:13.265868 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 07:41:13.370017 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 07:41:13.370719 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 07:56:13.472139 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 07:56:13.472825 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 08:11:13.575448 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 08:11:13.576152 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 08:26:13.677820 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 08:26:13.678517 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 08:41:13.779918 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 08:41:13.780608 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 08:56:13.882029 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 08:56:13.882726 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 09:11:13.986798 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 09:11:13.987500 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 09:26:14.089897 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 09:26:14.090609 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 09:41:14.191938 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 09:41:14.192639 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 09:56:14.297274 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 09:56:14.297989 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 10:11:14.401056 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 10:11:14.401774 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 10:26:14.506636 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 10:26:14.507330 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 10:41:14.610295 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 10:41:14.610994 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 10:56:14.715585 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 10:56:14.716284 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 11:11:14.819332 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 11:11:14.820033 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 11:26:14.920937 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 11:26:14.921644 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 11:41:15.027541 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 11:41:15.028239 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 11:56:15.130519 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 11:56:15.131214 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 12:11:15.235384 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 12:11:15.236079 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 12:26:15.338565 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 12:26:15.339272 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 12:41:15.441237 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 12:41:15.441941 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 12:56:15.545260 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 12:56:15.545946 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 13:11:15.647913 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 13:11:15.648610 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 13:26:15.750766 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 13:26:15.751464 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 13:41:15.852871 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 13:41:15.853586 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 13:56:15.956595 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 13:56:15.957285 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 14:11:16.060529 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 14:11:16.061226 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 14:26:16.164840 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 14:26:16.165536 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 14:41:16.268964 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 14:41:16.269669 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 14:56:16.371105 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 14:56:16.371803 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 15:11:16.476956 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 15:11:16.477654 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 15:26:16.581934 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 15:26:16.582637 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 15:41:16.687746 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 15:41:16.688435 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 15:56:16.791632 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 15:56:16.792347 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 16:11:16.893709 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 16:11:16.894434 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 16:26:16.998920 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 16:26:16.999633 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 16:41:17.102526 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 16:41:17.103236 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 16:56:17.205553 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 16:56:17.206277 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 17:11:17.308073 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 17:11:17.308778 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 17:26:17.413054 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 17:26:17.413786 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 17:41:17.518150 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 17:41:17.518858 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 17:56:17.623357 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 17:56:17.624094 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 18:11:17.728524 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 18:11:17.729205 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 18:26:17.832737 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 18:26:17.833464 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 18:41:17.936144 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 18:41:17.936843 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 18:56:18.041412 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 18:56:18.042156 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 19:11:18.145690 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 19:11:18.146383 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 19:26:18.247828 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 19:26:18.248570 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 19:41:18.349640 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 19:41:18.350337 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 19:56:18.451448 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 19:56:18.452162 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 20:11:18.554213 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 20:11:18.554917 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 20:26:18.659575 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 20:26:18.660276 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 20:41:18.762900 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 20:41:18.763593 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 20:56:18.866570 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 20:56:18.867324 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 21:11:18.967957 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 21:11:18.968644 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 21:26:19.071985 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 21:26:19.072690 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 21:41:19.177596 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 21:41:19.178294 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 21:56:19.280615 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 21:56:19.281360 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 22:11:19.385362 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 22:11:19.386070 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 22:26:19.487953 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 22:26:19.488656 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 22:41:19.594915 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 22:41:19.595612 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 22:56:19.698993 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 22:56:19.699725 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 23:11:19.802892 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 23:11:19.803623 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 23:26:19.904879 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 23:26:19.905599 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 23:41:20.008997 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 23:41:20.009709 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-11 23:56:20.112941 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-11 23:56:20.113641 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 00:11:20.215694 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 00:11:20.216398 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 00:26:20.318164 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 00:26:20.318874 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 00:41:20.423247 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 00:41:20.423979 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 00:56:20.529008 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 00:56:20.529747 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 01:11:20.633048 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 01:11:20.633770 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 01:26:20.736027 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 01:26:20.736736 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 01:41:20.840276 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 01:41:20.840956 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 01:56:20.942480 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 01:56:20.943190 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 02:11:21.046569 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 02:11:21.047264 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 02:26:21.151714 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 02:26:21.152425 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 02:41:21.255193 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 02:41:21.255913 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 02:56:21.357429 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 02:56:21.358308 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 03:11:21.461339 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 03:11:21.462071 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 03:26:21.564953 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 03:26:21.565863 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 03:41:21.668449 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 03:41:21.669159 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 03:56:21.773662 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 03:56:21.774348 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 04:11:21.876302 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 04:11:21.877007 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 04:26:21.979264 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 04:26:21.979944 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 04:41:22.084777 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 04:41:22.085495 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 04:56:22.188029 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 04:56:22.188715 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 05:11:22.289349 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 05:11:22.290031 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 05:26:22.392417 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 05:26:22.393157 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 05:41:22.498024 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 05:41:22.498730 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 05:56:22.601923 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 05:56:22.602642 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 06:11:22.703484 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 06:11:22.704198 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 06:26:22.810670 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 06:26:22.811382 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 06:41:22.911970 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 06:41:22.912676 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 06:56:23.016107 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 06:56:23.016816 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 07:11:23.121272 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 07:11:23.121981 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 07:26:23.226658 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 07:26:23.227369 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 07:41:23.331701 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 07:41:23.332574 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 07:56:23.434627 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 07:56:23.435323 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 08:11:23.540552 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 08:11:23.541242 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 08:26:23.644375 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 08:26:23.645074 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 08:41:23.747060 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 08:41:23.747747 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 08:56:23.853171 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 08:56:23.853866 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 09:11:23.955332 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 09:11:23.956035 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 09:26:24.060267 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 09:26:24.060975 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 09:41:24.164858 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 09:41:24.165561 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 09:56:24.268407 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 09:56:24.269104 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 10:11:24.371782 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 10:11:24.372464 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 10:26:24.475676 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 10:26:24.476357 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 10:41:24.578707 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 10:41:24.579390 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 10:56:24.682564 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 10:56:24.683245 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 11:11:24.783996 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 11:11:24.784689 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 11:26:24.888317 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 11:26:24.889064 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 11:41:24.989807 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 11:41:24.990495 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 11:56:25.092945 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 11:56:25.093641 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 12:11:25.197220 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 12:11:25.197898 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 12:26:25.300196 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 12:26:25.300896 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 12:41:25.405936 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 12:41:25.406627 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 12:56:25.507785 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 12:56:25.508489 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 13:11:25.613506 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 13:11:25.614206 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 13:26:25.715483 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 13:26:25.716174 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 13:41:25.819275 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 13:41:25.819989 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 13:56:25.922739 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 13:56:25.923462 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 14:11:26.026967 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 14:11:26.027668 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 14:26:26.129725 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 14:26:26.130424 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 14:41:26.234445 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 14:41:26.235144 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 14:56:26.339845 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 14:56:26.340550 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 15:11:26.443716 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 15:11:26.444414 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 15:26:26.546020 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 15:26:26.546724 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 15:41:26.649877 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 15:41:26.650588 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 15:56:26.755844 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 15:56:26.756556 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 16:11:26.857883 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 16:11:26.858577 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 16:26:26.962314 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 16:26:26.963020 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 16:41:27.066787 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 16:41:27.067487 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 16:56:27.171485 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 16:56:27.172175 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 17:11:27.274238 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 17:11:27.274941 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 17:26:27.378914 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 17:26:27.379622 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 17:41:27.481967 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 17:41:27.482671 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 17:56:27.586849 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 17:56:27.587556 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 18:11:27.690070 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 18:11:27.690765 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 18:26:27.794081 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 18:26:27.795016 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 18:41:27.897567 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 18:41:27.898297 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 18:56:28.002535 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 18:56:28.003255 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 19:11:28.107671 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 19:11:28.108391 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 19:26:28.212851 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 19:26:28.213551 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 19:41:28.317878 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 19:41:28.318574 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 19:56:28.422770 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 19:56:28.423447 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 20:11:28.525776 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 20:11:28.526479 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 20:26:28.630133 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 20:26:28.630793 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 20:41:28.734946 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 20:41:28.735764 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 20:56:28.838716 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 20:56:28.839438 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 21:11:28.943880 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 21:11:28.944579 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 21:26:29.049699 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 21:26:29.050420 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 21:41:29.153268 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 21:41:29.153988 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 21:56:29.256779 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 21:56:29.257465 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 22:11:29.362359 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 22:11:29.363075 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 22:26:29.466176 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 22:26:29.466905 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 22:41:29.571728 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 22:41:29.572464 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 22:56:29.676145 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 22:56:29.676849 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 23:11:29.780567 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 23:11:29.781343 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 23:26:29.883612 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 23:26:29.884384 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 23:41:29.987729 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 23:41:29.988447 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-12 23:56:30.093039 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-12 23:56:30.093733 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 00:11:30.198560 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 00:11:30.199265 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 00:26:30.303247 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 00:26:30.303922 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 00:41:30.406563 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 00:41:30.407254 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 00:56:30.513937 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 00:56:30.514638 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 01:11:30.617392 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 01:11:30.618082 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 01:26:30.722129 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 01:26:30.722807 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 01:41:30.826438 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 01:41:30.827142 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 01:56:30.931435 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 01:56:30.932133 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 02:11:31.033261 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 02:11:31.033947 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 02:26:31.138019 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 02:26:31.138726 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 02:41:31.242560 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 02:41:31.243253 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 02:56:31.346551 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 02:56:31.347250 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 03:11:31.451765 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 03:11:31.452459 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 03:26:31.553506 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 03:26:31.554202 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 03:41:31.655825 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 03:41:31.656513 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 03:56:31.760668 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 03:56:31.761371 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 04:11:31.865977 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 04:11:31.866654 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 04:26:31.970961 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 04:26:31.971696 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 04:41:32.073460 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 04:41:32.074153 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 04:56:32.177486 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 04:56:32.178183 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 05:11:32.280444 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 05:11:32.281145 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 05:26:32.385092 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 05:26:32.385800 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 05:41:32.488525 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 05:41:32.489215 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 05:56:32.591874 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 05:56:32.592574 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 06:11:32.697995 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 06:11:32.698689 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 06:26:32.800398 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 06:26:32.801105 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 06:41:32.904495 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 06:41:32.905194 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 06:56:33.008235 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 06:56:33.008941 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 07:11:33.112754 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 07:11:33.113461 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 07:26:33.216643 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 07:26:33.217340 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 07:41:33.320716 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 07:41:33.321437 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 07:56:33.423226 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 07:56:33.423930 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 08:11:33.526670 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 08:11:33.527380 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 08:26:33.628856 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 08:26:33.629569 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 08:41:33.731430 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 08:41:33.732130 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 08:56:33.837747 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 08:56:33.838464 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 09:11:33.941276 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 09:11:33.941997 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 09:26:34.043562 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 09:26:34.044247 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 09:41:34.147767 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 09:41:34.148463 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 09:56:34.254813 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 09:56:34.255514 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 10:11:34.358116 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 10:11:34.358835 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 10:26:34.461019 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 10:26:34.461760 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 10:41:34.564527 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 10:41:34.565247 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 10:56:34.667581 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 10:56:34.668303 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 11:11:34.771008 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 11:11:34.771715 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 11:26:34.876688 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 11:26:34.877405 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 11:41:34.981079 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 11:41:34.981783 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 11:56:35.085448 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 11:56:35.086172 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 12:11:35.186509 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 12:11:35.187225 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 12:26:35.291016 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 12:26:35.291727 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 12:41:35.393787 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 12:41:35.394482 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 12:56:35.498095 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 12:56:35.498810 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 13:11:35.603806 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 13:11:35.604522 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 13:26:35.707889 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 13:26:35.708596 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 13:41:35.809838 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 13:41:35.810553 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 13:56:35.914967 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 13:56:35.915678 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 14:11:36.019127 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 14:11:36.019828 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 14:26:36.121767 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 14:26:36.122495 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 14:41:36.226155 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 14:41:36.226857 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 14:56:36.330885 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 14:56:36.331586 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 15:11:36.436276 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 15:11:36.436979 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 15:26:36.541982 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 15:26:36.542670 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 15:41:36.646926 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 15:41:36.647630 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 15:56:36.751539 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 15:56:36.752234 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 16:11:36.853827 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 16:11:36.854528 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 16:26:36.957259 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 16:26:36.957989 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 16:41:37.061363 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 16:41:37.062087 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 16:56:37.165252 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 16:56:37.165970 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 17:11:37.269833 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 17:11:37.270541 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 17:26:37.376316 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 17:26:37.377036 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 17:41:37.481049 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 17:41:37.481757 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 17:56:37.586325 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 17:56:37.587038 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 18:11:37.689338 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 18:11:37.690156 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 18:26:37.793292 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 18:26:37.793998 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 18:41:37.897309 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 18:41:37.898027 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 18:56:38.003813 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 18:56:38.004553 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 19:11:38.107122 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 19:11:38.107812 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 19:26:38.209438 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 19:26:38.210128 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 19:41:38.315611 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 19:41:38.316377 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 19:56:38.416905 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 19:56:38.417593 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 20:11:38.520673 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 20:11:38.521381 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 20:26:38.626779 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 20:26:38.627479 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 20:41:38.730571 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 20:41:38.731264 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 20:56:38.832884 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 20:56:38.833585 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 21:11:38.936459 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 21:11:38.937163 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 21:26:39.037986 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 21:26:39.038688 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 21:41:39.142031 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 21:41:39.142771 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 21:56:39.245322 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 21:56:39.246024 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 22:11:39.350030 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 22:11:39.350776 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 22:26:39.452905 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 22:26:39.453611 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 22:41:39.556998 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 22:41:39.557696 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 22:56:39.661206 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 22:56:39.661902 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 23:11:39.766499 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 23:11:39.767192 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 23:26:39.871002 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 23:26:39.871700 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 23:41:39.973239 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 23:41:39.973925 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-13 23:56:40.077477 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-13 23:56:40.078155 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 00:11:40.179673 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 00:11:40.180380 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 00:26:40.287007 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 00:26:40.287725 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 00:41:40.391529 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 00:41:40.392223 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 00:56:40.496648 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 00:56:40.497347 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 01:11:40.600142 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 01:11:40.600854 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 01:26:40.704739 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 01:26:40.705440 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 01:41:40.807637 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 01:41:40.808333 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 01:56:40.914022 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 01:56:40.914758 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 02:11:41.017768 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 02:11:41.018460 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 02:26:41.123930 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 02:26:41.124618 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 02:41:41.230355 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 02:41:41.231064 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 02:56:41.333221 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 02:56:41.333913 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 03:11:41.437615 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 03:11:41.438347 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 03:26:41.538515 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 03:26:41.539217 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 03:41:41.640758 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 03:41:41.641464 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 03:56:41.744912 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 03:56:41.745625 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 04:11:41.849699 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 04:11:41.850400 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 04:26:41.952676 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 04:26:41.953384 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 04:41:42.056952 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 04:41:42.057672 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 04:56:42.162899 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 04:56:42.163613 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 05:11:42.267246 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 05:11:42.267970 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 05:26:42.371904 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 05:26:42.372612 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 05:41:42.479609 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 05:41:42.480319 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 05:56:42.581221 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 05:56:42.581924 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 06:11:42.684073 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 06:11:42.684770 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 06:26:42.789379 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 06:26:42.790077 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 06:41:42.894582 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 06:41:42.895279 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 06:56:42.997671 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 06:56:42.998387 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 07:11:43.103545 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 07:11:43.104239 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 07:26:43.207687 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 07:26:43.208393 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 07:41:43.311411 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 07:41:43.312130 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 07:56:43.416228 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 07:56:43.416937 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 08:11:43.521155 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 08:11:43.521868 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 08:26:43.627055 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 08:26:43.627751 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 08:41:43.730501 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 08:41:43.731220 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 08:56:43.833303 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 08:56:43.834006 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 09:11:43.937095 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 09:11:43.937843 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 09:26:44.039937 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 09:26:44.040641 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 09:41:44.145705 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 09:41:44.146414 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 09:56:44.248949 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 09:56:44.249663 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 10:11:44.351829 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 10:11:44.352690 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
2011-06-14 10:26:44.457588 7f931b987700 mds0.8 ms_handle_reset on 192.168.3.29:6800/1144
2011-06-14 10:26:44.458303 7f931b987700 mds0.8 ms_handle_connect on 192.168.3.29:6800/1144
(2-2/3)