Project

General

Profile

Bug #4644 » alexxy.mds.log

Joao Eduardo Luis, 04/04/2013 03:34 AM

 
2013-04-04 14:07:13.508815 7f1711e45700 1 mds.0.27 handle_mds_map i am now mds.0.27
2013-04-04 14:07:13.508817 7f1711e45700 1 mds.0.27 handle_mds_map state change up:boot --> up:replay
2013-04-04 14:07:13.508821 7f1711e45700 1 mds.0.27 replay_start
2013-04-04 14:07:13.508825 7f1711e45700 1 mds.0.27 recovery set is
2013-04-04 14:07:13.508826 7f1711e45700 1 mds.0.27 need osdmap epoch 496, have 496
2013-04-04 14:07:16.513798 7f1710541700 0 -- 10.0.0.254:6800/18557 >> 10.0.0.16:6800/9079 pipe(0x7f16fc009d10 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-04 14:07:16.513820 7f171033f700 0 -- 10.0.0.254:6800/18557 >> 10.0.0.17:6800/8922 pipe(0x7f16fc00cc80 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-04 14:07:16.513857 7f1710440700 0 -- 10.0.0.254:6800/18557 >> 10.0.0.14:6800/9107 pipe(0x7f16fc00ab40 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-04 14:18:39.206912 7f1711e45700 -1 *** Caught signal (Aborted) **
in thread 7f1711e45700

ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
1: /usr/bin/ceph-mds() [0x8578ad]
2: (()+0xf240) [0x7f1715f74240]
3: (gsignal()+0x35) [0x7f17146802f5]
4: (abort()+0x148) [0x7f1714681778]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f1714f6798d]
6: (()+0x63a86) [0x7f1714f65a86]
7: (()+0x63ab3) [0x7f1714f65ab3]
8: (()+0x63cde) [0x7f1714f65cde]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7bfa27]
10: (void decode<inodeno_t, inodeno_t>(std::map<inodeno_t, inodeno_t, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xe1) [0x5429a1]
11: (interval_set<inodeno_t>::decode(ceph::buffer::list::iterator&)+0x12) [0x5429d2]
12: (session_info_t::decode(ceph::buffer::list::iterator&)+0x27e) [0x7e71be]
13: (SessionMap::decode(ceph::buffer::list::iterator&)+0xdf8) [0x6e46e8]
14: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x6e) [0x6e493e]
15: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe2b) [0x70b21b]
16: (MDS::handle_core_message(Message*)+0xae7) [0x504377]
17: (MDS::_dispatch(Message*)+0x33) [0x504473]
18: (MDS::ms_dispatch(Message*)+0x93) [0x506223]
19: (DispatchQueue::entry()+0x32b) [0x82a5fb]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ae40d]
21: (()+0x7e5f) [0x7f1715f6ce5f]
22: (clone()+0x6d) [0x7f171473360d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-500> 2013-04-04 14:07:13.259445 7f171638d780 5 asok(0x1dc3e60) register_command perfcounters_dump hook 0x1dc5240
-499> 2013-04-04 14:07:13.259462 7f171638d780 5 asok(0x1dc3e60) register_command 1 hook 0x1dc5240
-498> 2013-04-04 14:07:13.259467 7f171638d780 5 asok(0x1dc3e60) register_command perf dump hook 0x1dc5240
-497> 2013-04-04 14:07:13.259477 7f171638d780 5 asok(0x1dc3e60) register_command perfcounters_schema hook 0x1dc5240
-496> 2013-04-04 14:07:13.259481 7f171638d780 5 asok(0x1dc3e60) register_command 2 hook 0x1dc5240
-495> 2013-04-04 14:07:13.259486 7f171638d780 5 asok(0x1dc3e60) register_command perf schema hook 0x1dc5240
-494> 2013-04-04 14:07:13.259489 7f171638d780 5 asok(0x1dc3e60) register_command config show hook 0x1dc5240
-493> 2013-04-04 14:07:13.259494 7f171638d780 5 asok(0x1dc3e60) register_command config set hook 0x1dc5240
-492> 2013-04-04 14:07:13.259498 7f171638d780 5 asok(0x1dc3e60) register_command log flush hook 0x1dc5240
-491> 2013-04-04 14:07:13.259500 7f171638d780 5 asok(0x1dc3e60) register_command log dump hook 0x1dc5240
-490> 2013-04-04 14:07:13.259503 7f171638d780 5 asok(0x1dc3e60) register_command log reopen hook 0x1dc5240
-489> 2013-04-04 14:07:13.261104 7f171638d780 0 ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c), process ceph-mds, pid 18557
-488> 2013-04-04 14:07:13.261888 7f171638d780 1 finished global_init_daemonize
-487> 2013-04-04 14:07:13.262049 7f171638d780 5 asok(0x1dc3e60) init /var/run/ceph/ceph-mds.alpha.asok
-486> 2013-04-04 14:07:13.262068 7f171638d780 5 asok(0x1dc3e60) bind_and_listen /var/run/ceph/ceph-mds.alpha.asok
-485> 2013-04-04 14:07:13.262089 7f171638d780 5 asok(0x1dc3e60) register_command 0 hook 0x1dc7c00
-484> 2013-04-04 14:07:13.262096 7f171638d780 5 asok(0x1dc3e60) register_command version hook 0x1dc7c00
-483> 2013-04-04 14:07:13.262100 7f171638d780 5 asok(0x1dc3e60) register_command git_version hook 0x1dc7c00
-482> 2013-04-04 14:07:13.262104 7f171638d780 5 asok(0x1dc3e60) register_command help hook 0x1dc8530
-481> 2013-04-04 14:07:13.262136 7f171638d780 10 monclient(hunting): build_initial_monmap
-480> 2013-04-04 14:07:13.262159 7f1713648700 5 asok(0x1dc3e60) entry start
-479> 2013-04-04 14:07:13.262301 7f171638d780 5 adding auth protocol: none
-478> 2013-04-04 14:07:13.262311 7f171638d780 5 adding auth protocol: none
-477> 2013-04-04 14:07:13.262383 7f171638d780 10 monclient(hunting): init
-476> 2013-04-04 14:07:13.262395 7f171638d780 5 adding auth protocol: none
-475> 2013-04-04 14:07:13.262398 7f171638d780 10 monclient(hunting): auth_supported 1 method none
-474> 2013-04-04 14:07:13.262445 7f171638d780 10 monclient(hunting): _reopen_session
-473> 2013-04-04 14:07:13.262481 7f171638d780 10 monclient(hunting): _pick_new_mon picked mon.alpha con 0x1dd25b0 addr 10.0.0.254:6789/0
-472> 2013-04-04 14:07:13.262509 7f171638d780 10 monclient(hunting): _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-471> 2013-04-04 14:07:13.262517 7f171638d780 10 monclient(hunting): renew_subs
-470> 2013-04-04 14:07:13.262918 7f1711e45700 5 mds.-1.0 ms_handle_connect on 10.0.0.254:6789/0
-469> 2013-04-04 14:07:13.311432 7f1711e45700 10 monclient(hunting): handle_monmap mon_map v1
-468> 2013-04-04 14:07:13.311458 7f1711e45700 10 monclient(hunting): got monmap 1, mon.alpha is now rank 0
-467> 2013-04-04 14:07:13.311463 7f1711e45700 10 monclient(hunting): dump:
epoch 1
fsid 34b90971-4164-40bf-8c70-6bc608e04b01
last_changed 2013-01-28 16:30:55.068102
created 2013-01-28 16:30:55.068102
0: 10.0.0.254:6789/0 mon.alpha

-466> 2013-04-04 14:07:13.311585 7f1711e45700 1 monclient(hunting): found mon.alpha
-465> 2013-04-04 14:07:13.311601 7f1711e45700 10 monclient: my global_id is 7197
-464> 2013-04-04 14:07:13.311606 7f1711e45700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-463> 2013-04-04 14:07:13.311632 7f1711e45700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:06:43.311629)
-462> 2013-04-04 14:07:13.311683 7f171638d780 5 monclient: authenticate success, global_id 7197
-461> 2013-04-04 14:07:13.311711 7f171638d780 5 asok(0x1dc3e60) register_command objecter_requests hook 0x1dd2dc0
-460> 2013-04-04 14:07:13.311764 7f171638d780 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-459> 2013-04-04 14:07:13.311778 7f1711e45700 10 monclient: handle_monmap mon_map v1
-458> 2013-04-04 14:07:13.311791 7f1711e45700 10 monclient: got monmap 1, mon.alpha is now rank 0
-457> 2013-04-04 14:07:13.311796 7f1711e45700 10 monclient: dump:
epoch 1
fsid 34b90971-4164-40bf-8c70-6bc608e04b01
last_changed 2013-01-28 16:30:55.068102
created 2013-01-28 16:30:55.068102
0: 10.0.0.254:6789/0 mon.alpha

-456> 2013-04-04 14:07:13.311828 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:07:13.262526 renew after 2013-04-04 14:09:43.262526
-455> 2013-04-04 14:07:13.311874 7f171638d780 10 monclient: renew_subs
-454> 2013-04-04 14:07:13.311884 7f171638d780 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-453> 2013-04-04 14:07:13.311900 7f171638d780 10 monclient: renew_subs
-452> 2013-04-04 14:07:13.311907 7f171638d780 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-451> 2013-04-04 14:07:13.508759 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:07:13.311882 renew after 2013-04-04 14:09:43.311882
-450> 2013-04-04 14:07:13.508785 7f1711e45700 5 mds.-1.0 handle_mds_map epoch 1093 from mon.0
-449> 2013-04-04 14:07:13.508815 7f1711e45700 1 mds.0.27 handle_mds_map i am now mds.0.27
-448> 2013-04-04 14:07:13.508817 7f1711e45700 1 mds.0.27 handle_mds_map state change up:boot --> up:replay
-447> 2013-04-04 14:07:13.508821 7f1711e45700 1 mds.0.27 replay_start
-446> 2013-04-04 14:07:13.508825 7f1711e45700 1 mds.0.27 recovery set is
-445> 2013-04-04 14:07:13.508826 7f1711e45700 1 mds.0.27 need osdmap epoch 496, have 496
-444> 2013-04-04 14:07:13.508830 7f1711e45700 2 mds.0.27 boot_start 1: opening inotable
-443> 2013-04-04 14:07:13.508966 7f1711e45700 2 mds.0.27 boot_start 1: opening sessionmap
-442> 2013-04-04 14:07:13.509007 7f1711e45700 2 mds.0.27 boot_start 1: opening anchor table
-441> 2013-04-04 14:07:13.509023 7f1711e45700 10 monclient: renew_subs
-440> 2013-04-04 14:07:13.509028 7f1711e45700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-439> 2013-04-04 14:07:13.509038 7f1711e45700 2 mds.0.27 boot_start 1: opening snap table
-438> 2013-04-04 14:07:13.509056 7f1711e45700 2 mds.0.27 boot_start 1: opening mds log
-437> 2013-04-04 14:07:13.509064 7f1711e45700 5 mds.0.log open discovering log bounds
-436> 2013-04-04 14:07:13.509071 7f1711e45700 1 mds.0.journaler(ro) recover start
-435> 2013-04-04 14:07:13.509076 7f1711e45700 1 mds.0.journaler(ro) read_head
-434> 2013-04-04 14:07:13.509148 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:07:13.509026 renew after 2013-04-04 14:09:43.509026
-433> 2013-04-04 14:07:13.509191 7f1711e45700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
-432> 2013-04-04 14:07:16.262571 7f1710e43700 10 monclient: tick
-431> 2013-04-04 14:07:16.262603 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:06:46.262602)
-430> 2013-04-04 14:07:16.262688 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:07:16.262688; renew after: 2013-04-04 14:09:43.509026) -- no
-429> 2013-04-04 14:07:16.513798 7f1710541700 0 -- 10.0.0.254:6800/18557 >> 10.0.0.16:6800/9079 pipe(0x7f16fc009d10 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
-428> 2013-04-04 14:07:16.513820 7f171033f700 0 -- 10.0.0.254:6800/18557 >> 10.0.0.17:6800/8922 pipe(0x7f16fc00cc80 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault
-427> 2013-04-04 14:07:16.513857 7f1710440700 0 -- 10.0.0.254:6800/18557 >> 10.0.0.14:6800/9107 pipe(0x7f16fc00ab40 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
-426> 2013-04-04 14:07:17.311913 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-425> 2013-04-04 14:07:18.311934 7f170b7fe700 10 monclient: renew_subs
-424> 2013-04-04 14:07:18.311952 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-423> 2013-04-04 14:07:18.312465 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:07:18.311949 renew after 2013-04-04 14:09:48.311949
-422> 2013-04-04 14:07:21.312098 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-421> 2013-04-04 14:07:25.312265 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-420> 2013-04-04 14:07:26.262843 7f1710e43700 10 monclient: tick
-419> 2013-04-04 14:07:26.262857 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:06:56.262855)
-418> 2013-04-04 14:07:26.262878 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:07:26.262878; renew after: 2013-04-04 14:09:48.311949) -- no
-417> 2013-04-04 14:07:29.312429 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-416> 2013-04-04 14:07:33.312591 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-415> 2013-04-04 14:07:36.263043 7f1710e43700 10 monclient: tick
-414> 2013-04-04 14:07:36.263057 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:07:06.263056)
-413> 2013-04-04 14:07:36.263075 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:07:36.263075; renew after: 2013-04-04 14:09:48.311949) -- no
-412> 2013-04-04 14:07:37.312776 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-411> 2013-04-04 14:07:41.312946 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-410> 2013-04-04 14:07:45.313080 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-409> 2013-04-04 14:07:46.263230 7f1710e43700 10 monclient: tick
-408> 2013-04-04 14:07:46.263244 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:07:16.263243)
-407> 2013-04-04 14:07:46.263262 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:07:46.263261; renew after: 2013-04-04 14:09:48.311949) -- no
-406> 2013-04-04 14:07:49.313251 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-405> 2013-04-04 14:07:53.313345 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-404> 2013-04-04 14:07:56.263419 7f1710e43700 10 monclient: tick
-403> 2013-04-04 14:07:56.263433 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:07:26.263431)
-402> 2013-04-04 14:07:56.263451 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:07:56.263451; renew after: 2013-04-04 14:09:48.311949) -- no
-401> 2013-04-04 14:07:57.313519 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-400> 2013-04-04 14:08:01.313681 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-399> 2013-04-04 14:08:05.313846 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-398> 2013-04-04 14:08:06.263605 7f1710e43700 10 monclient: tick
-397> 2013-04-04 14:08:06.263675 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:07:36.263674)
-396> 2013-04-04 14:08:06.263691 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:08:06.263691; renew after: 2013-04-04 14:09:48.311949) -- no
-395> 2013-04-04 14:08:09.314014 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-394> 2013-04-04 14:08:13.314101 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-393> 2013-04-04 14:08:16.263843 7f1710e43700 10 monclient: tick
-392> 2013-04-04 14:08:16.263855 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:07:46.263854)
-391> 2013-04-04 14:08:16.263877 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:08:16.263877; renew after: 2013-04-04 14:09:48.311949) -- no
-390> 2013-04-04 14:08:17.314226 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-389> 2013-04-04 14:08:21.314394 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-388> 2013-04-04 14:08:25.314564 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-387> 2013-04-04 14:08:26.264034 7f1710e43700 10 monclient: tick
-386> 2013-04-04 14:08:26.264048 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:07:56.264046)
-385> 2013-04-04 14:08:26.264066 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:08:26.264066; renew after: 2013-04-04 14:09:48.311949) -- no
-384> 2013-04-04 14:08:29.314715 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-383> 2013-04-04 14:08:33.314809 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-382> 2013-04-04 14:08:36.264205 7f1710e43700 10 monclient: tick
-381> 2013-04-04 14:08:36.264218 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:08:06.264217)
-380> 2013-04-04 14:08:36.264237 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:08:36.264237; renew after: 2013-04-04 14:09:48.311949) -- no
-379> 2013-04-04 14:08:37.314976 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-378> 2013-04-04 14:08:41.315117 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-377> 2013-04-04 14:08:45.315291 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-376> 2013-04-04 14:08:46.264393 7f1710e43700 10 monclient: tick
-375> 2013-04-04 14:08:46.264406 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:08:16.264405)
-374> 2013-04-04 14:08:46.264424 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:08:46.264424; renew after: 2013-04-04 14:09:48.311949) -- no
-373> 2013-04-04 14:08:49.315466 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-372> 2013-04-04 14:08:53.315557 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-371> 2013-04-04 14:08:56.264682 7f1710e43700 10 monclient: tick
-370> 2013-04-04 14:08:56.264695 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:08:26.264694)
-369> 2013-04-04 14:08:56.264715 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:08:56.264714; renew after: 2013-04-04 14:09:48.311949) -- no
-368> 2013-04-04 14:08:57.315709 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-367> 2013-04-04 14:09:01.315841 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-366> 2013-04-04 14:09:05.316021 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-365> 2013-04-04 14:09:06.264867 7f1710e43700 10 monclient: tick
-364> 2013-04-04 14:09:06.264881 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:08:36.264880)
-363> 2013-04-04 14:09:06.264899 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:09:06.264898; renew after: 2013-04-04 14:09:48.311949) -- no
-362> 2013-04-04 14:09:09.316159 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-361> 2013-04-04 14:09:13.316290 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-360> 2013-04-04 14:09:16.265051 7f1710e43700 10 monclient: tick
-359> 2013-04-04 14:09:16.265063 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:08:46.265062)
-358> 2013-04-04 14:09:16.265080 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:09:16.265080; renew after: 2013-04-04 14:09:48.311949) -- no
-357> 2013-04-04 14:09:17.316419 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-356> 2013-04-04 14:09:21.316608 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-355> 2013-04-04 14:09:25.316836 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-354> 2013-04-04 14:09:26.265233 7f1710e43700 10 monclient: tick
-353> 2013-04-04 14:09:26.265245 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:08:56.265244)
-352> 2013-04-04 14:09:26.265261 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:09:26.265261; renew after: 2013-04-04 14:09:48.311949) -- no
-351> 2013-04-04 14:09:29.317018 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-350> 2013-04-04 14:09:33.317135 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-349> 2013-04-04 14:09:36.265417 7f1710e43700 10 monclient: tick
-348> 2013-04-04 14:09:36.265433 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:09:06.265432)
-347> 2013-04-04 14:09:36.265452 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:09:36.265451; renew after: 2013-04-04 14:09:48.311949) -- no
-346> 2013-04-04 14:09:37.317308 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-345> 2013-04-04 14:09:41.317484 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-344> 2013-04-04 14:09:45.317669 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-343> 2013-04-04 14:09:46.265527 7f1710e43700 10 monclient: tick
-342> 2013-04-04 14:09:46.265538 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:09:16.265537)
-341> 2013-04-04 14:09:46.265554 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:09:46.265554; renew after: 2013-04-04 14:09:48.311949) -- no
-340> 2013-04-04 14:09:49.317836 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-339> 2013-04-04 14:09:53.317932 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-338> 2013-04-04 14:09:56.265674 7f1710e43700 10 monclient: tick
-337> 2013-04-04 14:09:56.265697 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:09:26.265696)
-336> 2013-04-04 14:09:56.265715 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:09:56.265715; renew after: 2013-04-04 14:09:48.311949) -- yes
-335> 2013-04-04 14:09:56.265722 7f1710e43700 10 monclient: renew_subs
-334> 2013-04-04 14:09:56.265725 7f1710e43700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-333> 2013-04-04 14:09:56.266161 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:09:56.265723 renew after 2013-04-04 14:12:26.265723
-332> 2013-04-04 14:09:57.318065 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-331> 2013-04-04 14:10:01.318163 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-330> 2013-04-04 14:10:05.318336 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-329> 2013-04-04 14:10:06.265882 7f1710e43700 10 monclient: tick
-328> 2013-04-04 14:10:06.265895 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:09:36.265894)
-327> 2013-04-04 14:10:06.265914 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:10:06.265914; renew after: 2013-04-04 14:12:26.265723) -- no
-326> 2013-04-04 14:10:09.318507 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-325> 2013-04-04 14:10:13.318633 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-324> 2013-04-04 14:10:16.266047 7f1710e43700 10 monclient: tick
-323> 2013-04-04 14:10:16.266061 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:09:46.266059)
-322> 2013-04-04 14:10:16.266076 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:10:16.266076; renew after: 2013-04-04 14:12:26.265723) -- no
-321> 2013-04-04 14:10:17.318783 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-320> 2013-04-04 14:10:21.318928 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-319> 2013-04-04 14:10:25.319091 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-318> 2013-04-04 14:10:26.266201 7f1710e43700 10 monclient: tick
-317> 2013-04-04 14:10:26.266213 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:09:56.266212)
-316> 2013-04-04 14:10:26.266234 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:10:26.266234; renew after: 2013-04-04 14:12:26.265723) -- no
-315> 2013-04-04 14:10:29.319241 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-314> 2013-04-04 14:10:33.319371 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-313> 2013-04-04 14:10:36.266482 7f1710e43700 10 monclient: tick
-312> 2013-04-04 14:10:36.266500 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:10:06.266498)
-311> 2013-04-04 14:10:36.266517 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:10:36.266516; renew after: 2013-04-04 14:12:26.265723) -- no
-310> 2013-04-04 14:10:37.319516 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-309> 2013-04-04 14:10:41.319685 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-308> 2013-04-04 14:10:45.319848 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-307> 2013-04-04 14:10:46.266624 7f1710e43700 10 monclient: tick
-306> 2013-04-04 14:10:46.266639 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:10:16.266638)
-305> 2013-04-04 14:10:46.266659 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:10:46.266659; renew after: 2013-04-04 14:12:26.265723) -- no
-304> 2013-04-04 14:10:49.320011 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-303> 2013-04-04 14:10:53.320150 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-302> 2013-04-04 14:10:56.266813 7f1710e43700 10 monclient: tick
-301> 2013-04-04 14:10:56.266836 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:10:26.266834)
-300> 2013-04-04 14:10:56.266857 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:10:56.266857; renew after: 2013-04-04 14:12:26.265723) -- no
-299> 2013-04-04 14:10:57.320307 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-298> 2013-04-04 14:11:01.320408 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-297> 2013-04-04 14:11:05.320588 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-296> 2013-04-04 14:11:06.267010 7f1710e43700 10 monclient: tick
-295> 2013-04-04 14:11:06.267023 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:10:36.267022)
-294> 2013-04-04 14:11:06.267042 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:11:06.267042; renew after: 2013-04-04 14:12:26.265723) -- no
-293> 2013-04-04 14:11:09.320871 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-292> 2013-04-04 14:11:13.321009 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-291> 2013-04-04 14:11:16.267200 7f1710e43700 10 monclient: tick
-290> 2013-04-04 14:11:16.267219 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:10:46.267217)
-289> 2013-04-04 14:11:16.267246 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:11:16.267245; renew after: 2013-04-04 14:12:26.265723) -- no
-288> 2013-04-04 14:11:17.321151 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-287> 2013-04-04 14:11:21.321353 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-286> 2013-04-04 14:11:25.321536 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-285> 2013-04-04 14:11:26.267404 7f1710e43700 10 monclient: tick
-284> 2013-04-04 14:11:26.267417 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:10:56.267416)
-283> 2013-04-04 14:11:26.267435 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:11:26.267435; renew after: 2013-04-04 14:12:26.265723) -- no
-282> 2013-04-04 14:11:29.321712 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-281> 2013-04-04 14:11:33.321868 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-280> 2013-04-04 14:11:36.267655 7f1710e43700 10 monclient: tick
-279> 2013-04-04 14:11:36.267672 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:11:06.267671)
-278> 2013-04-04 14:11:36.267691 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:11:36.267691; renew after: 2013-04-04 14:12:26.265723) -- no
-277> 2013-04-04 14:11:37.322042 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-276> 2013-04-04 14:11:41.322216 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-275> 2013-04-04 14:11:45.322393 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-274> 2013-04-04 14:11:46.267851 7f1710e43700 10 monclient: tick
-273> 2013-04-04 14:11:46.267864 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:11:16.267863)
-272> 2013-04-04 14:11:46.267882 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:11:46.267882; renew after: 2013-04-04 14:12:26.265723) -- no
-271> 2013-04-04 14:11:49.322544 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-270> 2013-04-04 14:11:53.322677 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-269> 2013-04-04 14:11:56.268039 7f1710e43700 10 monclient: tick
-268> 2013-04-04 14:11:56.268059 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:11:26.268057)
-267> 2013-04-04 14:11:56.268080 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:11:56.268079; renew after: 2013-04-04 14:12:26.265723) -- no
-266> 2013-04-04 14:11:57.322857 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-265> 2013-04-04 14:12:01.323042 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-264> 2013-04-04 14:12:05.323225 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-263> 2013-04-04 14:12:06.268235 7f1710e43700 10 monclient: tick
-262> 2013-04-04 14:12:06.268249 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:11:36.268247)
-261> 2013-04-04 14:12:06.268273 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:12:06.268273; renew after: 2013-04-04 14:12:26.265723) -- no
-260> 2013-04-04 14:12:09.323404 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-259> 2013-04-04 14:12:13.323543 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-258> 2013-04-04 14:12:16.268485 7f1710e43700 10 monclient: tick
-257> 2013-04-04 14:12:16.268500 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:11:46.268499)
-256> 2013-04-04 14:12:16.268518 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:12:16.268518; renew after: 2013-04-04 14:12:26.265723) -- no
-255> 2013-04-04 14:12:17.323715 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-254> 2013-04-04 14:12:21.323888 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-253> 2013-04-04 14:12:25.324066 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-252> 2013-04-04 14:12:26.268675 7f1710e43700 10 monclient: tick
-251> 2013-04-04 14:12:26.268688 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:11:56.268687)
-250> 2013-04-04 14:12:26.268707 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:12:26.268707; renew after: 2013-04-04 14:12:26.265723) -- yes
-249> 2013-04-04 14:12:26.268717 7f1710e43700 10 monclient: renew_subs
-248> 2013-04-04 14:12:26.268721 7f1710e43700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-247> 2013-04-04 14:12:26.269167 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:12:26.268718 renew after 2013-04-04 14:14:56.268718
-246> 2013-04-04 14:12:29.324243 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-245> 2013-04-04 14:12:33.324384 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-244> 2013-04-04 14:12:36.268857 7f1710e43700 10 monclient: tick
-243> 2013-04-04 14:12:36.268874 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:12:06.268872)
-242> 2013-04-04 14:12:36.268900 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:12:36.268899; renew after: 2013-04-04 14:14:56.268718) -- no
-241> 2013-04-04 14:12:37.324518 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-240> 2013-04-04 14:12:41.324691 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-239> 2013-04-04 14:12:45.324862 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-238> 2013-04-04 14:12:46.269048 7f1710e43700 10 monclient: tick
-237> 2013-04-04 14:12:46.269060 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:12:16.269059)
-236> 2013-04-04 14:12:46.269077 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:12:46.269077; renew after: 2013-04-04 14:14:56.268718) -- no
-235> 2013-04-04 14:12:49.325030 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-234> 2013-04-04 14:12:53.325163 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-233> 2013-04-04 14:12:56.269241 7f1710e43700 10 monclient: tick
-232> 2013-04-04 14:12:56.269254 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:12:26.269253)
-231> 2013-04-04 14:12:56.269273 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:12:56.269272; renew after: 2013-04-04 14:14:56.268718) -- no
-230> 2013-04-04 14:12:57.325337 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-229> 2013-04-04 14:13:01.325572 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-228> 2013-04-04 14:13:05.325747 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-227> 2013-04-04 14:13:06.269428 7f1710e43700 10 monclient: tick
-226> 2013-04-04 14:13:06.269442 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:12:36.269441)
-225> 2013-04-04 14:13:06.269462 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:13:06.269462; renew after: 2013-04-04 14:14:56.268718) -- no
-224> 2013-04-04 14:13:09.325920 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-223> 2013-04-04 14:13:13.326059 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-222> 2013-04-04 14:13:16.269622 7f1710e43700 10 monclient: tick
-221> 2013-04-04 14:13:16.269636 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:12:46.269635)
-220> 2013-04-04 14:13:16.269655 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:13:16.269654; renew after: 2013-04-04 14:14:56.268718) -- no
-219> 2013-04-04 14:13:17.326227 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-218> 2013-04-04 14:13:21.326392 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-217> 2013-04-04 14:13:25.326673 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-216> 2013-04-04 14:13:26.269826 7f1710e43700 10 monclient: tick
-215> 2013-04-04 14:13:26.269839 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:12:56.269838)
-214> 2013-04-04 14:13:26.269863 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:13:26.269862; renew after: 2013-04-04 14:14:56.268718) -- no
-213> 2013-04-04 14:13:29.326848 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-212> 2013-04-04 14:13:33.326978 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-211> 2013-04-04 14:13:36.270021 7f1710e43700 10 monclient: tick
-210> 2013-04-04 14:13:36.270037 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:13:06.270036)
-209> 2013-04-04 14:13:36.270056 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:13:36.270056; renew after: 2013-04-04 14:14:56.268718) -- no
-208> 2013-04-04 14:13:37.327150 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-207> 2013-04-04 14:13:41.327331 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-206> 2013-04-04 14:13:45.327516 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-205> 2013-04-04 14:13:46.270212 7f1710e43700 10 monclient: tick
-204> 2013-04-04 14:13:46.270225 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:13:16.270224)
-203> 2013-04-04 14:13:46.270243 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:13:46.270243; renew after: 2013-04-04 14:14:56.268718) -- no
-202> 2013-04-04 14:13:49.327694 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-201> 2013-04-04 14:13:53.327826 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-200> 2013-04-04 14:13:56.270358 7f1710e43700 10 monclient: tick
-199> 2013-04-04 14:13:56.270371 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:13:26.270370)
-198> 2013-04-04 14:13:56.270389 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:13:56.270389; renew after: 2013-04-04 14:14:56.268718) -- no
-197> 2013-04-04 14:13:57.327994 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-196> 2013-04-04 14:14:01.328165 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-195> 2013-04-04 14:14:05.328337 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-194> 2013-04-04 14:14:06.270545 7f1710e43700 10 monclient: tick
-193> 2013-04-04 14:14:06.270558 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:13:36.270556)
-192> 2013-04-04 14:14:06.270575 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:14:06.270575; renew after: 2013-04-04 14:14:56.268718) -- no
-191> 2013-04-04 14:14:09.328513 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-190> 2013-04-04 14:14:13.328694 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-189> 2013-04-04 14:14:16.270732 7f1710e43700 10 monclient: tick
-188> 2013-04-04 14:14:16.270745 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:13:46.270744)
-187> 2013-04-04 14:14:16.270769 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:14:16.270769; renew after: 2013-04-04 14:14:56.268718) -- no
-186> 2013-04-04 14:14:17.328868 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-185> 2013-04-04 14:14:21.329043 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-184> 2013-04-04 14:14:25.329220 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-183> 2013-04-04 14:14:26.270903 7f1710e43700 10 monclient: tick
-182> 2013-04-04 14:14:26.270919 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:13:56.270918)
-181> 2013-04-04 14:14:26.270937 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:14:26.270937; renew after: 2013-04-04 14:14:56.268718) -- no
-180> 2013-04-04 14:14:29.329397 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-179> 2013-04-04 14:14:33.329557 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-178> 2013-04-04 14:14:36.271094 7f1710e43700 10 monclient: tick
-177> 2013-04-04 14:14:36.271109 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:14:06.271108)
-176> 2013-04-04 14:14:36.271126 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:14:36.271126; renew after: 2013-04-04 14:14:56.268718) -- no
-175> 2013-04-04 14:14:37.329729 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-174> 2013-04-04 14:14:41.329899 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-173> 2013-04-04 14:14:45.330074 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-172> 2013-04-04 14:14:46.271282 7f1710e43700 10 monclient: tick
-171> 2013-04-04 14:14:46.271296 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:14:16.271295)
-170> 2013-04-04 14:14:46.271314 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:14:46.271314; renew after: 2013-04-04 14:14:56.268718) -- no
-169> 2013-04-04 14:14:49.330250 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-168> 2013-04-04 14:14:53.330346 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-167> 2013-04-04 14:14:56.271467 7f1710e43700 10 monclient: tick
-166> 2013-04-04 14:14:56.271482 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:14:26.271481)
-165> 2013-04-04 14:14:56.271500 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:14:56.271500; renew after: 2013-04-04 14:14:56.268718) -- yes
-164> 2013-04-04 14:14:56.271511 7f1710e43700 10 monclient: renew_subs
-163> 2013-04-04 14:14:56.271520 7f1710e43700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-162> 2013-04-04 14:14:56.272016 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:14:56.271518 renew after 2013-04-04 14:17:26.271518
-161> 2013-04-04 14:14:57.330518 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-160> 2013-04-04 14:15:01.330675 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-159> 2013-04-04 14:15:05.330847 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-158> 2013-04-04 14:15:06.271673 7f1710e43700 10 monclient: tick
-157> 2013-04-04 14:15:06.271687 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:14:36.271686)
-156> 2013-04-04 14:15:06.271705 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:15:06.271705; renew after: 2013-04-04 14:17:26.271518) -- no
-155> 2013-04-04 14:15:09.331022 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-154> 2013-04-04 14:15:13.331191 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-153> 2013-04-04 14:15:16.271854 7f1710e43700 10 monclient: tick
-152> 2013-04-04 14:15:16.271867 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:14:46.271866)
-151> 2013-04-04 14:15:16.271887 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:15:16.271887; renew after: 2013-04-04 14:17:26.271518) -- no
-150> 2013-04-04 14:15:17.331289 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-149> 2013-04-04 14:15:21.331469 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-148> 2013-04-04 14:15:25.331647 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-147> 2013-04-04 14:15:26.272108 7f1710e43700 10 monclient: tick
-146> 2013-04-04 14:15:26.272124 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:14:56.272123)
-145> 2013-04-04 14:15:26.272142 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:15:26.272142; renew after: 2013-04-04 14:17:26.271518) -- no
-144> 2013-04-04 14:15:29.331822 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-143> 2013-04-04 14:15:33.331992 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-142> 2013-04-04 14:15:36.272297 7f1710e43700 10 monclient: tick
-141> 2013-04-04 14:15:36.272311 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:15:06.272310)
-140> 2013-04-04 14:15:36.272329 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:15:36.272328; renew after: 2013-04-04 14:17:26.271518) -- no
-139> 2013-04-04 14:15:37.332162 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-138> 2013-04-04 14:15:41.332377 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-137> 2013-04-04 14:15:45.332558 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-136> 2013-04-04 14:15:46.272483 7f1710e43700 10 monclient: tick
-135> 2013-04-04 14:15:46.272496 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:15:16.272495)
-134> 2013-04-04 14:15:46.272521 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:15:46.272521; renew after: 2013-04-04 14:17:26.271518) -- no
-133> 2013-04-04 14:15:49.332731 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-132> 2013-04-04 14:15:53.332828 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-131> 2013-04-04 14:15:56.272669 7f1710e43700 10 monclient: tick
-130> 2013-04-04 14:15:56.272683 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:15:26.272682)
-129> 2013-04-04 14:15:56.272702 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:15:56.272702; renew after: 2013-04-04 14:17:26.271518) -- no
-128> 2013-04-04 14:15:57.333000 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-127> 2013-04-04 14:16:01.333167 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-126> 2013-04-04 14:16:05.333325 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-125> 2013-04-04 14:16:06.272855 7f1710e43700 10 monclient: tick
-124> 2013-04-04 14:16:06.272868 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:15:36.272867)
-123> 2013-04-04 14:16:06.272886 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:16:06.272885; renew after: 2013-04-04 14:17:26.271518) -- no
-122> 2013-04-04 14:16:09.333499 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-121> 2013-04-04 14:16:13.333636 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-120> 2013-04-04 14:16:16.273043 7f1710e43700 10 monclient: tick
-119> 2013-04-04 14:16:16.273057 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:15:46.273056)
-118> 2013-04-04 14:16:16.273075 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:16:16.273075; renew after: 2013-04-04 14:17:26.271518) -- no
-117> 2013-04-04 14:16:17.333808 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-116> 2013-04-04 14:16:21.333986 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-115> 2013-04-04 14:16:25.334165 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-114> 2013-04-04 14:16:26.273228 7f1710e43700 10 monclient: tick
-113> 2013-04-04 14:16:26.273241 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:15:56.273240)
-112> 2013-04-04 14:16:26.273264 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:16:26.273264; renew after: 2013-04-04 14:17:26.271518) -- no
-111> 2013-04-04 14:16:29.334338 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-110> 2013-04-04 14:16:33.334503 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-109> 2013-04-04 14:16:36.273424 7f1710e43700 10 monclient: tick
-108> 2013-04-04 14:16:36.273440 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:16:06.273438)
-107> 2013-04-04 14:16:36.273459 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:16:36.273459; renew after: 2013-04-04 14:17:26.271518) -- no
-106> 2013-04-04 14:16:37.334676 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-105> 2013-04-04 14:16:41.334850 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-104> 2013-04-04 14:16:45.335025 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-103> 2013-04-04 14:16:46.273679 7f1710e43700 10 monclient: tick
-102> 2013-04-04 14:16:46.273694 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:16:16.273693)
-101> 2013-04-04 14:16:46.273712 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:16:46.273712; renew after: 2013-04-04 14:17:26.271518) -- no
-100> 2013-04-04 14:16:49.335198 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-99> 2013-04-04 14:16:53.335327 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-98> 2013-04-04 14:16:56.273854 7f1710e43700 10 monclient: tick
-97> 2013-04-04 14:16:56.273872 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:16:26.273871)
-96> 2013-04-04 14:16:56.273893 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:16:56.273893; renew after: 2013-04-04 14:17:26.271518) -- no
-95> 2013-04-04 14:16:57.335503 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-94> 2013-04-04 14:16:58.330132 7f170b7fe700 10 monclient: renew_subs
-93> 2013-04-04 14:16:58.330148 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-92> 2013-04-04 14:16:58.330439 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:16:58.330146 renew after 2013-04-04 14:19:28.330146
-91> 2013-04-04 14:17:01.335675 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-90> 2013-04-04 14:17:03.330284 7f170b7fe700 10 monclient: renew_subs
-89> 2013-04-04 14:17:03.330299 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-88> 2013-04-04 14:17:03.330555 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:17:03.330297 renew after 2013-04-04 14:19:33.330297
-87> 2013-04-04 14:17:05.335862 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-86> 2013-04-04 14:17:06.274101 7f1710e43700 10 monclient: tick
-85> 2013-04-04 14:17:06.274116 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:16:36.274115)
-84> 2013-04-04 14:17:06.274135 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:17:06.274135; renew after: 2013-04-04 14:19:33.330297) -- no
-83> 2013-04-04 14:17:09.336040 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-82> 2013-04-04 14:17:13.336215 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-81> 2013-04-04 14:17:16.274293 7f1710e43700 10 monclient: tick
-80> 2013-04-04 14:17:16.274312 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:16:46.274310)
-79> 2013-04-04 14:17:16.274337 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:17:16.274337; renew after: 2013-04-04 14:19:33.330297) -- no
-78> 2013-04-04 14:17:17.336444 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-77> 2013-04-04 14:17:21.336679 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-76> 2013-04-04 14:17:25.336855 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-75> 2013-04-04 14:17:26.274500 7f1710e43700 10 monclient: tick
-74> 2013-04-04 14:17:26.274513 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:16:56.274512)
-73> 2013-04-04 14:17:26.274531 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:17:26.274531; renew after: 2013-04-04 14:19:33.330297) -- no
-72> 2013-04-04 14:17:29.337036 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-71> 2013-04-04 14:17:33.337169 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-70> 2013-04-04 14:17:36.274692 7f1710e43700 10 monclient: tick
-69> 2013-04-04 14:17:36.274708 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:17:06.274707)
-68> 2013-04-04 14:17:36.274728 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:17:36.274728; renew after: 2013-04-04 14:19:33.330297) -- no
-67> 2013-04-04 14:17:37.337342 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-66> 2013-04-04 14:17:41.337520 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-65> 2013-04-04 14:17:45.337715 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-64> 2013-04-04 14:17:46.274849 7f1710e43700 10 monclient: tick
-63> 2013-04-04 14:17:46.274865 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:17:16.274864)
-62> 2013-04-04 14:17:46.274886 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:17:46.274886; renew after: 2013-04-04 14:19:33.330297) -- no
-61> 2013-04-04 14:17:48.331738 7f170b7fe700 10 monclient: renew_subs
-60> 2013-04-04 14:17:48.331755 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-59> 2013-04-04 14:17:48.332438 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:17:48.331752 renew after 2013-04-04 14:20:18.331752
-58> 2013-04-04 14:17:49.337880 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-57> 2013-04-04 14:17:53.331926 7f170b7fe700 10 monclient: renew_subs
-56> 2013-04-04 14:17:53.331947 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-55> 2013-04-04 14:17:53.332409 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:17:53.331944 renew after 2013-04-04 14:20:23.331944
-54> 2013-04-04 14:17:53.338051 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-53> 2013-04-04 14:17:56.275047 7f1710e43700 10 monclient: tick
-52> 2013-04-04 14:17:56.275065 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:17:26.275064)
-51> 2013-04-04 14:17:56.275086 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:17:56.275086; renew after: 2013-04-04 14:20:23.331944) -- no
-50> 2013-04-04 14:17:57.338225 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-49> 2013-04-04 14:18:01.338404 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-48> 2013-04-04 14:18:05.338564 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-47> 2013-04-04 14:18:06.275236 7f1710e43700 10 monclient: tick
-46> 2013-04-04 14:18:06.275254 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:17:36.275253)
-45> 2013-04-04 14:18:06.275281 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:18:06.275281; renew after: 2013-04-04 14:20:23.331944) -- no
-44> 2013-04-04 14:18:08.332407 7f170b7fe700 10 monclient: renew_subs
-43> 2013-04-04 14:18:08.332429 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-42> 2013-04-04 14:18:08.333418 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:18:08.332426 renew after 2013-04-04 14:20:38.332426
-41> 2013-04-04 14:18:08.334292 7f1711e45700 5 mds.0.27 ms_handle_connect on 10.0.0.6:6800/2004
-40> 2013-04-04 14:18:09.338748 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-39> 2013-04-04 14:18:13.332663 7f170b7fe700 10 monclient: renew_subs
-38> 2013-04-04 14:18:13.332682 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-37> 2013-04-04 14:18:13.333517 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:18:13.332679 renew after 2013-04-04 14:20:43.332679
-36> 2013-04-04 14:18:13.334362 7f1711e45700 5 mds.0.27 ms_handle_connect on 10.0.0.11:6800/1865
-35> 2013-04-04 14:18:13.338928 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-34> 2013-04-04 14:18:16.275442 7f1710e43700 10 monclient: tick
-33> 2013-04-04 14:18:16.275459 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:17:46.275458)
-32> 2013-04-04 14:18:16.275480 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:18:16.275480; renew after: 2013-04-04 14:20:43.332679) -- no
-31> 2013-04-04 14:18:17.339086 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-30> 2013-04-04 14:18:18.332852 7f170b7fe700 10 monclient: renew_subs
-29> 2013-04-04 14:18:18.332870 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-28> 2013-04-04 14:18:18.333834 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:18:18.332867 renew after 2013-04-04 14:20:48.332867
-27> 2013-04-04 14:18:18.334631 7f1711e45700 5 mds.0.27 ms_handle_connect on 10.0.0.13:6800/1889
-26> 2013-04-04 14:18:18.334656 7f1711e45700 5 mds.0.27 ms_handle_connect on 10.0.0.9:6800/2047
-25> 2013-04-04 14:18:21.339270 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-24> 2013-04-04 14:18:22.532581 7f1711e45700 1 mds.0.journaler(ro) _finish_read_head loghead(trim 2867159105536, expire 2867159107969, write 2867288211783). probing for end of log (from 2867288211783)...
-23> 2013-04-04 14:18:22.532600 7f1711e45700 1 mds.0.journaler(ro) probing for end of the log
-22> 2013-04-04 14:18:22.533605 7f1711e45700 5 mds.0.27 ms_handle_connect on 10.0.0.7:6800/2026
-21> 2013-04-04 14:18:23.332958 7f170b7fe700 10 monclient: renew_subs
-20> 2013-04-04 14:18:23.332975 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-19> 2013-04-04 14:18:23.333793 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:18:23.332973 renew after 2013-04-04 14:20:53.332973
-18> 2013-04-04 14:18:23.334290 7f1711e45700 5 mds.0.27 ms_handle_connect on 10.0.0.10:6800/2068
-17> 2013-04-04 14:18:23.334415 7f1711e45700 5 mds.0.27 ms_handle_connect on 10.0.0.14:6800/1923
-16> 2013-04-04 14:18:25.339457 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-15> 2013-04-04 14:18:26.275638 7f1710e43700 10 monclient: tick
-14> 2013-04-04 14:18:26.275653 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:17:56.275652)
-13> 2013-04-04 14:18:26.275674 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:18:26.275673; renew after: 2013-04-04 14:20:53.332973) -- no
-12> 2013-04-04 14:18:29.339604 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-11> 2013-04-04 14:18:33.333277 7f170b7fe700 10 monclient: renew_subs
-10> 2013-04-04 14:18:33.333296 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-9> 2013-04-04 14:18:33.334041 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:18:33.333293 renew after 2013-04-04 14:21:03.333293
-8> 2013-04-04 14:18:33.339790 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-7> 2013-04-04 14:18:36.275807 7f1710e43700 10 monclient: tick
-6> 2013-04-04 14:18:36.275830 7f1710e43700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-04-04 14:18:06.275829)
-5> 2013-04-04 14:18:36.275854 7f1710e43700 10 monclient: renew subs? (now: 2013-04-04 14:18:36.275854; renew after: 2013-04-04 14:21:03.333293) -- no
-4> 2013-04-04 14:18:37.339967 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-3> 2013-04-04 14:18:38.333398 7f170b7fe700 10 monclient: renew_subs
-2> 2013-04-04 14:18:38.333418 7f170b7fe700 10 monclient: _send_mon_message to mon.alpha at 10.0.0.254:6789/0
-1> 2013-04-04 14:18:38.733894 7f1711e45700 10 monclient: handle_subscribe_ack sent 2013-04-04 14:18:38.333415 renew after 2013-04-04 14:21:08.333415
0> 2013-04-04 14:18:39.206912 7f1711e45700 -1 *** Caught signal (Aborted) **
in thread 7f1711e45700

ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
1: /usr/bin/ceph-mds() [0x8578ad]
2: (()+0xf240) [0x7f1715f74240]
3: (gsignal()+0x35) [0x7f17146802f5]
4: (abort()+0x148) [0x7f1714681778]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f1714f6798d]
6: (()+0x63a86) [0x7f1714f65a86]
7: (()+0x63ab3) [0x7f1714f65ab3]
8: (()+0x63cde) [0x7f1714f65cde]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7bfa27]
10: (void decode<inodeno_t, inodeno_t>(std::map<inodeno_t, inodeno_t, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xe1) [0x5429a1]
11: (interval_set<inodeno_t>::decode(ceph::buffer::list::iterator&)+0x12) [0x5429d2]
12: (session_info_t::decode(ceph::buffer::list::iterator&)+0x27e) [0x7e71be]
13: (SessionMap::decode(ceph::buffer::list::iterator&)+0xdf8) [0x6e46e8]
14: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x6e) [0x6e493e]
15: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe2b) [0x70b21b]
16: (MDS::handle_core_message(Message*)+0xae7) [0x504377]
17: (MDS::_dispatch(Message*)+0x33) [0x504473]
18: (MDS::ms_dispatch(Message*)+0x93) [0x506223]
19: (DispatchQueue::entry()+0x32b) [0x82a5fb]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ae40d]
21: (()+0x7e5f) [0x7f1715f6ce5f]
22: (clone()+0x6d) [0x7f171473360d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.alpha.log
--- end dump of recent events ---

(1-1/3)