2013-04-25 18:37:53.011757 7fcccb474780 0 ceph version 0.60-641-gc7a0477 (c7a0477bad6bfbec4ef325295ca0489ec1977926), process ceph-mon, pid 19271 2013-04-25 18:37:53.011800 7fcccb474780 10 needs_conversion 2013-04-25 18:37:53.750614 7fcccb474780 10 obtain_monmap 2013-04-25 18:37:53.750770 7fcccb474780 10 obtain_monmap read last committed monmap ver 1 2013-04-25 18:37:53.750917 7fcccb474780 10 -- :/0 rank.bind 10.1.0.3:6789/0 2013-04-25 18:37:53.750930 7fcccb474780 10 accepter.accepter.bind 2013-04-25 18:37:53.750952 7fcccb474780 10 accepter.accepter.bind bound to 10.1.0.3:6789/0 2013-04-25 18:37:53.750965 7fcccb474780 1 -- 10.1.0.3:6789/0 learned my addr 10.1.0.3:6789/0 2013-04-25 18:37:53.750973 7fcccb474780 1 accepter.accepter.bind my_inst.addr is 10.1.0.3:6789/0 need_addr=0 2013-04-25 18:37:53.751035 7fcccb474780 1 mon.a@-1(probing) e1 preinit fsid 2f2730c5-0504-4433-ae0b-331dd41d99a4 2013-04-25 18:37:53.751133 7fcccb474780 10 mon.a@-1(probing) e1 check_fsid cluster_uuid contains '2f2730c5-0504-4433-ae0b-331dd41d99a4' 2013-04-25 18:37:53.751177 7fcccb474780 10 mon.a@-1(probing) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?)} 2013-04-25 18:37:53.751219 7fcccb474780 10 mon.a@-1(probing) e1 has_ever_joined = 1 2013-04-25 18:37:53.751256 7fcccb474780 1 mon.a@-1(probing) e1 preinit clean up potentially inconsistent store state 2013-04-25 18:37:53.934864 7fcccb474780 10 mon.a@-1(probing) e1 init_paxos 2013-04-25 18:37:54.241726 7fcccb474780 10 mon.a@-1(probing).paxos(paxos recovering c 0..0) init last_pn: 0 accepted_pn: 0 last_committed: 0 first_committed: 0 2013-04-25 18:37:54.241783 7fcccb474780 10 mon.a@-1(probing).paxos(paxos recovering c 0..0) init 2013-04-25 18:37:54.315308 7fcccb474780 10 mon.a@-1(probing).log v0 update_from_paxos 2013-04-25 18:37:54.315709 7fcccb474780 10 mon.a@-1(probing).log v0 update_from_paxos version 0 summary v 0 2013-04-25 18:37:54.400076 7fcccb474780 10 mon.a@-1(probing).auth v0 update_from_paxos 2013-04-25 18:37:54.461172 7fcccb474780 10 mon.a@-1(probing).health(0) init 2013-04-25 18:37:54.513893 7fcccb474780 10 mon.a@-1(probing) e1 loading initial keyring to bootstrap authentication for mkfs 2013-04-25 18:37:54.516358 7fcccb474780 1 -- 10.1.0.3:6789/0 messenger.start 2013-04-25 18:37:54.516509 7fcccb474780 2 mon.a@-1(probing) e1 init 2013-04-25 18:37:54.516557 7fcccb474780 10 -- 10.1.0.3:6789/0 ready 10.1.0.3:6789/0 2013-04-25 18:37:54.516782 7fcccb474780 1 accepter.accepter.start 2013-04-25 18:37:54.517243 7fcccb474780 10 mon.a@-1(probing) e1 bootstrap 2013-04-25 18:37:54.517258 7fcccb474780 10 mon.a@-1(probing) e1 unregister_cluster_logger - not registered 2013-04-25 18:37:54.517319 7fcccb474780 10 mon.a@-1(probing) e1 cancel_probe_timeout (none scheduled) 2013-04-25 18:37:54.516647 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper_entry start 2013-04-25 18:37:54.517325 7fcccb474780 0 mon.a@-1(probing) e1 my rank is now 0 (was -1) 2013-04-25 18:37:54.517328 7fcccb474780 1 -- 10.1.0.3:6789/0 mark_down_all 2013-04-25 18:37:54.517392 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper 2013-04-25 18:37:54.517325 7fccbfcd4700 10 accepter.accepter starting 2013-04-25 18:37:54.517407 7fccbfcd4700 20 accepter.accepter calling poll 2013-04-25 18:37:54.517405 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper done 2013-04-25 18:37:54.517467 7fcccb474780 10 mon.a@0(probing) e1 reset_sync 2013-04-25 18:37:54.517557 7fcccb474780 10 mon.a@0(probing) e1 reset 2013-04-25 18:37:54.517561 7fcccb474780 10 mon.a@0(probing) e1 timecheck_finish 2013-04-25 18:37:54.517567 7fcccb474780 10 mon.a@0(probing).paxos(paxos recovering c 0..0) restart -- canceling timeouts 2013-04-25 18:37:54.517587 7fcccb474780 10 mon.a@0(probing).paxos(paxos recovering c 0..0) restart -- clearing queued proposals 2013-04-25 18:37:54.517596 7fcccb474780 10 mon.a@0(probing).paxosservice(pgmap) restart 2013-04-25 18:37:54.517605 7fcccb474780 10 mon.a@0(probing).paxosservice(mdsmap) restart 2013-04-25 18:37:54.517610 7fcccb474780 10 mon.a@0(probing).paxosservice(osdmap) restart 2013-04-25 18:37:54.517612 7fcccb474780 10 mon.a@0(probing).paxosservice(logm) restart 2013-04-25 18:37:54.517614 7fcccb474780 10 mon.a@0(probing).paxosservice(monmap) restart 2013-04-25 18:37:54.517617 7fcccb474780 10 mon.a@0(probing).paxosservice(auth) restart 2013-04-25 18:37:54.517625 7fcccb474780 10 mon.a@0(probing) e1 cancel_probe_timeout (none scheduled) 2013-04-25 18:37:54.517628 7fcccb474780 10 mon.a@0(probing) e1 reset_probe_timeout 0x23d9340 after 2 seconds 2013-04-25 18:37:54.517691 7fcccb474780 10 mon.a@0(probing) e1 probing other monitors 2013-04-25 18:37:54.517699 7fcccb474780 1 -- 10.1.0.3:6789/0 --> mon.1 10.1.0.67:6789/0 -- mon_probe(probe 2f2730c5-0504-4433-ae0b-331dd41d99a4 name a) v4 -- ?+0 0x23ddf90 2013-04-25 18:37:54.517721 7fcccb474780 20 -- 10.1.0.3:6789/0 submit_message mon_probe(probe 2f2730c5-0504-4433-ae0b-331dd41d99a4 name a) v4 remote, 10.1.0.67:6789/0, new pipe. 2013-04-25 18:37:54.517728 7fcccb474780 10 -- 10.1.0.3:6789/0 connect_rank to 10.1.0.67:6789/0, creating pipe and registering 2013-04-25 18:37:54.517767 7fcccb474780 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=-1 :0 s=1 pgs=0 cs=0 l=0).register_pipe 2013-04-25 18:37:54.517805 7fcccb474780 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_probe(probe 2f2730c5-0504-4433-ae0b-331dd41d99a4 name a) v4 -- ?+0 0x241a540 2013-04-25 18:37:54.517812 7fcccb474780 20 -- 10.1.0.3:6789/0 submit_message mon_probe(probe 2f2730c5-0504-4433-ae0b-331dd41d99a4 name a) v4 remote, 10.1.0.130:6789/0, new pipe. 2013-04-25 18:37:54.517817 7fcccb474780 10 -- 10.1.0.3:6789/0 connect_rank to 10.1.0.130:6789/0, creating pipe and registering 2013-04-25 18:37:54.517807 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=-1 :0 s=1 pgs=0 cs=0 l=0).writer: state = connecting policy.server=0 2013-04-25 18:37:54.517856 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=-1 :0 s=1 pgs=0 cs=0 l=0).connect 0 2013-04-25 18:37:54.517859 7fcccb474780 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=-1 :0 s=1 pgs=0 cs=0 l=0).register_pipe 2013-04-25 18:37:54.517869 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :0 s=1 pgs=0 cs=0 l=0).connecting to 10.1.0.67:6789/0 2013-04-25 18:37:54.517874 7fcccb474780 10 -- 10.1.0.3:6789/0 wait: waiting for dispatch queue 2013-04-25 18:37:54.517912 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=-1 :0 s=1 pgs=0 cs=0 l=0).writer: state = connecting policy.server=0 2013-04-25 18:37:54.517951 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=-1 :0 s=1 pgs=0 cs=0 l=0).connect 0 2013-04-25 18:37:54.517969 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :0 s=1 pgs=0 cs=0 l=0).connecting to 10.1.0.130:6789/0 2013-04-25 18:37:54.518602 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect read peer addr 10.1.0.130:6789/0 on socket 22 2013-04-25 18:37:54.518629 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect peer addr for me is 10.1.0.3:48619/0 2013-04-25 18:37:54.518672 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect sent my addr 10.1.0.3:6789/0 2013-04-25 18:37:54.518705 7fccbf3d2700 10 mon.a@0(probing) e1 ms_get_authorizer for mon 2013-04-25 18:37:54.518713 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect read peer addr 10.1.0.67:6789/0 on socket 21 2013-04-25 18:37:54.518736 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect peer addr for me is 10.1.0.3:53248/0 2013-04-25 18:37:54.518769 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect sent my addr 10.1.0.3:6789/0 2013-04-25 18:37:54.518782 7fccbf4d3700 10 mon.a@0(probing) e1 ms_get_authorizer for mon 2013-04-25 18:37:54.519232 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect.authorizer_len=158 protocol=2 2013-04-25 18:37:54.519235 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect.authorizer_len=158 protocol=2 2013-04-25 18:37:54.519244 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect sending gseq=1 cseq=0 proto=10 2013-04-25 18:37:54.519247 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect sending gseq=2 cseq=0 proto=10 2013-04-25 18:37:54.519260 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply 2013-04-25 18:37:54.519269 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply 2013-04-25 18:37:54.519565 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect got reply tag 5 connect_seq 0 global_seq 149 proto 10 flags 0 2013-04-25 18:37:54.519592 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).reply.authorizer_len=36 2013-04-25 18:37:54.519623 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect got reply tag 5 connect_seq 0 global_seq 148 proto 10 flags 0 2013-04-25 18:37:54.519643 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).reply.authorizer_len=36 2013-04-25 18:37:54.519673 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect got RETRY_GLOBAL 149 chose new 150 2013-04-25 18:37:54.519692 7fccbf3d2700 10 mon.a@0(probing) e1 ms_get_authorizer for mon 2013-04-25 18:37:54.519692 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect got RETRY_GLOBAL 148 chose new 151 2013-04-25 18:37:54.519706 7fccbf4d3700 10 mon.a@0(probing) e1 ms_get_authorizer for mon 2013-04-25 18:37:54.519749 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect.authorizer_len=158 protocol=2 2013-04-25 18:37:54.519763 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect.authorizer_len=158 protocol=2 2013-04-25 18:37:54.519766 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect sending gseq=150 cseq=0 proto=10 2013-04-25 18:37:54.519775 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect sending gseq=151 cseq=0 proto=10 2013-04-25 18:37:54.519791 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply 2013-04-25 18:37:54.519795 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply 2013-04-25 18:37:54.520080 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).connect got reply tag 13 connect_seq 1 global_seq 3736 proto 10 flags 0 2013-04-25 18:37:54.520103 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).reply.authorizer_len=36 2013-04-25 18:37:54.520132 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).got CEPH_MSGR_TAG_SEQ, reading acked_seq and writing in_seq 2013-04-25 18:37:54.520151 7fccbf3d2700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=1 pgs=0 cs=0 l=0).reader got ack seq 0 2013-04-25 18:37:54.520172 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).connect success 1, lossy = 0, features 4294967295 2013-04-25 18:37:54.520194 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).connect starting reader 2013-04-25 18:37:54.520277 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).connect got reply tag 13 connect_seq 1 global_seq 3562 proto 10 flags 0 2013-04-25 18:37:54.520297 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).reply.authorizer_len=36 2013-04-25 18:37:54.520332 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).got CEPH_MSGR_TAG_SEQ, reading acked_seq and writing in_seq 2013-04-25 18:37:54.520340 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.520355 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 1 0x241a540 mon_probe(probe 2f2730c5-0504-4433-ae0b-331dd41d99a4 name a) v4 2013-04-25 18:37:54.520357 7fccbf4d3700 15 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=1 pgs=0 cs=0 l=0).reader got ack seq 0 2013-04-25 18:37:54.520387 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).connect success 1, lossy = 0, features 4294967295 2013-04-25 18:37:54.520403 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).connect starting reader 2013-04-25 18:37:54.520408 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 1): sig = 709577496562025090 2013-04-25 18:37:54.520430 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 1 0x241a540 2013-04-25 18:37:54.520447 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.520449 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.520463 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer encoding 1 0x23ddf90 mon_probe(probe 2f2730c5-0504-4433-ae0b-331dd41d99a4 name a) v4 2013-04-25 18:37:54.520467 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.520491 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer signed seq # 1): sig = 13473941960361791549 2013-04-25 18:37:54.520499 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sending 1 0x23ddf90 2013-04-25 18:37:54.520493 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.520519 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.520525 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.520514 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.521055 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ACK 2013-04-25 18:37:54.521070 7fccbf1d0700 15 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ack seq 1 2013-04-25 18:37:54.521075 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ack seq 1 >= 1 on 0x23ddf90 mon_probe(probe 2f2730c5-0504-4433-ae0b-331dd41d99a4 name a) v4 2013-04-25 18:37:54.521084 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.521089 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got MSG 2013-04-25 18:37:54.521102 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got envelope type=67 src mon.1 front=563 data=0 off 0 2013-04-25 18:37:54.521114 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader wants 563 from dispatch throttler 0/104857600 2013-04-25 18:37:54.521112 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:54.521127 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got front 563 2013-04-25 18:37:54.521133 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).aborted = 0 2013-04-25 18:37:54.521130 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 1 2013-04-25 18:37:54.521142 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got 563 + 0 + 0 byte message 2013-04-25 18:37:54.521146 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 1 >= 1 on 0x241a540 mon_probe(probe 2f2730c5-0504-4433-ae0b-331dd41d99a4 name a) v4 2013-04-25 18:37:54.521157 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.521162 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:54.521173 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=67 src mon.2 front=563 data=0 off 0 2013-04-25 18:37:54.521182 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 563 from dispatch throttler 563/104857600 2013-04-25 18:37:54.521191 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 563 2013-04-25 18:37:54.521201 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:54.521197 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got message 1454101234 0x7fca28001a20 mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name b quorum 1,2 paxos( fc 785672 lc 785779 )) v4 2013-04-25 18:37:54.521205 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 563 + 0 + 0 byte message 2013-04-25 18:37:54.521210 7fccbf1d0700 20 -- 10.1.0.3:6789/0 queue 0x7fca28001a20 prio 196 2013-04-25 18:37:54.521221 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.521250 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797986 0x7fca34001a20 mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name c quorum 1,2 paxos( fc 785672 lc 785779 )) v4 2013-04-25 18:37:54.521249 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.521261 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34001a20 prio 196 2013-04-25 18:37:54.521264 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).write_ack 1454101234 2013-04-25 18:37:54.521269 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.521276 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.521283 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.521276 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.521292 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797986 2013-04-25 18:37:54.521304 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.521311 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.521295 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.1 10.1.0.67:6789/0 1454101234 ==== mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name b quorum 1,2 paxos( fc 785672 lc 785779 )) v4 ==== 563+0+0 (4052182092 0 0) 0x7fca28001a20 con 0x23de8d0 2013-04-25 18:37:54.521364 7fccc04d5700 20 mon.a@0(probing) e1 have connection 2013-04-25 18:37:54.521367 7fccc04d5700 10 mon.a@0(probing) e1 do not have session, making new one 2013-04-25 18:37:54.521371 7fccc04d5700 10 mon.a@0(probing) e1 ms_dispatch new session MonSession: mon.1 10.1.0.67:6789/0 is open for mon.1 10.1.0.67:6789/0 2013-04-25 18:37:54.521377 7fccc04d5700 5 mon.a@0(probing) e1 setting monitor caps on this connection 2013-04-25 18:37:54.521382 7fccc04d5700 20 mon.a@0(probing) e1 caps allow * 2013-04-25 18:37:54.521385 7fccc04d5700 10 mon.a@0(probing) e1 handle_probe mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name b quorum 1,2 paxos( fc 785672 lc 785779 )) v4 2013-04-25 18:37:54.521398 7fccc04d5700 10 mon.a@0(probing) e1 handle_probe_reply mon.1 10.1.0.67:6789/0mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name b quorum 1,2 paxos( fc 785672 lc 785779 )) v4 2013-04-25 18:37:54.521405 7fccc04d5700 10 mon.a@0(probing) e1 monmap is e1: 3 mons at {a=10.1.0.3:6789/0,b=10.1.0.67:6789/0,c=10.1.0.130:6789/0} 2013-04-25 18:37:54.521426 7fccc04d5700 10 mon.a@0(probing) e1 peer name is b 2013-04-25 18:37:54.521436 7fccc04d5700 10 mon.a@0(probing) e1 existing quorum 1,2 2013-04-25 18:37:54.521438 7fccc04d5700 10 mon.a@0(probing) e1 peer paxos version 785779 vs my version 0 (too far ahead) 2013-04-25 18:37:54.521445 7fccc04d5700 10 mon.a@0(probing) e1 cancel_probe_timeout 0x23d9340 2013-04-25 18:37:54.521449 7fccc04d5700 10 mon.a@0(probing) e1 sync_start entity( mon.1 10.1.0.67:6789/0 ) 2013-04-25 18:37:54.918356 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )).monmap v1 get_monmap ver 0 2013-04-25 18:37:54.987813 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 sync_store_init backup current monmap 2013-04-25 18:37:54.995703 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.1 10.1.0.67:6789/0 -- mon_sync( start ) v1 -- ?+0 0x7fca2c082510 2013-04-25 18:37:54.995731 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( start ) v1 remote, 10.1.0.67:6789/0, have pipe. 2013-04-25 18:37:54.995782 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 563 to dispatch throttler 1126/104857600 2013-04-25 18:37:54.995790 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca28001a20 2013-04-25 18:37:54.995801 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797986 ==== mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name c quorum 1,2 paxos( fc 785672 lc 785779 )) v4 ==== 563+0+0 (386503924 0 0) 0x7fca34001a20 con 0x241ae80 2013-04-25 18:37:54.995818 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state start )) e1 have connection 2013-04-25 18:37:54.995823 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 do not have session, making new one 2013-04-25 18:37:54.995828 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_dispatch new session MonSession: mon.2 10.1.0.130:6789/0 is open for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:54.995837 7fccc04d5700 5 mon.a@0(synchronizing sync( requester state start )) e1 setting monitor caps on this connection 2013-04-25 18:37:54.995842 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state start )) e1 caps allow * 2013-04-25 18:37:54.995846 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 handle_probe mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name c quorum 1,2 paxos( fc 785672 lc 785779 )) v4 2013-04-25 18:37:54.995840 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.995855 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 handle_probe_reply mon.2 10.1.0.130:6789/0mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name c quorum 1,2 paxos( fc 785672 lc 785779 )) v4 2013-04-25 18:37:54.995863 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer encoding 2 0x7fca2c082510 mon_sync( start ) v1 2013-04-25 18:37:54.995866 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 monmap is e1: 3 mons at {a=10.1.0.3:6789/0,b=10.1.0.67:6789/0,c=10.1.0.130:6789/0} 2013-04-25 18:37:54.995876 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 563 to dispatch throttler 563/104857600 2013-04-25 18:37:54.995879 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34001a20 2013-04-25 18:37:54.995913 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer signed seq # 2): sig = 5298457032732625181 2013-04-25 18:37:54.995922 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sending 2 0x7fca2c082510 2013-04-25 18:37:54.995972 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.995979 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.996640 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ACK 2013-04-25 18:37:54.996667 7fccbf1d0700 15 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ack seq 2 2013-04-25 18:37:54.996676 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ack seq 2 >= 2 on 0x7fca2c082510 mon_sync( start ) v1 2013-04-25 18:37:54.996685 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.996693 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got MSG 2013-04-25 18:37:54.996703 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got envelope type=69 src mon.1 front=174 data=0 off 0 2013-04-25 18:37:54.996712 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader wants 174 from dispatch throttler 0/104857600 2013-04-25 18:37:54.996723 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got front 174 2013-04-25 18:37:54.996732 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).aborted = 0 2013-04-25 18:37:54.996740 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got 174 + 0 + 0 byte message 2013-04-25 18:37:54.996794 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got message 1454101235 0x7fca28002c10 mon_sync( start_reply flags( reply-to(mon.2 10.1.0.130:6789/0) ) ) v1 2013-04-25 18:37:54.996804 7fccbf1d0700 20 -- 10.1.0.3:6789/0 queue 0x7fca28002c10 prio 196 2013-04-25 18:37:54.996812 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.996818 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.996831 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).write_ack 1454101235 2013-04-25 18:37:54.996846 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.996854 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.996873 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.1 10.1.0.67:6789/0 1454101235 ==== mon_sync( start_reply flags( reply-to(mon.2 10.1.0.130:6789/0) ) ) v1 ==== 174+0+0 (3989053524 0 0) 0x7fca28002c10 con 0x23de8d0 2013-04-25 18:37:54.996899 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state start )) e1 have connection 2013-04-25 18:37:54.996908 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state start )) e1 ms_dispatch existing session MonSession: mon.1 10.1.0.67:6789/0 is openallow * for mon.1 10.1.0.67:6789/0 2013-04-25 18:37:54.996920 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state start )) e1 caps allow * 2013-04-25 18:37:54.996930 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 handle_sync mon_sync( start_reply flags( reply-to(mon.2 10.1.0.130:6789/0) ) ) v1 2013-04-25 18:37:54.996947 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 handle_sync_start_reply mon_sync( start_reply flags( reply-to(mon.2 10.1.0.130:6789/0) ) ) v1 2013-04-25 18:37:54.996957 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 handle_sync_start_reply leader told us to use mon.2 10.1.0.130:6789/0 as sync provider 2013-04-25 18:37:54.996967 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 sync_send_heartbeat mon.1 10.1.0.67:6789/0 reply(0) 2013-04-25 18:37:54.996975 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.1 10.1.0.67:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x7fca2c083380 2013-04-25 18:37:54.996981 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( heartbeat ) v1 remote, 10.1.0.67:6789/0, have pipe. 2013-04-25 18:37:54.996992 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state start )) e1 sync_start_chunks provider(mon.2 10.1.0.130:6789/0) 2013-04-25 18:37:54.997002 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( start_chunks ) v1 -- ?+0 0x7fca2c083a20 2013-04-25 18:37:54.997008 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( start_chunks ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:54.997016 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 174 to dispatch throttler 174/104857600 2013-04-25 18:37:54.997020 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca28002c10 2013-04-25 18:37:54.997037 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.997043 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.997052 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer encoding 3 0x7fca2c083380 mon_sync( heartbeat ) v1 2013-04-25 18:37:54.997060 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 2 0x7fca2c083a20 mon_sync( start_chunks ) v1 2013-04-25 18:37:54.997093 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer signed seq # 3): sig = 16199049791773376837 2013-04-25 18:37:54.997100 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 2): sig = 3567462118199286101 2013-04-25 18:37:54.997110 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 2 0x7fca2c083a20 2013-04-25 18:37:54.997107 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sending 3 0x7fca2c083380 2013-04-25 18:37:54.997132 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.997138 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.997135 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.997146 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.997788 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ACK 2013-04-25 18:37:54.997815 7fccbf1d0700 15 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ack seq 3 2013-04-25 18:37:54.997825 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got ack seq 3 >= 3 on 0x7fca2c083380 mon_sync( heartbeat ) v1 2013-04-25 18:37:54.997835 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.997843 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got MSG 2013-04-25 18:37:54.997852 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got envelope type=69 src mon.1 front=174 data=0 off 0 2013-04-25 18:37:54.997860 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader wants 174 from dispatch throttler 0/104857600 2013-04-25 18:37:54.997871 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got front 174 2013-04-25 18:37:54.997880 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).aborted = 0 2013-04-25 18:37:54.997889 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got 174 + 0 + 0 byte message 2013-04-25 18:37:54.997924 7fccbf1d0700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader got message 1454101236 0x7fca28003c70 mon_sync( heartbeat_reply ) v1 2013-04-25 18:37:54.997931 7fccbf1d0700 20 -- 10.1.0.3:6789/0 queue 0x7fca28003c70 prio 196 2013-04-25 18:37:54.997937 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader reading tag... 2013-04-25 18:37:54.997973 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.997985 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).write_ack 1454101236 2013-04-25 18:37:54.997995 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:54.998002 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).writer sleeping 2013-04-25 18:37:54.997999 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.1 10.1.0.67:6789/0 1454101236 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x7fca28003c70 con 0x23de8d0 2013-04-25 18:37:54.998021 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:54.998028 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.1 10.1.0.67:6789/0 is openallow * for mon.1 10.1.0.67:6789/0 2013-04-25 18:37:54.998042 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:54.998046 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 2013-04-25 18:37:54.998050 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 2013-04-25 18:37:54.998056 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 174 to dispatch throttler 174/104857600 2013-04-25 18:37:54.998061 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca28003c70 2013-04-25 18:37:55.206666 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.206697 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 2 2013-04-25 18:37:55.206717 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 2 >= 2 on 0x7fca2c083a20 mon_sync( start_chunks ) v1 2013-04-25 18:37:55.206731 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.262802 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.262838 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1038756 data=0 off 0 2013-04-25 18:37:55.262850 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1038756 from dispatch throttler 0/104857600 2013-04-25 18:37:55.274107 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1038756 2013-04-25 18:37:55.274140 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.274159 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1038756 + 0 + 0 byte message 2013-04-25 18:37:55.274953 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797987 0x7fca34100330 mon_sync( chunk bl 1038567 bytes last_key ( logm,full_309184 ) ) v1 2013-04-25 18:37:55.274966 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34100330 prio 196 2013-04-25 18:37:55.274974 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.275023 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.275040 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797987 2013-04-25 18:37:55.275052 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.275060 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.275060 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797987 ==== mon_sync( chunk bl 1038567 bytes last_key ( logm,full_309184 ) ) v1 ==== 1038756+0+0 (3290816178 0 0) 0x7fca34100330 con 0x241ae80 2013-04-25 18:37:55.275083 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.275091 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.275102 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.275106 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038567 bytes last_key ( logm,full_309184 ) ) v1 2013-04-25 18:37:55.275116 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038567 bytes last_key ( logm,full_309184 ) ) v1 2013-04-25 18:37:55.275815 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c0b6f20 2013-04-25 18:37:55.275827 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.276000 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.276025 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 3 0x7fca2c0b6f20 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.276067 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 3): sig = 2770877624728815153 2013-04-25 18:37:55.276087 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 3 0x7fca2c0b6f20 2013-04-25 18:37:55.276106 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.276113 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.282400 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.282428 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 3 2013-04-25 18:37:55.282517 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 3 >= 3 on 0x7fca2c0b6f20 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.282539 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.282560 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.282569 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1039846 data=0 off 0 2013-04-25 18:37:55.282577 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1039846 from dispatch throttler 1038756/104857600 2013-04-25 18:37:55.290147 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1038756 to dispatch throttler 2078602/104857600 2013-04-25 18:37:55.290169 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34100330 2013-04-25 18:37:55.291194 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1039846 2013-04-25 18:37:55.291222 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.291229 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1039846 + 0 + 0 byte message 2013-04-25 18:37:55.291978 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797988 0x7fca340027b0 mon_sync( chunk bl 1039657 bytes last_key ( logm,full_309254 ) ) v1 2013-04-25 18:37:55.291990 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca340027b0 prio 196 2013-04-25 18:37:55.292001 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.292002 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.292006 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797988 ==== mon_sync( chunk bl 1039657 bytes last_key ( logm,full_309254 ) ) v1 ==== 1039846+0+0 (512143511 0 0) 0x7fca340027b0 con 0x241ae80 2013-04-25 18:37:55.292018 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.292018 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797988 2013-04-25 18:37:55.292024 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.292031 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.292034 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1039657 bytes last_key ( logm,full_309254 ) ) v1 2013-04-25 18:37:55.292030 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.292038 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1039657 bytes last_key ( logm,full_309254 ) ) v1 2013-04-25 18:37:55.292039 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.292092 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c1df740 2013-04-25 18:37:55.292099 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.292151 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.292169 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 4 0x7fca2c1df740 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.292200 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 4): sig = 2950678740948563378 2013-04-25 18:37:55.292212 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 4 0x7fca2c1df740 2013-04-25 18:37:55.292232 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.292240 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.297504 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.297535 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 4 2013-04-25 18:37:55.297549 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 4 >= 4 on 0x7fca2c1df740 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.297602 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.297647 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.297657 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1039154 data=0 off 0 2013-04-25 18:37:55.297669 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1039154 from dispatch throttler 1039846/104857600 2013-04-25 18:37:55.306288 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1039154 2013-04-25 18:37:55.306310 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.306315 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1039154 + 0 + 0 byte message 2013-04-25 18:37:55.306555 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1039846 to dispatch throttler 2079000/104857600 2013-04-25 18:37:55.306571 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca340027b0 2013-04-25 18:37:55.307106 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797989 0x7fca340027b0 mon_sync( chunk bl 1038965 bytes last_key ( logm,full_309324 ) ) v1 2013-04-25 18:37:55.307118 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca340027b0 prio 196 2013-04-25 18:37:55.307127 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.307151 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.307167 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797989 2013-04-25 18:37:55.307180 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.307188 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.307217 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797989 ==== mon_sync( chunk bl 1038965 bytes last_key ( logm,full_309324 ) ) v1 ==== 1039154+0+0 (1960894935 0 0) 0x7fca340027b0 con 0x241ae80 2013-04-25 18:37:55.307234 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.307241 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.307250 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.307254 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038965 bytes last_key ( logm,full_309324 ) ) v1 2013-04-25 18:37:55.307258 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038965 bytes last_key ( logm,full_309324 ) ) v1 2013-04-25 18:37:55.307313 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c085230 2013-04-25 18:37:55.307324 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.307410 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.307433 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 5 0x7fca2c085230 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.307473 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 5): sig = 10608811847928868873 2013-04-25 18:37:55.307482 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 5 0x7fca2c085230 2013-04-25 18:37:55.307502 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.307508 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.312671 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.312760 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 5 2013-04-25 18:37:55.312774 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 5 >= 5 on 0x7fca2c085230 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.312790 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.312801 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.312811 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1039301 data=0 off 0 2013-04-25 18:37:55.312820 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1039301 from dispatch throttler 1039154/104857600 2013-04-25 18:37:55.318355 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1039154 to dispatch throttler 2078455/104857600 2013-04-25 18:37:55.318374 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca340027b0 2013-04-25 18:37:55.321677 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1039301 2013-04-25 18:37:55.321708 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.321722 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1039301 + 0 + 0 byte message 2013-04-25 18:37:55.322524 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797990 0x7fca34004ab0 mon_sync( chunk bl 1039112 bytes last_key ( logm,full_309394 ) ) v1 2013-04-25 18:37:55.322536 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34004ab0 prio 196 2013-04-25 18:37:55.322544 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.322592 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.322613 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797990 2013-04-25 18:37:55.322632 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.322643 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.322647 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797990 ==== mon_sync( chunk bl 1039112 bytes last_key ( logm,full_309394 ) ) v1 ==== 1039301+0+0 (643574121 0 0) 0x7fca34004ab0 con 0x241ae80 2013-04-25 18:37:55.322670 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.322678 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.322691 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.322696 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1039112 bytes last_key ( logm,full_309394 ) ) v1 2013-04-25 18:37:55.322702 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1039112 bytes last_key ( logm,full_309394 ) ) v1 2013-04-25 18:37:55.322765 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c085780 2013-04-25 18:37:55.322777 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.322843 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.322858 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 6 0x7fca2c085780 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.322889 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 6): sig = 911070850942013719 2013-04-25 18:37:55.322903 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 6 0x7fca2c085780 2013-04-25 18:37:55.322925 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.322931 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.328137 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.328169 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 6 2013-04-25 18:37:55.328175 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 6 >= 6 on 0x7fca2c085780 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.328186 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.328192 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.328202 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1039162 data=0 off 0 2013-04-25 18:37:55.328209 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1039162 from dispatch throttler 1039301/104857600 2013-04-25 18:37:55.335744 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1039301 to dispatch throttler 2078463/104857600 2013-04-25 18:37:55.335763 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34004ab0 2013-04-25 18:37:55.336918 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1039162 2013-04-25 18:37:55.336942 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.336947 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1039162 + 0 + 0 byte message 2013-04-25 18:37:55.337711 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797991 0x7fca34005cb0 mon_sync( chunk bl 1038973 bytes last_key ( logm,full_309464 ) ) v1 2013-04-25 18:37:55.337724 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34005cb0 prio 196 2013-04-25 18:37:55.337733 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.337797 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.337798 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797991 ==== mon_sync( chunk bl 1038973 bytes last_key ( logm,full_309464 ) ) v1 ==== 1039162+0+0 (364931340 0 0) 0x7fca34005cb0 con 0x241ae80 2013-04-25 18:37:55.337813 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797991 2013-04-25 18:37:55.337815 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.337822 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.337828 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.337832 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.337835 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.337836 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038973 bytes last_key ( logm,full_309464 ) ) v1 2013-04-25 18:37:55.337840 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038973 bytes last_key ( logm,full_309464 ) ) v1 2013-04-25 18:37:55.337896 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c1d59a0 2013-04-25 18:37:55.337904 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.337977 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.337996 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 7 0x7fca2c1d59a0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.338037 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 7): sig = 1759742916706816354 2013-04-25 18:37:55.338047 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 7 0x7fca2c1d59a0 2013-04-25 18:37:55.338068 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.338074 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.343304 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.343336 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 7 2013-04-25 18:37:55.343348 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 7 >= 7 on 0x7fca2c1d59a0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.343360 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.343369 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.343379 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1040092 data=0 off 0 2013-04-25 18:37:55.343388 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1040092 from dispatch throttler 1039162/104857600 2013-04-25 18:37:55.349095 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1039162 to dispatch throttler 2079254/104857600 2013-04-25 18:37:55.349115 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34005cb0 2013-04-25 18:37:55.352261 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1040092 2013-04-25 18:37:55.352291 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.352302 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1040092 + 0 + 0 byte message 2013-04-25 18:37:55.353266 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797992 0x7fca34006eb0 mon_sync( chunk bl 1039903 bytes last_key ( logm,full_309534 ) ) v1 2013-04-25 18:37:55.353281 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34006eb0 prio 196 2013-04-25 18:37:55.353291 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.353311 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.353328 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797992 2013-04-25 18:37:55.353340 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.353348 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.353379 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797992 ==== mon_sync( chunk bl 1039903 bytes last_key ( logm,full_309534 ) ) v1 ==== 1040092+0+0 (2740963768 0 0) 0x7fca34006eb0 con 0x241ae80 2013-04-25 18:37:55.353396 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.353403 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.353413 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.353417 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1039903 bytes last_key ( logm,full_309534 ) ) v1 2013-04-25 18:37:55.353421 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1039903 bytes last_key ( logm,full_309534 ) ) v1 2013-04-25 18:37:55.353477 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c1d8e20 2013-04-25 18:37:55.353489 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.353574 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.353590 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 8 0x7fca2c1d8e20 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.353629 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 8): sig = 16113205116614110123 2013-04-25 18:37:55.353642 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 8 0x7fca2c1d8e20 2013-04-25 18:37:55.353660 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.353666 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.358897 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.358929 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 8 2013-04-25 18:37:55.358937 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 8 >= 8 on 0x7fca2c1d8e20 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.358947 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.358956 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.358964 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1038915 data=0 off 0 2013-04-25 18:37:55.358973 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1038915 from dispatch throttler 1040092/104857600 2013-04-25 18:37:55.367462 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1040092 to dispatch throttler 2079007/104857600 2013-04-25 18:37:55.367480 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34006eb0 2013-04-25 18:37:55.367654 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1038915 2013-04-25 18:37:55.367675 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.367684 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1038915 + 0 + 0 byte message 2013-04-25 18:37:55.368499 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797993 0x7fca34008110 mon_sync( chunk bl 1038726 bytes last_key ( logm,full_309604 ) ) v1 2013-04-25 18:37:55.368512 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34008110 prio 196 2013-04-25 18:37:55.368521 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.368549 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.368565 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797993 2013-04-25 18:37:55.368578 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.368586 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.368577 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797993 ==== mon_sync( chunk bl 1038726 bytes last_key ( logm,full_309604 ) ) v1 ==== 1038915+0+0 (3737555516 0 0) 0x7fca34008110 con 0x241ae80 2013-04-25 18:37:55.368594 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.368601 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.368610 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.368614 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038726 bytes last_key ( logm,full_309604 ) ) v1 2013-04-25 18:37:55.368619 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038726 bytes last_key ( logm,full_309604 ) ) v1 2013-04-25 18:37:55.368674 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c1d9440 2013-04-25 18:37:55.368682 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.368736 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.368752 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 9 0x7fca2c1d9440 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.368788 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 9): sig = 6370254146917773501 2013-04-25 18:37:55.368799 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 9 0x7fca2c1d9440 2013-04-25 18:37:55.368819 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.368824 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.379280 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1038915 to dispatch throttler 1038915/104857600 2013-04-25 18:37:55.379301 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34008110 2013-04-25 18:37:55.404664 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.404721 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 9 2013-04-25 18:37:55.404730 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 9 >= 9 on 0x7fca2c1d9440 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.404740 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.404746 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.404753 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1031823 data=0 off 0 2013-04-25 18:37:55.404760 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1031823 from dispatch throttler 0/104857600 2013-04-25 18:37:55.413369 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1031823 2013-04-25 18:37:55.413399 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.413409 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1031823 + 0 + 0 byte message 2013-04-25 18:37:55.414153 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797994 0x7fca340093c0 mon_sync( chunk bl 1031640 bytes last_key ( osdmap,725 ) ) v1 2013-04-25 18:37:55.414165 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca340093c0 prio 196 2013-04-25 18:37:55.414174 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.414229 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.414231 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797994 ==== mon_sync( chunk bl 1031640 bytes last_key ( osdmap,725 ) ) v1 ==== 1031823+0+0 (1464295352 0 0) 0x7fca340093c0 con 0x241ae80 2013-04-25 18:37:55.414245 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797994 2013-04-25 18:37:55.414248 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.414255 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.414258 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.414264 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.414266 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.414269 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1031640 bytes last_key ( osdmap,725 ) ) v1 2013-04-25 18:37:55.414273 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1031640 bytes last_key ( osdmap,725 ) ) v1 2013-04-25 18:37:55.414507 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c521ff0 2013-04-25 18:37:55.414520 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.414590 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.414605 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 10 0x7fca2c521ff0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.414644 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 10): sig = 11286446586133159870 2013-04-25 18:37:55.414658 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 10 0x7fca2c521ff0 2013-04-25 18:37:55.414677 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.414683 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.428982 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1031823 to dispatch throttler 1031823/104857600 2013-04-25 18:37:55.429002 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca340093c0 2013-04-25 18:37:55.447181 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.447213 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 10 2013-04-25 18:37:55.447225 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 10 >= 10 on 0x7fca2c521ff0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.447243 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.447253 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.447263 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1026994 data=0 off 0 2013-04-25 18:37:55.447272 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1026994 from dispatch throttler 0/104857600 2013-04-25 18:37:55.455993 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1026994 2013-04-25 18:37:55.456024 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.456035 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1026994 + 0 + 0 byte message 2013-04-25 18:37:55.456852 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797995 0x7fca3400a550 mon_sync( chunk bl 1026805 bytes last_key ( osdmap,full_1021 ) ) v1 2013-04-25 18:37:55.456867 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3400a550 prio 196 2013-04-25 18:37:55.456884 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.456884 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.456899 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797995 2013-04-25 18:37:55.456898 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797995 ==== mon_sync( chunk bl 1026805 bytes last_key ( osdmap,full_1021 ) ) v1 ==== 1026994+0+0 (3427797981 0 0) 0x7fca3400a550 con 0x241ae80 2013-04-25 18:37:55.456914 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.456922 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.456922 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.456934 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.456949 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.456955 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1026805 bytes last_key ( osdmap,full_1021 ) ) v1 2013-04-25 18:37:55.456962 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1026805 bytes last_key ( osdmap,full_1021 ) ) v1 2013-04-25 18:37:55.457320 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c5227e0 2013-04-25 18:37:55.457340 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.457401 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.457413 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 11 0x7fca2c5227e0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.457451 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 11): sig = 15502609671159252611 2013-04-25 18:37:55.457469 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 11 0x7fca2c5227e0 2013-04-25 18:37:55.457490 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.457495 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.460815 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.460842 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 11 2013-04-25 18:37:55.460848 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 11 >= 11 on 0x7fca2c5227e0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.460861 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.460870 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.460877 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037239 data=0 off 0 2013-04-25 18:37:55.460911 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037239 from dispatch throttler 1026994/104857600 2013-04-25 18:37:55.469546 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037239 2013-04-25 18:37:55.469587 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.469593 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037239 + 0 + 0 byte message 2013-04-25 18:37:55.469993 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1026994 to dispatch throttler 2064233/104857600 2013-04-25 18:37:55.470017 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3400a550 2013-04-25 18:37:55.470409 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797996 0x7fca3400a550 mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1050 ) ) v1 2013-04-25 18:37:55.470423 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3400a550 prio 196 2013-04-25 18:37:55.470436 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.470438 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.470455 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797996 2013-04-25 18:37:55.470448 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797996 ==== mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1050 ) ) v1 ==== 1037239+0+0 (1598584995 0 0) 0x7fca3400a550 con 0x241ae80 2013-04-25 18:37:55.470469 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.470470 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.470479 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.470479 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.470492 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.470497 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1050 ) ) v1 2013-04-25 18:37:55.470503 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1050 ) ) v1 2013-04-25 18:37:55.470548 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c0b6290 2013-04-25 18:37:55.470561 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.470614 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.470626 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 12 0x7fca2c0b6290 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.470659 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 12): sig = 10992318104398312395 2013-04-25 18:37:55.470673 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 12 0x7fca2c0b6290 2013-04-25 18:37:55.470697 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.470703 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.473819 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.473851 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 12 2013-04-25 18:37:55.473863 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 12 >= 12 on 0x7fca2c0b6290 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.473883 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.473893 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.473905 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037239 data=0 off 0 2013-04-25 18:37:55.473917 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037239 from dispatch throttler 1037239/104857600 2013-04-25 18:37:55.482623 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037239 2013-04-25 18:37:55.482667 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.482681 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037239 + 0 + 0 byte message 2013-04-25 18:37:55.483126 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037239 to dispatch throttler 2074478/104857600 2013-04-25 18:37:55.483147 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3400a550 2013-04-25 18:37:55.483664 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797997 0x7fca3400a550 mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1079 ) ) v1 2013-04-25 18:37:55.483683 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3400a550 prio 196 2013-04-25 18:37:55.483695 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.483704 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797997 ==== mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1079 ) ) v1 ==== 1037239+0+0 (824023551 0 0) 0x7fca3400a550 con 0x241ae80 2013-04-25 18:37:55.483719 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.483711 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.483725 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.483729 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797997 2013-04-25 18:37:55.483733 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.483737 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1079 ) ) v1 2013-04-25 18:37:55.483741 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1079 ) ) v1 2013-04-25 18:37:55.483744 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.483752 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.483774 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c0b2690 2013-04-25 18:37:55.483781 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.483859 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.483874 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 13 0x7fca2c0b2690 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.483916 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 13): sig = 2015015910638205481 2013-04-25 18:37:55.483930 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 13 0x7fca2c0b2690 2013-04-25 18:37:55.483959 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.483965 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.487275 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.487303 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 13 2013-04-25 18:37:55.487309 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 13 >= 13 on 0x7fca2c0b2690 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.487323 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.487330 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.487337 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037239 data=0 off 0 2013-04-25 18:37:55.487344 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037239 from dispatch throttler 1037239/104857600 2013-04-25 18:37:55.493635 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037239 to dispatch throttler 2074478/104857600 2013-04-25 18:37:55.493656 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3400a550 2013-04-25 18:37:55.496071 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037239 2013-04-25 18:37:55.496091 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.496099 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037239 + 0 + 0 byte message 2013-04-25 18:37:55.496895 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797998 0x7fca3400d920 mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1108 ) ) v1 2013-04-25 18:37:55.496908 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3400d920 prio 196 2013-04-25 18:37:55.496922 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.496944 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.496948 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797998 ==== mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1108 ) ) v1 ==== 1037239+0+0 (570329346 0 0) 0x7fca3400d920 con 0x241ae80 2013-04-25 18:37:55.496963 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797998 2013-04-25 18:37:55.496965 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.496974 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.496982 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.496990 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.496991 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.496996 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1108 ) ) v1 2013-04-25 18:37:55.497002 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1108 ) ) v1 2013-04-25 18:37:55.497040 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2caa6f90 2013-04-25 18:37:55.497049 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.497064 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.497080 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 14 0x7fca2caa6f90 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.497126 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 14): sig = 11390228712199208102 2013-04-25 18:37:55.497147 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 14 0x7fca2caa6f90 2013-04-25 18:37:55.497185 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.497199 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.500785 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.500810 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 14 2013-04-25 18:37:55.500820 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 14 >= 14 on 0x7fca2caa6f90 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.500830 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.500837 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.500844 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037239 data=0 off 0 2013-04-25 18:37:55.500850 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037239 from dispatch throttler 1037239/104857600 2013-04-25 18:37:55.509495 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037239 2013-04-25 18:37:55.509538 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.509548 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037239 + 0 + 0 byte message 2013-04-25 18:37:55.509785 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037239 to dispatch throttler 2074478/104857600 2013-04-25 18:37:55.509803 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3400d920 2013-04-25 18:37:55.510355 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192797999 0x7fca3400d920 mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1137 ) ) v1 2013-04-25 18:37:55.510372 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3400d920 prio 196 2013-04-25 18:37:55.510385 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.510401 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.510405 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192797999 ==== mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1137 ) ) v1 ==== 1037239+0+0 (457154565 0 0) 0x7fca3400d920 con 0x241ae80 2013-04-25 18:37:55.510423 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.510420 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192797999 2013-04-25 18:37:55.510432 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.510441 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.510447 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.510450 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.510453 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1137 ) ) v1 2013-04-25 18:37:55.510459 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037050 bytes last_key ( osdmap,full_1137 ) ) v1 2013-04-25 18:37:55.510499 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c1d9cd0 2013-04-25 18:37:55.510509 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.510524 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.510539 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 15 0x7fca2c1d9cd0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.510591 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 15): sig = 2834729274282869546 2013-04-25 18:37:55.510613 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 15 0x7fca2c1d9cd0 2013-04-25 18:37:55.510651 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.510664 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.513552 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.513582 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 15 2013-04-25 18:37:55.513588 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 15 >= 15 on 0x7fca2c1d9cd0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.513602 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.513608 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.513615 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1038647 data=0 off 0 2013-04-25 18:37:55.513627 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1038647 from dispatch throttler 1037239/104857600 2013-04-25 18:37:55.521069 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037239 to dispatch throttler 2075886/104857600 2013-04-25 18:37:55.521100 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3400d920 2013-04-25 18:37:55.522303 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1038647 2013-04-25 18:37:55.522323 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.522328 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1038647 + 0 + 0 byte message 2013-04-25 18:37:55.523117 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798000 0x7fca3400fa70 mon_sync( chunk bl 1038458 bytes last_key ( osdmap,full_1166 ) ) v1 2013-04-25 18:37:55.523130 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3400fa70 prio 196 2013-04-25 18:37:55.523146 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.523147 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.523156 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798000 ==== mon_sync( chunk bl 1038458 bytes last_key ( osdmap,full_1166 ) ) v1 ==== 1038647+0+0 (3277312127 0 0) 0x7fca3400fa70 con 0x241ae80 2013-04-25 18:37:55.523164 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798000 2013-04-25 18:37:55.523174 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.523182 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.523184 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.523193 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.523199 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.523205 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038458 bytes last_key ( osdmap,full_1166 ) ) v1 2013-04-25 18:37:55.523212 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038458 bytes last_key ( osdmap,full_1166 ) ) v1 2013-04-25 18:37:55.523248 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c2f0030 2013-04-25 18:37:55.523257 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.523272 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.523288 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 16 0x7fca2c2f0030 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.523334 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 16): sig = 339650802408764611 2013-04-25 18:37:55.523356 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 16 0x7fca2c2f0030 2013-04-25 18:37:55.523385 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.523400 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.526131 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.526158 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 16 2013-04-25 18:37:55.526163 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 16 >= 16 on 0x7fca2c2f0030 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.526172 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.526178 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.526185 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1039095 data=0 off 0 2013-04-25 18:37:55.526191 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1039095 from dispatch throttler 1038647/104857600 2013-04-25 18:37:55.534935 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1039095 2013-04-25 18:37:55.534975 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.534988 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1039095 + 0 + 0 byte message 2013-04-25 18:37:55.535818 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798001 0x7fca34010eb0 mon_sync( chunk bl 1038906 bytes last_key ( osdmap,full_1195 ) ) v1 2013-04-25 18:37:55.535837 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34010eb0 prio 196 2013-04-25 18:37:55.535855 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.535857 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.535873 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798001 2013-04-25 18:37:55.535887 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.535895 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.535916 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1038647 to dispatch throttler 2077742/104857600 2013-04-25 18:37:55.535930 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3400fa70 2013-04-25 18:37:55.536095 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798001 ==== mon_sync( chunk bl 1038906 bytes last_key ( osdmap,full_1195 ) ) v1 ==== 1039095+0+0 (754964379 0 0) 0x7fca34010eb0 con 0x241ae80 2013-04-25 18:37:55.536110 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.536116 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.536128 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.536131 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038906 bytes last_key ( osdmap,full_1195 ) ) v1 2013-04-25 18:37:55.536135 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038906 bytes last_key ( osdmap,full_1195 ) ) v1 2013-04-25 18:37:55.536168 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c2f08c0 2013-04-25 18:37:55.536175 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.536227 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.536243 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 17 0x7fca2c2f08c0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.536285 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 17): sig = 2330945313903472738 2013-04-25 18:37:55.536298 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 17 0x7fca2c2f08c0 2013-04-25 18:37:55.536322 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.536330 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.539945 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.539976 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 17 2013-04-25 18:37:55.539984 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 17 >= 17 on 0x7fca2c2f08c0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.540000 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.540009 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.540018 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037794 data=0 off 0 2013-04-25 18:37:55.540027 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037794 from dispatch throttler 1039095/104857600 2013-04-25 18:37:55.546705 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1039095 to dispatch throttler 2076889/104857600 2013-04-25 18:37:55.546741 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34010eb0 2013-04-25 18:37:55.548748 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037794 2013-04-25 18:37:55.548773 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.548783 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037794 + 0 + 0 byte message 2013-04-25 18:37:55.549584 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798002 0x7fca34011e60 mon_sync( chunk bl 1037606 bytes last_key ( osdmap,full_693 ) ) v1 2013-04-25 18:37:55.549600 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34011e60 prio 196 2013-04-25 18:37:55.549615 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.549635 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.549652 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798002 2013-04-25 18:37:55.549666 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.549656 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798002 ==== mon_sync( chunk bl 1037606 bytes last_key ( osdmap,full_693 ) ) v1 ==== 1037794+0+0 (876201757 0 0) 0x7fca34011e60 con 0x241ae80 2013-04-25 18:37:55.549675 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.549680 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.549691 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.549709 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.549715 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037606 bytes last_key ( osdmap,full_693 ) ) v1 2013-04-25 18:37:55.549722 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037606 bytes last_key ( osdmap,full_693 ) ) v1 2013-04-25 18:37:55.549781 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c79b3a0 2013-04-25 18:37:55.549793 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.549848 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.549869 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 18 0x7fca2c79b3a0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.549904 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 18): sig = 13739428951462607701 2013-04-25 18:37:55.549917 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 18 0x7fca2c79b3a0 2013-04-25 18:37:55.549940 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.549946 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.553171 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.553209 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 18 2013-04-25 18:37:55.553221 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 18 >= 18 on 0x7fca2c79b3a0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.553235 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.553245 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.553256 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.553267 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037794/104857600 2013-04-25 18:37:55.561956 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.561992 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.562000 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.562786 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798003 0x7fca341fed30 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_722 ) ) v1 2013-04-25 18:37:55.562802 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca341fed30 prio 196 2013-04-25 18:37:55.562814 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.562837 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.562853 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798003 2013-04-25 18:37:55.562868 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.562876 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.564221 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037794 to dispatch throttler 2075003/104857600 2013-04-25 18:37:55.564247 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34011e60 2013-04-25 18:37:55.564396 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798003 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_722 ) ) v1 ==== 1037209+0+0 (3728255758 0 0) 0x7fca341fed30 con 0x241ae80 2013-04-25 18:37:55.564418 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.564427 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.564441 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.564446 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_722 ) ) v1 2013-04-25 18:37:55.564451 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_722 ) ) v1 2013-04-25 18:37:55.564500 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c83a5d0 2013-04-25 18:37:55.564520 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.564591 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.564611 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 19 0x7fca2c83a5d0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.564664 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 19): sig = 5207791601088025747 2013-04-25 18:37:55.564676 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 19 0x7fca2c83a5d0 2013-04-25 18:37:55.564709 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.564720 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.568052 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.568082 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 19 2013-04-25 18:37:55.568088 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 19 >= 19 on 0x7fca2c83a5d0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.568103 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.568109 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.568116 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.568123 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.575690 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.575721 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca341fed30 2013-04-25 18:37:55.576848 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.576870 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.576879 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.577675 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798004 0x7fca34013ac0 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_751 ) ) v1 2013-04-25 18:37:55.577689 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34013ac0 prio 196 2013-04-25 18:37:55.577703 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.577715 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798004 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_751 ) ) v1 ==== 1037209+0+0 (1482425405 0 0) 0x7fca34013ac0 con 0x241ae80 2013-04-25 18:37:55.577728 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.577737 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.577745 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798004 2013-04-25 18:37:55.577746 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.577760 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.577759 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.577765 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_751 ) ) v1 2013-04-25 18:37:55.577766 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.577770 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_751 ) ) v1 2013-04-25 18:37:55.577817 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c83b0d0 2013-04-25 18:37:55.577831 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.577917 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.577936 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 20 0x7fca2c83b0d0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.577976 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 20): sig = 13369622967797957966 2013-04-25 18:37:55.577987 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 20 0x7fca2c83b0d0 2013-04-25 18:37:55.578009 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.578015 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.581146 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.581171 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 20 2013-04-25 18:37:55.581177 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 20 >= 20 on 0x7fca2c83b0d0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.581185 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.581194 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.581202 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.581209 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.589974 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.590008 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.590016 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.590816 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798005 0x7fca34014ea0 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_780 ) ) v1 2013-04-25 18:37:55.590834 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34014ea0 prio 196 2013-04-25 18:37:55.590843 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.590865 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.590882 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798005 2013-04-25 18:37:55.590899 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.590906 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.591164 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.591187 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34013ac0 2013-04-25 18:37:55.591312 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798005 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_780 ) ) v1 ==== 1037209+0+0 (4221058601 0 0) 0x7fca34014ea0 con 0x241ae80 2013-04-25 18:37:55.591335 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.591344 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.591358 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.591367 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_780 ) ) v1 2013-04-25 18:37:55.591373 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_780 ) ) v1 2013-04-25 18:37:55.591419 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c524120 2013-04-25 18:37:55.591434 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.591504 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.591523 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 21 0x7fca2c524120 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.591564 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 21): sig = 16547433658134173270 2013-04-25 18:37:55.591576 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 21 0x7fca2c524120 2013-04-25 18:37:55.591607 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.591616 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.594933 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.594963 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 21 2013-04-25 18:37:55.594970 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 21 >= 21 on 0x7fca2c524120 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.594986 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.594994 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.595000 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.595007 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.602539 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.602572 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34014ea0 2013-04-25 18:37:55.603723 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.603749 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.603760 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.604575 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798006 0x7fca34015d80 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_809 ) ) v1 2013-04-25 18:37:55.604589 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34015d80 prio 196 2013-04-25 18:37:55.604599 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.604617 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798006 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_809 ) ) v1 ==== 1037209+0+0 (1753385100 0 0) 0x7fca34015d80 con 0x241ae80 2013-04-25 18:37:55.604637 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.604639 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.604646 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.604656 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798006 2013-04-25 18:37:55.604659 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.604665 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_809 ) ) v1 2013-04-25 18:37:55.604670 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_809 ) ) v1 2013-04-25 18:37:55.604670 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.604678 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.604718 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c524cd0 2013-04-25 18:37:55.604731 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.604801 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.604820 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 22 0x7fca2c524cd0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.604864 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 22): sig = 9092408283627831328 2013-04-25 18:37:55.604875 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 22 0x7fca2c524cd0 2013-04-25 18:37:55.604898 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.604904 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.608247 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.608273 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 22 2013-04-25 18:37:55.608279 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 22 >= 22 on 0x7fca2c524cd0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.608291 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.608297 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.608304 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.608338 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.617019 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.617042 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.617051 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.617838 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798007 0x7fca34017160 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_838 ) ) v1 2013-04-25 18:37:55.617852 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34017160 prio 196 2013-04-25 18:37:55.617863 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.617887 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.617903 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798007 2013-04-25 18:37:55.617919 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.617927 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.618090 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.618112 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34015d80 2013-04-25 18:37:55.618226 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798007 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_838 ) ) v1 ==== 1037209+0+0 (3765719183 0 0) 0x7fca34017160 con 0x241ae80 2013-04-25 18:37:55.618246 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.618254 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.618267 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.618271 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_838 ) ) v1 2013-04-25 18:37:55.618277 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_838 ) ) v1 2013-04-25 18:37:55.618322 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c61e4c0 2013-04-25 18:37:55.618338 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.618397 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.618417 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 23 0x7fca2c61e4c0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.618451 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 23): sig = 2038678363268954533 2013-04-25 18:37:55.618464 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 23 0x7fca2c61e4c0 2013-04-25 18:37:55.618488 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.618494 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.621640 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.621664 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 23 2013-04-25 18:37:55.621670 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 23 >= 23 on 0x7fca2c61e4c0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.621681 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.621687 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.621694 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.621700 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.629819 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.629852 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34017160 2013-04-25 18:37:55.630410 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.630429 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.630434 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.631229 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798008 0x7fca34018110 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_867 ) ) v1 2013-04-25 18:37:55.631243 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34018110 prio 196 2013-04-25 18:37:55.631256 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.631267 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798008 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_867 ) ) v1 ==== 1037209+0+0 (2918096651 0 0) 0x7fca34018110 con 0x241ae80 2013-04-25 18:37:55.631288 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.631282 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.631299 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798008 2013-04-25 18:37:55.631296 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.631309 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.631314 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_867 ) ) v1 2013-04-25 18:37:55.631314 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.631320 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_867 ) ) v1 2013-04-25 18:37:55.631322 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.631368 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c2edbc0 2013-04-25 18:37:55.631382 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.631471 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.631491 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 24 0x7fca2c2edbc0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.631544 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 24): sig = 17246893973208170543 2013-04-25 18:37:55.631556 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 24 0x7fca2c2edbc0 2013-04-25 18:37:55.631584 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.631595 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.634809 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.634838 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 24 2013-04-25 18:37:55.634846 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 24 >= 24 on 0x7fca2c2edbc0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.634864 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.634874 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.634883 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.634892 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.643568 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.643597 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.643602 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.644391 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798009 0x7fca340194f0 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_896 ) ) v1 2013-04-25 18:37:55.644406 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca340194f0 prio 196 2013-04-25 18:37:55.644415 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.644470 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.644492 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798009 2013-04-25 18:37:55.644507 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.644515 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.645908 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.645935 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34018110 2013-04-25 18:37:55.646057 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798009 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_896 ) ) v1 ==== 1037209+0+0 (449723834 0 0) 0x7fca340194f0 con 0x241ae80 2013-04-25 18:37:55.646078 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.646086 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.646099 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.646103 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_896 ) ) v1 2013-04-25 18:37:55.646109 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_896 ) ) v1 2013-04-25 18:37:55.646155 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c2ee870 2013-04-25 18:37:55.646167 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.646235 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.646255 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 25 0x7fca2c2ee870 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.646297 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 25): sig = 1723024637482228308 2013-04-25 18:37:55.646308 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 25 0x7fca2c2ee870 2013-04-25 18:37:55.646337 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.646346 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.649550 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.649577 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 25 2013-04-25 18:37:55.649583 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 25 >= 25 on 0x7fca2c2ee870 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.649594 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.649601 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.649607 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.649614 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.658323 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.658363 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.658373 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.659200 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798010 0x7fca3401a610 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_925 ) ) v1 2013-04-25 18:37:55.659216 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3401a610 prio 196 2013-04-25 18:37:55.659224 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.659248 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.659269 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798010 2013-04-25 18:37:55.659291 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.659301 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.660356 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.660376 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca340194f0 2013-04-25 18:37:55.660396 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798010 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_925 ) ) v1 ==== 1037209+0+0 (3508335832 0 0) 0x7fca3401a610 con 0x241ae80 2013-04-25 18:37:55.660412 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.660420 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.660434 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.660439 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_925 ) ) v1 2013-04-25 18:37:55.660445 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_925 ) ) v1 2013-04-25 18:37:55.660484 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c25eef0 2013-04-25 18:37:55.660493 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.660511 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.660528 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 26 0x7fca2c25eef0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.660576 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 26): sig = 2042205131621493086 2013-04-25 18:37:55.660593 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 26 0x7fca2c25eef0 2013-04-25 18:37:55.660671 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.660683 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.663932 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.663956 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 26 2013-04-25 18:37:55.663963 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 26 >= 26 on 0x7fca2c25eef0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.663974 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.663980 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.663986 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.663992 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.672745 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.672778 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.672785 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.673571 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798011 0x7fca3401b6b0 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_954 ) ) v1 2013-04-25 18:37:55.673593 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3401b6b0 prio 196 2013-04-25 18:37:55.673602 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.673621 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.673641 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798011 2013-04-25 18:37:55.673663 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.673673 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.676132 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.676158 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3401a610 2013-04-25 18:37:55.676244 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798011 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_954 ) ) v1 ==== 1037209+0+0 (2188051951 0 0) 0x7fca3401b6b0 con 0x241ae80 2013-04-25 18:37:55.676260 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.676267 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.676276 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.676279 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_954 ) ) v1 2013-04-25 18:37:55.676283 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_954 ) ) v1 2013-04-25 18:37:55.676318 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c2d8de0 2013-04-25 18:37:55.676327 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.676346 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.676366 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 27 0x7fca2c2d8de0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.676635 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 27): sig = 14316366032445732584 2013-04-25 18:37:55.676650 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 27 0x7fca2c2d8de0 2013-04-25 18:37:55.676684 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.676697 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.679857 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.679899 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 27 2013-04-25 18:37:55.679910 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 27 >= 27 on 0x7fca2c2d8de0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.679923 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.679932 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.679943 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1037209 data=0 off 0 2013-04-25 18:37:55.679954 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1037209 from dispatch throttler 1037209/104857600 2013-04-25 18:37:55.688677 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1037209 2013-04-25 18:37:55.688717 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.688722 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1037209 + 0 + 0 byte message 2013-04-25 18:37:55.689485 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798012 0x7fca3401c8a0 mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_983 ) ) v1 2013-04-25 18:37:55.689499 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3401c8a0 prio 196 2013-04-25 18:37:55.689509 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.689515 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.689534 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798012 2013-04-25 18:37:55.689552 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.689564 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.696474 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 2074418/104857600 2013-04-25 18:37:55.696494 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3401b6b0 2013-04-25 18:37:55.696503 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798012 ==== mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_983 ) ) v1 ==== 1037209+0+0 (1240722895 0 0) 0x7fca3401c8a0 con 0x241ae80 2013-04-25 18:37:55.696514 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.696519 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.696531 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.696536 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_983 ) ) v1 2013-04-25 18:37:55.696542 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1037021 bytes last_key ( osdmap,full_983 ) ) v1 2013-04-25 18:37:55.696583 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c2d9a90 2013-04-25 18:37:55.696593 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.696609 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.696625 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 28 0x7fca2c2d9a90 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.696680 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 28): sig = 14606965054534644891 2013-04-25 18:37:55.696698 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 28 0x7fca2c2d9a90 2013-04-25 18:37:55.696729 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.696741 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.711529 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1037209 to dispatch throttler 1037209/104857600 2013-04-25 18:37:55.711552 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca3401c8a0 2013-04-25 18:37:55.754151 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.754180 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 28 2013-04-25 18:37:55.754189 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 28 >= 28 on 0x7fca2c2d9a90 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.754234 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.754273 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.754293 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=610556 data=0 off 0 2013-04-25 18:37:55.754301 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 610556 from dispatch throttler 0/104857600 2013-04-25 18:37:55.759279 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 610556 2013-04-25 18:37:55.759305 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.759314 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 610556 + 0 + 0 byte message 2013-04-25 18:37:55.759778 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798013 0x7fca340b29b0 mon_sync( chunk bl 610371 bytes last_key ( paxos,785675 ) ) v1 2013-04-25 18:37:55.759790 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca340b29b0 prio 196 2013-04-25 18:37:55.759799 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.759840 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.759865 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798013 2013-04-25 18:37:55.759891 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.759895 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798013 ==== mon_sync( chunk bl 610371 bytes last_key ( paxos,785675 ) ) v1 ==== 610556+0+0 (4190350905 0 0) 0x7fca340b29b0 con 0x241ae80 2013-04-25 18:37:55.759907 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.759917 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.759926 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.759940 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.759945 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 610371 bytes last_key ( paxos,785675 ) ) v1 2013-04-25 18:37:55.759951 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 610371 bytes last_key ( paxos,785675 ) ) v1 2013-04-25 18:37:55.759991 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c260bd0 2013-04-25 18:37:55.759999 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.760014 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.760026 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 29 0x7fca2c260bd0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.760061 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 29): sig = 18206254909816645690 2013-04-25 18:37:55.760076 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 29 0x7fca2c260bd0 2013-04-25 18:37:55.760102 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.760116 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.771100 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 610556 to dispatch throttler 610556/104857600 2013-04-25 18:37:55.771120 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca340b29b0 2013-04-25 18:37:55.775454 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.775482 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 29 2013-04-25 18:37:55.775491 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 29 >= 29 on 0x7fca2c260bd0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.775537 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.775580 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.775596 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=11014338 data=0 off 0 2013-04-25 18:37:55.775610 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 11014338 from dispatch throttler 0/104857600 2013-04-25 18:37:55.869121 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 11014338 2013-04-25 18:37:55.869175 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.869182 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 11014338 + 0 + 0 byte message 2013-04-25 18:37:55.876872 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798014 0x7fca340b3b70 mon_sync( chunk bl 11014153 bytes last_key ( paxos,785676 ) ) v1 2013-04-25 18:37:55.876905 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca340b3b70 prio 196 2013-04-25 18:37:55.876917 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.876943 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.876969 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798014 2013-04-25 18:37:55.876992 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.876994 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798014 ==== mon_sync( chunk bl 11014153 bytes last_key ( paxos,785676 ) ) v1 ==== 11014338+0+0 (1326898816 0 0) 0x7fca340b3b70 con 0x241ae80 2013-04-25 18:37:55.877004 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.877015 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.877024 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.877035 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.877039 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 11014153 bytes last_key ( paxos,785676 ) ) v1 2013-04-25 18:37:55.877043 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 11014153 bytes last_key ( paxos,785676 ) ) v1 2013-04-25 18:37:55.877060 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2d0df320 2013-04-25 18:37:55.877068 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.877084 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.877098 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 30 0x7fca2d0df320 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.877172 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 30): sig = 7587103233108578292 2013-04-25 18:37:55.877191 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 30 0x7fca2d0df320 2013-04-25 18:37:55.877239 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.877250 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.889237 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.889287 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 30 2013-04-25 18:37:55.889298 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 30 >= 30 on 0x7fca2d0df320 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.889311 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.889317 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.889325 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=345027 data=0 off 0 2013-04-25 18:37:55.889333 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 345027 from dispatch throttler 11014338/104857600 2013-04-25 18:37:55.892271 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 345027 2013-04-25 18:37:55.892291 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:55.892299 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 345027 + 0 + 0 byte message 2013-04-25 18:37:55.892629 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798015 0x7fca34071c70 mon_sync( chunk bl 344842 bytes last_key ( paxos,785719 ) ) v1 2013-04-25 18:37:55.892642 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34071c70 prio 196 2013-04-25 18:37:55.892654 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.892679 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.892700 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798015 2013-04-25 18:37:55.892726 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.892738 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.963430 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 11014338 to dispatch throttler 11359365/104857600 2013-04-25 18:37:55.963478 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca340b3b70 2013-04-25 18:37:55.964247 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798015 ==== mon_sync( chunk bl 344842 bytes last_key ( paxos,785719 ) ) v1 ==== 345027+0+0 (3599457543 0 0) 0x7fca34071c70 con 0x241ae80 2013-04-25 18:37:55.964266 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:55.964275 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:55.964283 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:55.964286 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 344842 bytes last_key ( paxos,785719 ) ) v1 2013-04-25 18:37:55.964289 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 344842 bytes last_key ( paxos,785719 ) ) v1 2013-04-25 18:37:55.964375 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2d0dfdd0 2013-04-25 18:37:55.964383 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:55.964406 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.964429 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 31 0x7fca2d0dfdd0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.964525 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 31): sig = 14315003383455782544 2013-04-25 18:37:55.964540 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 31 0x7fca2d0dfdd0 2013-04-25 18:37:55.964591 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:55.964603 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:55.975121 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 345027 to dispatch throttler 345027/104857600 2013-04-25 18:37:55.975136 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34071c70 2013-04-25 18:37:55.979837 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:55.979914 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 31 2013-04-25 18:37:55.979923 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 31 >= 31 on 0x7fca2d0dfdd0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:55.979935 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:55.979942 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:55.979948 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=11014326 data=0 off 0 2013-04-25 18:37:55.979954 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 11014326 from dispatch throttler 0/104857600 2013-04-25 18:37:56.073479 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 11014326 2013-04-25 18:37:56.073528 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:56.073534 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 11014326 + 0 + 0 byte message 2013-04-25 18:37:56.081271 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798016 0x7fca34072bd0 mon_sync( chunk bl 11014141 bytes last_key ( paxos,785720 ) ) v1 2013-04-25 18:37:56.081300 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34072bd0 prio 196 2013-04-25 18:37:56.081311 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.081337 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.081362 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798016 2013-04-25 18:37:56.081383 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.081395 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.081388 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798016 ==== mon_sync( chunk bl 11014141 bytes last_key ( paxos,785720 ) ) v1 ==== 11014326+0+0 (3464416974 0 0) 0x7fca34072bd0 con 0x241ae80 2013-04-25 18:37:56.081408 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:56.081417 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:56.081428 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:56.081433 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 11014141 bytes last_key ( paxos,785720 ) ) v1 2013-04-25 18:37:56.081438 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 11014141 bytes last_key ( paxos,785720 ) ) v1 2013-04-25 18:37:56.081454 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c2ec730 2013-04-25 18:37:56.081463 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:56.081478 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.081493 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 32 0x7fca2c2ec730 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.081533 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 32): sig = 6805001392852929442 2013-04-25 18:37:56.081546 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 32 0x7fca2c2ec730 2013-04-25 18:37:56.081584 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.081596 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.093478 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:56.093530 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 32 2013-04-25 18:37:56.093538 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 32 >= 32 on 0x7fca2c2ec730 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.093583 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.093592 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:56.093600 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=348027 data=0 off 0 2013-04-25 18:37:56.093608 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 348027 from dispatch throttler 11014326/104857600 2013-04-25 18:37:56.096502 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 348027 2013-04-25 18:37:56.096529 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:56.096537 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 348027 + 0 + 0 byte message 2013-04-25 18:37:56.096856 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798017 0x7fca34073fb0 mon_sync( chunk bl 347842 bytes last_key ( paxos,785763 ) ) v1 2013-04-25 18:37:56.096870 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34073fb0 prio 196 2013-04-25 18:37:56.096881 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.096910 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.096931 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798017 2013-04-25 18:37:56.096956 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.096968 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.158534 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 11014326 to dispatch throttler 11362353/104857600 2013-04-25 18:37:56.158582 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34072bd0 2013-04-25 18:37:56.159357 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798017 ==== mon_sync( chunk bl 347842 bytes last_key ( paxos,785763 ) ) v1 ==== 348027+0+0 (1090202902 0 0) 0x7fca34073fb0 con 0x241ae80 2013-04-25 18:37:56.159377 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:56.159384 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:56.159393 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:56.159395 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 347842 bytes last_key ( paxos,785763 ) ) v1 2013-04-25 18:37:56.159399 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 347842 bytes last_key ( paxos,785763 ) ) v1 2013-04-25 18:37:56.159476 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2ca8af80 2013-04-25 18:37:56.159483 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:56.159502 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.159523 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 33 0x7fca2ca8af80 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.159705 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 33): sig = 16338646321290210180 2013-04-25 18:37:56.159722 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 33 0x7fca2ca8af80 2013-04-25 18:37:56.159769 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.159780 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.168073 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 348027 to dispatch throttler 348027/104857600 2013-04-25 18:37:56.168088 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34073fb0 2013-04-25 18:37:56.176452 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:56.176481 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 33 2013-04-25 18:37:56.176490 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 33 >= 33 on 0x7fca2ca8af80 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.176506 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.176551 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:56.176571 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=11013509 data=0 off 0 2013-04-25 18:37:56.176585 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 11013509 from dispatch throttler 0/104857600 2013-04-25 18:37:56.270063 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 11013509 2013-04-25 18:37:56.270120 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:56.270134 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 11013509 + 0 + 0 byte message 2013-04-25 18:37:56.277871 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798018 0x7fca34074e90 mon_sync( chunk bl 11013324 bytes last_key ( paxos,785764 ) ) v1 2013-04-25 18:37:56.277907 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34074e90 prio 196 2013-04-25 18:37:56.277919 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.277943 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.277968 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798018 2013-04-25 18:37:56.277989 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.277987 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798018 ==== mon_sync( chunk bl 11013324 bytes last_key ( paxos,785764 ) ) v1 ==== 11013509+0+0 (2895450773 0 0) 0x7fca34074e90 con 0x241ae80 2013-04-25 18:37:56.278001 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.278008 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:56.278017 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:56.278028 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:56.278033 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 11013324 bytes last_key ( paxos,785764 ) ) v1 2013-04-25 18:37:56.278037 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 11013324 bytes last_key ( paxos,785764 ) ) v1 2013-04-25 18:37:56.278054 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c74bf10 2013-04-25 18:37:56.278062 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:56.278077 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.278091 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 34 0x7fca2c74bf10 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.278152 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 34): sig = 5865529561451539391 2013-04-25 18:37:56.278165 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 34 0x7fca2c74bf10 2013-04-25 18:37:56.278196 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.278210 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.351548 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:56.351655 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 34 2013-04-25 18:37:56.351665 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 34 >= 34 on 0x7fca2c74bf10 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.351677 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.351684 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:56.351695 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=1045906 data=0 off 0 2013-04-25 18:37:56.351703 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 1045906 from dispatch throttler 11013509/104857600 2013-04-25 18:37:56.360484 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 1045906 2013-04-25 18:37:56.360532 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:56.360537 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 1045906 + 0 + 0 byte message 2013-04-25 18:37:56.361337 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798019 0x7fca34076270 mon_sync( chunk bl 1045721 bytes last_key ( pgmap,407502 ) ) v1 2013-04-25 18:37:56.361352 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34076270 prio 196 2013-04-25 18:37:56.361364 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.361369 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.361389 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798019 2013-04-25 18:37:56.361412 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.361424 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.362941 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 11013509 to dispatch throttler 12059415/104857600 2013-04-25 18:37:56.362962 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34074e90 2013-04-25 18:37:56.362972 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798019 ==== mon_sync( chunk bl 1045721 bytes last_key ( pgmap,407502 ) ) v1 ==== 1045906+0+0 (4004922128 0 0) 0x7fca34076270 con 0x241ae80 2013-04-25 18:37:56.362988 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:56.362995 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:56.363003 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:56.363006 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1045721 bytes last_key ( pgmap,407502 ) ) v1 2013-04-25 18:37:56.363010 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1045721 bytes last_key ( pgmap,407502 ) ) v1 2013-04-25 18:37:56.363317 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c520010 2013-04-25 18:37:56.363332 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:56.363362 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.363379 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 35 0x7fca2c520010 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.363427 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 35): sig = 9034809911124782205 2013-04-25 18:37:56.363444 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 35 0x7fca2c520010 2013-04-25 18:37:56.363487 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.363498 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.377947 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 1045906 to dispatch throttler 1045906/104857600 2013-04-25 18:37:56.377968 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34076270 2013-04-25 18:37:56.453478 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:56.453535 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 35 2013-04-25 18:37:56.453545 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 35 >= 35 on 0x7fca2c520010 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.453554 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.453560 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:56.453568 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=481092 data=0 off 0 2013-04-25 18:37:56.453574 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 481092 from dispatch throttler 0/104857600 2013-04-25 18:37:56.457460 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 481092 2013-04-25 18:37:56.457491 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:56.457505 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 481092 + 0 + 0 byte message 2013-04-25 18:37:56.457920 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798020 0x7fca34077150 mon_sync( chunk bl 480907 bytes last_key ( pgmap,full_0 ) ) v1 2013-04-25 18:37:56.457933 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34077150 prio 196 2013-04-25 18:37:56.457943 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.457980 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.458005 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798020 2013-04-25 18:37:56.458030 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.458035 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798020 ==== mon_sync( chunk bl 480907 bytes last_key ( pgmap,full_0 ) ) v1 ==== 481092+0+0 (3228208858 0 0) 0x7fca34077150 con 0x241ae80 2013-04-25 18:37:56.458047 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.458057 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:56.458067 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:56.458079 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:56.458084 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 480907 bytes last_key ( pgmap,full_0 ) ) v1 2013-04-25 18:37:56.458090 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 480907 bytes last_key ( pgmap,full_0 ) ) v1 2013-04-25 18:37:56.458220 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2c520730 2013-04-25 18:37:56.458232 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:56.458321 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.458339 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 36 0x7fca2c520730 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.458405 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 36): sig = 16523432575391387593 2013-04-25 18:37:56.458426 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 36 0x7fca2c520730 2013-04-25 18:37:56.458453 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.458466 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.467695 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 481092 to dispatch throttler 481092/104857600 2013-04-25 18:37:56.467712 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34077150 2013-04-25 18:37:56.480233 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:56.480260 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 36 2013-04-25 18:37:56.480269 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 36 >= 36 on 0x7fca2c520730 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.480318 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.480359 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:56.480397 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=11012309 data=0 off 0 2013-04-25 18:37:56.480405 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 11012309 from dispatch throttler 0/104857600 2013-04-25 18:37:56.573766 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 11012309 2013-04-25 18:37:56.573818 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:56.573823 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 11012309 + 0 + 0 byte message 2013-04-25 18:37:56.581513 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798021 0x7fca34078240 mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407177 ) ) v1 2013-04-25 18:37:56.581546 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca34078240 prio 196 2013-04-25 18:37:56.581558 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.581563 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.581588 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798021 2013-04-25 18:37:56.581611 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.581623 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.581618 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798021 ==== mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407177 ) ) v1 ==== 11012309+0+0 (110494865 0 0) 0x7fca34078240 con 0x241ae80 2013-04-25 18:37:56.581639 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:56.581648 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:56.581659 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:56.581664 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407177 ) ) v1 2013-04-25 18:37:56.581668 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407177 ) ) v1 2013-04-25 18:37:56.581683 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2dbed3c0 2013-04-25 18:37:56.581692 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:56.581707 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.581722 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 37 0x7fca2dbed3c0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.581762 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 37): sig = 8566291727192614947 2013-04-25 18:37:56.581775 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 37 0x7fca2dbed3c0 2013-04-25 18:37:56.581822 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.581837 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.602513 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:56.602590 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 37 2013-04-25 18:37:56.602606 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 37 >= 37 on 0x7fca2dbed3c0 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.602619 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.602626 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:56.602634 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=11012309 data=0 off 0 2013-04-25 18:37:56.602642 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 11012309 from dispatch throttler 11012309/104857600 2013-04-25 18:37:56.665259 7fccc04d5700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 11012309 to dispatch throttler 22024618/104857600 2013-04-25 18:37:56.665299 7fccc04d5700 20 -- 10.1.0.3:6789/0 done calling dispatch on 0x7fca34078240 2013-04-25 18:37:56.696003 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 11012309 2013-04-25 18:37:56.696046 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:56.696054 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 11012309 + 0 + 0 byte message 2013-04-25 18:37:56.703791 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798022 0x7fca340793e0 mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407203 ) ) v1 2013-04-25 18:37:56.703824 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca340793e0 prio 196 2013-04-25 18:37:56.703837 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.703862 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.703885 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798022 2013-04-25 18:37:56.703907 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.703908 7fccc04d5700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1192798022 ==== mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407203 ) ) v1 ==== 11012309+0+0 (584991991 0 0) 0x7fca340793e0 con 0x241ae80 2013-04-25 18:37:56.703928 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.703941 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection 2013-04-25 18:37:56.703950 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 2013-04-25 18:37:56.703961 7fccc04d5700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * 2013-04-25 18:37:56.703966 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407203 ) ) v1 2013-04-25 18:37:56.703970 7fccc04d5700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407203 ) ) v1 2013-04-25 18:37:56.703987 7fccc04d5700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x7fca2dbede40 2013-04-25 18:37:56.703997 7fccc04d5700 20 -- 10.1.0.3:6789/0 submit_message mon_sync( chunk_reply ) v1 remote, 10.1.0.130:6789/0, have pipe. 2013-04-25 18:37:56.704014 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.704028 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer encoding 38 0x7fca2dbede40 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.704091 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer signed seq # 38): sig = 13499848800555915934 2013-04-25 18:37:56.704104 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sending 38 0x7fca2dbede40 2013-04-25 18:37:56.704151 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.704162 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:37:56.727640 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ACK 2013-04-25 18:37:56.727694 7fccbf2d1700 15 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 38 2013-04-25 18:37:56.727703 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got ack seq 38 >= 38 on 0x7fca2dbede40 mon_sync( chunk_reply ) v1 2013-04-25 18:37:56.727716 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.727722 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got MSG 2013-04-25 18:37:56.727729 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got envelope type=69 src mon.2 front=11012309 data=0 off 0 2013-04-25 18:37:56.727737 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader wants 11012309 from dispatch throttler 11012309/104857600 2013-04-25 18:37:56.821216 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got front 11012309 2013-04-25 18:37:56.821274 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).aborted = 0 2013-04-25 18:37:56.821284 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got 11012309 + 0 + 0 byte message 2013-04-25 18:37:56.829095 7fccbf2d1700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader got message 1192798023 0x7fca3407a7c0 mon_sync( chunk bl 11012119 bytes last_key ( pgmap,full_407229 ) ) v1 2013-04-25 18:37:56.829128 7fccbf2d1700 20 -- 10.1.0.3:6789/0 queue 0x7fca3407a7c0 prio 196 2013-04-25 18:37:56.829138 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader reading tag... 2013-04-25 18:37:56.829161 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.829186 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).write_ack 1192798023 2013-04-25 18:37:56.829215 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer: state = open policy.server=0 2013-04-25 18:37:56.829231 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:38:00.873664 7fccbfcd4700 20 accepter.accepter poll got 1 2013-04-25 18:38:00.873697 7fccbfcd4700 10 accepter.pfd.revents=1 2013-04-25 18:38:00.873718 7fccbfcd4700 10 accepter.accepted incoming on sd 24 2013-04-25 18:38:00.873785 7fccbfcd4700 20 accepter.accepter calling poll 2013-04-25 18:38:00.873867 7fccc8402700 10 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50000cd0 sd=24 :0 s=0 pgs=0 cs=0 l=0).accept 2013-04-25 18:38:00.874016 7fccc8402700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50000cd0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.1.255.22:47585/0 2013-04-25 18:38:00.874961 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is 10.1.255.22:0/1017455 2013-04-25 18:38:00.874998 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept got peer connect_seq 0 global_seq 1 2013-04-25 18:38:00.875008 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept of host_type 8, policy.lossy=1 2013-04-25 18:38:00.875016 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept my proto 15, their proto 15 2013-04-25 18:38:00.875025 7fccc8402700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.1.255.22:0/1017455 client protocol 0 2013-04-25 18:38:00.875036 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept: setting up session_security. 2013-04-25 18:38:00.875043 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept new session 2013-04-25 18:38:00.875052 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept success, connect_seq = 1, sending READY 2013-04-25 18:38:00.875059 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept features 18446744073709551615 2013-04-25 18:38:00.875075 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).register_pipe 2013-04-25 18:38:00.875091 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).discard_requeued_up_to 0 2013-04-25 18:38:00.875098 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept starting writer, state open 2013-04-25 18:38:00.875135 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept done 2013-04-25 18:38:00.875153 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:38:00.875201 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:38:00.875247 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:38:00.875256 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:38:00.876608 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got MSG 2013-04-25 18:38:00.876636 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got envelope type=17 src client.? front=62 data=0 off 0 2013-04-25 18:38:00.876656 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 bytes from policy throttler 0/104857600 2013-04-25 18:38:00.876666 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 from dispatch throttler 22024618/104857600 2013-04-25 18:38:00.876678 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got front 62 2013-04-25 18:38:00.876687 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).aborted = 0 2013-04-25 18:38:00.876694 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got 62 + 0 + 0 byte message 2013-04-25 18:38:00.876720 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got message 1 0x7fca20001a80 auth(proto 0 32 bytes epoch 0) v1 2013-04-25 18:38:00.876781 7fccc8402700 20 -- 10.1.0.3:6789/0 queue 0x7fca20001a80 prio 127 2013-04-25 18:38:00.876787 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:38:00.876794 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:38:00.876815 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).write_ack 1 2013-04-25 18:38:00.876832 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:38:00.876852 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:38:03.873397 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success 2013-04-25 18:38:03.873465 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success 2013-04-25 18:38:03.873496 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault on lossy channel, failing 2013-04-25 18:38:03.873504 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).stop 2013-04-25 18:38:03.873514 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe 2013-04-25 18:38:03.873529 7fccc8402700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 62 to dispatch throttler 22024680/104857600 2013-04-25 18:38:03.873548 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:38:03.873565 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).reader done 2013-04-25 18:38:03.873579 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer finishing 2013-04-25 18:38:03.873612 7fccc8301700 10 -- 10.1.0.3:6789/0 queue_reap 0x7fca50000cd0 2013-04-25 18:38:03.873624 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer done 2013-04-25 18:38:03.873707 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper 2013-04-25 18:38:03.873732 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaping pipe 0x7fca50000cd0 10.1.255.22:0/1017455 2013-04-25 18:38:03.873786 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:38:03.873796 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1017455 pipe(0x7fca50000cd0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe - not registered 2013-04-25 18:38:03.873818 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaped pipe 0x7fca50000cd0 10.1.255.22:0/1017455 2013-04-25 18:38:03.873833 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper deleted pipe 0x7fca50000cd0 2013-04-25 18:38:03.873835 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper done 2013-04-25 18:44:07.219375 7fccbfcd4700 20 accepter.accepter poll got 1 2013-04-25 18:44:07.219395 7fccbfcd4700 10 accepter.pfd.revents=1 2013-04-25 18:44:07.219406 7fccbfcd4700 10 accepter.accepted incoming on sd 24 2013-04-25 18:44:07.219469 7fccbfcd4700 20 accepter.accepter calling poll 2013-04-25 18:44:07.219545 7fccc8402700 10 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca500013d0 sd=24 :0 s=0 pgs=0 cs=0 l=0).accept 2013-04-25 18:44:07.219622 7fccc8402700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca500013d0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.1.255.22:48566/0 2013-04-25 18:44:07.220326 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is 10.1.255.22:0/1018736 2013-04-25 18:44:07.220353 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept got peer connect_seq 0 global_seq 1 2013-04-25 18:44:07.220425 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept of host_type 8, policy.lossy=1 2013-04-25 18:44:07.220435 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept my proto 15, their proto 15 2013-04-25 18:44:07.220441 7fccc8402700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.1.255.22:0/1018736 client protocol 0 2013-04-25 18:44:07.220449 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept: setting up session_security. 2013-04-25 18:44:07.220454 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept new session 2013-04-25 18:44:07.220458 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept success, connect_seq = 1, sending READY 2013-04-25 18:44:07.220462 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept features 18446744073709551615 2013-04-25 18:44:07.220469 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).register_pipe 2013-04-25 18:44:07.220484 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).discard_requeued_up_to 0 2013-04-25 18:44:07.220491 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept starting writer, state open 2013-04-25 18:44:07.220511 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept done 2013-04-25 18:44:07.220520 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:44:07.220602 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:44:07.220628 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:44:07.220639 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:44:07.220992 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got MSG 2013-04-25 18:44:07.221019 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got envelope type=17 src client.? front=62 data=0 off 0 2013-04-25 18:44:07.221032 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 bytes from policy throttler 0/104857600 2013-04-25 18:44:07.221041 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 from dispatch throttler 22024618/104857600 2013-04-25 18:44:07.221052 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got front 62 2013-04-25 18:44:07.221061 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).aborted = 0 2013-04-25 18:44:07.221068 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got 62 + 0 + 0 byte message 2013-04-25 18:44:07.221086 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got message 1 0x7fca24002a60 auth(proto 0 32 bytes epoch 0) v1 2013-04-25 18:44:07.221098 7fccc8402700 20 -- 10.1.0.3:6789/0 queue 0x7fca24002a60 prio 127 2013-04-25 18:44:07.221104 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:44:07.221182 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:44:07.221211 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).write_ack 1 2013-04-25 18:44:07.221236 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:44:07.221246 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:44:10.218886 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success 2013-04-25 18:44:10.218927 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success 2013-04-25 18:44:10.219007 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault on lossy channel, failing 2013-04-25 18:44:10.219016 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).stop 2013-04-25 18:44:10.219025 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe 2013-04-25 18:44:10.219038 7fccc8402700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 62 to dispatch throttler 22024680/104857600 2013-04-25 18:44:10.219049 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:44:10.219061 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).reader done 2013-04-25 18:44:10.219103 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer finishing 2013-04-25 18:44:10.219122 7fccc8301700 10 -- 10.1.0.3:6789/0 queue_reap 0x7fca500013d0 2013-04-25 18:44:10.219132 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer done 2013-04-25 18:44:10.219189 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper 2013-04-25 18:44:10.219203 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaping pipe 0x7fca500013d0 10.1.255.22:0/1018736 2013-04-25 18:44:10.219209 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:44:10.219256 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1018736 pipe(0x7fca500013d0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe - not registered 2013-04-25 18:44:10.219279 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaped pipe 0x7fca500013d0 10.1.255.22:0/1018736 2013-04-25 18:44:10.219290 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper deleted pipe 0x7fca500013d0 2013-04-25 18:44:10.219293 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper done 2013-04-25 18:47:10.409013 7fccbfcd4700 20 accepter.accepter poll got 1 2013-04-25 18:47:10.409040 7fccbfcd4700 10 accepter.pfd.revents=1 2013-04-25 18:47:10.409050 7fccbfcd4700 10 accepter.accepted incoming on sd 24 2013-04-25 18:47:10.409092 7fccbfcd4700 20 accepter.accepter calling poll 2013-04-25 18:47:10.409163 7fccc8402700 10 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50001920 sd=24 :0 s=0 pgs=0 cs=0 l=0).accept 2013-04-25 18:47:10.409240 7fccc8402700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50001920 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.1.255.22:49054/0 2013-04-25 18:47:10.409722 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is 10.1.255.22:0/1019817 2013-04-25 18:47:10.409751 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept got peer connect_seq 0 global_seq 1 2013-04-25 18:47:10.409761 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept of host_type 8, policy.lossy=1 2013-04-25 18:47:10.409768 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept my proto 15, their proto 15 2013-04-25 18:47:10.409776 7fccc8402700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.1.255.22:0/1019817 client protocol 0 2013-04-25 18:47:10.409790 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept: setting up session_security. 2013-04-25 18:47:10.409796 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept new session 2013-04-25 18:47:10.409803 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept success, connect_seq = 1, sending READY 2013-04-25 18:47:10.409810 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept features 18446744073709551615 2013-04-25 18:47:10.409820 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).register_pipe 2013-04-25 18:47:10.409842 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).discard_requeued_up_to 0 2013-04-25 18:47:10.409852 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept starting writer, state open 2013-04-25 18:47:10.409877 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept done 2013-04-25 18:47:10.409887 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:47:10.409972 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:47:10.410000 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:47:10.410010 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:47:10.410456 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got MSG 2013-04-25 18:47:10.410485 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got envelope type=17 src client.? front=62 data=0 off 0 2013-04-25 18:47:10.410498 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 bytes from policy throttler 0/104857600 2013-04-25 18:47:10.410509 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 from dispatch throttler 22024618/104857600 2013-04-25 18:47:10.410519 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got front 62 2013-04-25 18:47:10.410529 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).aborted = 0 2013-04-25 18:47:10.410536 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got 62 + 0 + 0 byte message 2013-04-25 18:47:10.410557 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got message 1 0x7fca20005a40 auth(proto 0 32 bytes epoch 0) v1 2013-04-25 18:47:10.410567 7fccc8402700 20 -- 10.1.0.3:6789/0 queue 0x7fca20005a40 prio 127 2013-04-25 18:47:10.410577 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:47:10.410629 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:47:10.410649 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).write_ack 1 2013-04-25 18:47:10.410665 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:47:10.410675 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:47:13.408808 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success 2013-04-25 18:47:13.408858 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success 2013-04-25 18:47:13.408886 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault on lossy channel, failing 2013-04-25 18:47:13.408921 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=2 pgs=1 cs=1 l=1).stop 2013-04-25 18:47:13.408931 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe 2013-04-25 18:47:13.408946 7fccc8402700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 62 to dispatch throttler 22024680/104857600 2013-04-25 18:47:13.408959 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:47:13.408974 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=4 pgs=1 cs=1 l=1).reader done 2013-04-25 18:47:13.409017 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer finishing 2013-04-25 18:47:13.409069 7fccc8301700 10 -- 10.1.0.3:6789/0 queue_reap 0x7fca50001920 2013-04-25 18:47:13.409082 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer done 2013-04-25 18:47:13.409193 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper 2013-04-25 18:47:13.409210 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaping pipe 0x7fca50001920 10.1.255.22:0/1019817 2013-04-25 18:47:13.409216 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:47:13.409224 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1019817 pipe(0x7fca50001920 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe - not registered 2013-04-25 18:47:13.409280 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaped pipe 0x7fca50001920 10.1.255.22:0/1019817 2013-04-25 18:47:13.409293 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper deleted pipe 0x7fca50001920 2013-04-25 18:47:13.409296 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper done 2013-04-25 18:52:55.098113 7fccbf1d0700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).reader couldn't read tag, Success 2013-04-25 18:52:55.098157 7fccbf1d0700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).fault 0: Success 2013-04-25 18:52:55.098190 7fccbf1d0700 0 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=2 pgs=3562 cs=1 l=0).fault with nothing to send, going to standby 2013-04-25 18:52:55.098199 7fccbf1d0700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=3 pgs=3562 cs=1 l=0).reader sleeping during reconnect|standby 2013-04-25 18:52:55.098275 7fccbf4d3700 10 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=3 pgs=3562 cs=1 l=0).writer: state = standby policy.server=0 2013-04-25 18:52:55.098295 7fccbf4d3700 20 -- 10.1.0.3:6789/0 >> 10.1.0.67:6789/0 pipe(0x23de670 sd=21 :53248 s=3 pgs=3562 cs=1 l=0).writer sleeping 2013-04-25 18:52:56.929307 7fccbf2d1700 2 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).reader couldn't read tag, Success 2013-04-25 18:52:56.929359 7fccbf2d1700 2 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).fault 0: Success 2013-04-25 18:52:56.929382 7fccbf2d1700 0 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=2 pgs=3736 cs=1 l=0).fault with nothing to send, going to standby 2013-04-25 18:52:56.929390 7fccbf2d1700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=3 pgs=3736 cs=1 l=0).reader sleeping during reconnect|standby 2013-04-25 18:52:56.929416 7fccbf3d2700 10 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=3 pgs=3736 cs=1 l=0).writer: state = standby policy.server=0 2013-04-25 18:52:56.929439 7fccbf3d2700 20 -- 10.1.0.3:6789/0 >> 10.1.0.130:6789/0 pipe(0x241ac20 sd=22 :48619 s=3 pgs=3736 cs=1 l=0).writer sleeping 2013-04-25 18:53:13.891336 7fccbfcd4700 20 accepter.accepter poll got 1 2013-04-25 18:53:13.891354 7fccbfcd4700 10 accepter.pfd.revents=1 2013-04-25 18:53:13.891366 7fccbfcd4700 10 accepter.accepted incoming on sd 24 2013-04-25 18:53:13.891445 7fccbfcd4700 20 accepter.accepter calling poll 2013-04-25 18:53:13.891529 7fccc8402700 10 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50001e70 sd=24 :0 s=0 pgs=0 cs=0 l=0).accept 2013-04-25 18:53:13.891625 7fccc8402700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50001e70 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.1.255.22:49879/0 2013-04-25 18:53:13.892307 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is 10.1.255.22:0/1021066 2013-04-25 18:53:13.892333 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept got peer connect_seq 0 global_seq 1 2013-04-25 18:53:13.892343 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept of host_type 8, policy.lossy=1 2013-04-25 18:53:13.892349 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept my proto 15, their proto 15 2013-04-25 18:53:13.892357 7fccc8402700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.1.255.22:0/1021066 client protocol 0 2013-04-25 18:53:13.892367 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept: setting up session_security. 2013-04-25 18:53:13.892413 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept new session 2013-04-25 18:53:13.892417 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept success, connect_seq = 1, sending READY 2013-04-25 18:53:13.892422 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept features 18446744073709551615 2013-04-25 18:53:13.892429 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).register_pipe 2013-04-25 18:53:13.892445 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).discard_requeued_up_to 0 2013-04-25 18:53:13.892451 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept starting writer, state open 2013-04-25 18:53:13.892471 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept done 2013-04-25 18:53:13.892479 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:53:13.892538 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:53:13.892559 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:53:13.892569 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:53:13.893076 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got MSG 2013-04-25 18:53:13.893104 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got envelope type=17 src client.? front=62 data=0 off 0 2013-04-25 18:53:13.893117 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 bytes from policy throttler 0/104857600 2013-04-25 18:53:13.893126 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 from dispatch throttler 22024618/104857600 2013-04-25 18:53:13.893136 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got front 62 2013-04-25 18:53:13.893146 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).aborted = 0 2013-04-25 18:53:13.893153 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got 62 + 0 + 0 byte message 2013-04-25 18:53:13.893171 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got message 1 0x7fca24006a50 auth(proto 0 32 bytes epoch 0) v1 2013-04-25 18:53:13.893184 7fccc8402700 20 -- 10.1.0.3:6789/0 queue 0x7fca24006a50 prio 127 2013-04-25 18:53:13.893190 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:53:13.893265 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:53:13.893286 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).write_ack 1 2013-04-25 18:53:13.893302 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:53:13.893312 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:53:16.890922 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success 2013-04-25 18:53:16.890964 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success 2013-04-25 18:53:16.890990 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault on lossy channel, failing 2013-04-25 18:53:16.890998 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=2 pgs=1 cs=1 l=1).stop 2013-04-25 18:53:16.891007 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe 2013-04-25 18:53:16.891021 7fccc8402700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 62 to dispatch throttler 22024680/104857600 2013-04-25 18:53:16.891039 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:53:16.891048 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=4 pgs=1 cs=1 l=1).reader done 2013-04-25 18:53:16.891054 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer finishing 2013-04-25 18:53:16.891079 7fccc8301700 10 -- 10.1.0.3:6789/0 queue_reap 0x7fca50001e70 2013-04-25 18:53:16.891095 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer done 2013-04-25 18:53:16.891103 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper 2013-04-25 18:53:16.891118 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaping pipe 0x7fca50001e70 10.1.255.22:0/1021066 2013-04-25 18:53:16.891127 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:53:16.891138 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021066 pipe(0x7fca50001e70 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe - not registered 2013-04-25 18:53:16.891163 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaped pipe 0x7fca50001e70 10.1.255.22:0/1021066 2013-04-25 18:53:16.891173 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper deleted pipe 0x7fca50001e70 2013-04-25 18:53:16.891177 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper done 2013-04-25 18:55:17.004443 7fccbfcd4700 20 accepter.accepter poll got 1 2013-04-25 18:55:17.004462 7fccbfcd4700 10 accepter.pfd.revents=1 2013-04-25 18:55:17.004472 7fccbfcd4700 10 accepter.accepted incoming on sd 24 2013-04-25 18:55:17.004541 7fccbfcd4700 20 accepter.accepter calling poll 2013-04-25 18:55:17.004610 7fccc8402700 10 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca500023c0 sd=24 :0 s=0 pgs=0 cs=0 l=0).accept 2013-04-25 18:55:17.004690 7fccc8402700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca500023c0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.1.255.22:50156/0 2013-04-25 18:55:17.005385 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is 10.1.255.22:0/1021534 2013-04-25 18:55:17.005411 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept got peer connect_seq 0 global_seq 1 2013-04-25 18:55:17.005421 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept of host_type 8, policy.lossy=1 2013-04-25 18:55:17.005427 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept my proto 15, their proto 15 2013-04-25 18:55:17.005435 7fccc8402700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.1.255.22:0/1021534 client protocol 0 2013-04-25 18:55:17.005445 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept: setting up session_security. 2013-04-25 18:55:17.005474 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept new session 2013-04-25 18:55:17.005478 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept success, connect_seq = 1, sending READY 2013-04-25 18:55:17.005483 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept features 18446744073709551615 2013-04-25 18:55:17.005490 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).register_pipe 2013-04-25 18:55:17.005504 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).discard_requeued_up_to 0 2013-04-25 18:55:17.005511 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept starting writer, state open 2013-04-25 18:55:17.005530 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept done 2013-04-25 18:55:17.005538 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:55:17.005595 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:55:17.005616 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:55:17.005626 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:55:17.006005 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got MSG 2013-04-25 18:55:17.006032 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got envelope type=17 src client.? front=62 data=0 off 0 2013-04-25 18:55:17.006045 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 bytes from policy throttler 0/104857600 2013-04-25 18:55:17.006054 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 from dispatch throttler 22024618/104857600 2013-04-25 18:55:17.006064 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got front 62 2013-04-25 18:55:17.006074 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).aborted = 0 2013-04-25 18:55:17.006081 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got 62 + 0 + 0 byte message 2013-04-25 18:55:17.006102 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got message 1 0x7fca20009a60 auth(proto 0 32 bytes epoch 0) v1 2013-04-25 18:55:17.006109 7fccc8402700 20 -- 10.1.0.3:6789/0 queue 0x7fca20009a60 prio 127 2013-04-25 18:55:17.006115 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:55:17.006174 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:55:17.006193 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).write_ack 1 2013-04-25 18:55:17.006209 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:55:17.006220 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:55:20.003769 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success 2013-04-25 18:55:20.003812 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success 2013-04-25 18:55:20.003840 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault on lossy channel, failing 2013-04-25 18:55:20.003848 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=2 pgs=1 cs=1 l=1).stop 2013-04-25 18:55:20.003858 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe 2013-04-25 18:55:20.003870 7fccc8402700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 62 to dispatch throttler 22024680/104857600 2013-04-25 18:55:20.003881 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:55:20.003892 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).reader done 2013-04-25 18:55:20.003906 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer finishing 2013-04-25 18:55:20.003933 7fccc8301700 10 -- 10.1.0.3:6789/0 queue_reap 0x7fca500023c0 2013-04-25 18:55:20.003943 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer done 2013-04-25 18:55:20.003992 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper 2013-04-25 18:55:20.004003 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaping pipe 0x7fca500023c0 10.1.255.22:0/1021534 2013-04-25 18:55:20.004008 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:55:20.004017 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1021534 pipe(0x7fca500023c0 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe - not registered 2013-04-25 18:55:20.004035 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaped pipe 0x7fca500023c0 10.1.255.22:0/1021534 2013-04-25 18:55:20.004043 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper deleted pipe 0x7fca500023c0 2013-04-25 18:55:20.004045 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper done 2013-04-25 18:58:20.168528 7fccbfcd4700 20 accepter.accepter poll got 1 2013-04-25 18:58:20.168546 7fccbfcd4700 10 accepter.pfd.revents=1 2013-04-25 18:58:20.168556 7fccbfcd4700 10 accepter.accepted incoming on sd 24 2013-04-25 18:58:20.168590 7fccbfcd4700 20 accepter.accepter calling poll 2013-04-25 18:58:20.168636 7fccc8402700 10 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50002910 sd=24 :0 s=0 pgs=0 cs=0 l=0).accept 2013-04-25 18:58:20.168723 7fccc8402700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50002910 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.1.255.22:50546/0 2013-04-25 18:58:20.169434 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is 10.1.255.22:0/1022124 2013-04-25 18:58:20.169461 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept got peer connect_seq 0 global_seq 1 2013-04-25 18:58:20.169504 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept of host_type 8, policy.lossy=1 2013-04-25 18:58:20.169511 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept my proto 15, their proto 15 2013-04-25 18:58:20.169519 7fccc8402700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.1.255.22:0/1022124 client protocol 0 2013-04-25 18:58:20.169529 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept: setting up session_security. 2013-04-25 18:58:20.169536 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept new session 2013-04-25 18:58:20.169542 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept success, connect_seq = 1, sending READY 2013-04-25 18:58:20.169552 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept features 18446744073709551615 2013-04-25 18:58:20.169559 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).register_pipe 2013-04-25 18:58:20.169574 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).discard_requeued_up_to 0 2013-04-25 18:58:20.169580 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept starting writer, state open 2013-04-25 18:58:20.169601 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept done 2013-04-25 18:58:20.169613 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:58:20.169650 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:58:20.169682 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:58:20.169693 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:58:20.170104 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got MSG 2013-04-25 18:58:20.170131 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got envelope type=17 src client.? front=62 data=0 off 0 2013-04-25 18:58:20.170144 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 bytes from policy throttler 0/104857600 2013-04-25 18:58:20.170153 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 from dispatch throttler 22024618/104857600 2013-04-25 18:58:20.170163 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got front 62 2013-04-25 18:58:20.170173 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).aborted = 0 2013-04-25 18:58:20.170179 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got 62 + 0 + 0 byte message 2013-04-25 18:58:20.170198 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got message 1 0x7fca24009ab0 auth(proto 0 32 bytes epoch 0) v1 2013-04-25 18:58:20.170210 7fccc8402700 20 -- 10.1.0.3:6789/0 queue 0x7fca24009ab0 prio 127 2013-04-25 18:58:20.170217 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 18:58:20.170286 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:58:20.170315 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).write_ack 1 2013-04-25 18:58:20.170332 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 18:58:20.170349 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 18:58:23.168167 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success 2013-04-25 18:58:23.168211 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success 2013-04-25 18:58:23.168237 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault on lossy channel, failing 2013-04-25 18:58:23.168245 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=2 pgs=1 cs=1 l=1).stop 2013-04-25 18:58:23.168254 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe 2013-04-25 18:58:23.168268 7fccc8402700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 62 to dispatch throttler 22024680/104857600 2013-04-25 18:58:23.168278 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:58:23.168287 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=4 pgs=1 cs=1 l=1).reader done 2013-04-25 18:58:23.168314 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer finishing 2013-04-25 18:58:23.168337 7fccc8301700 10 -- 10.1.0.3:6789/0 queue_reap 0x7fca50002910 2013-04-25 18:58:23.168346 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer done 2013-04-25 18:58:23.168411 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper 2013-04-25 18:58:23.168421 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaping pipe 0x7fca50002910 10.1.255.22:0/1022124 2013-04-25 18:58:23.168426 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 18:58:23.168433 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1022124 pipe(0x7fca50002910 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe - not registered 2013-04-25 18:58:23.168458 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaped pipe 0x7fca50002910 10.1.255.22:0/1022124 2013-04-25 18:58:23.168465 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper deleted pipe 0x7fca50002910 2013-04-25 18:58:23.168467 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper done 2013-04-25 19:05:23.872193 7fccbfcd4700 20 accepter.accepter poll got 1 2013-04-25 19:05:23.872213 7fccbfcd4700 10 accepter.pfd.revents=1 2013-04-25 19:05:23.872223 7fccbfcd4700 10 accepter.accepted incoming on sd 24 2013-04-25 19:05:23.872305 7fccbfcd4700 20 accepter.accepter calling poll 2013-04-25 19:05:23.872385 7fccc8402700 10 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50002e60 sd=24 :0 s=0 pgs=0 cs=0 l=0).accept 2013-04-25 19:05:23.872461 7fccc8402700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x7fca50002e60 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.1.255.22:51550/0 2013-04-25 19:05:23.873203 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is 10.1.255.22:0/1024364 2013-04-25 19:05:23.873230 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept got peer connect_seq 0 global_seq 1 2013-04-25 19:05:23.873240 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept of host_type 8, policy.lossy=1 2013-04-25 19:05:23.873246 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept my proto 15, their proto 15 2013-04-25 19:05:23.873254 7fccc8402700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.1.255.22:0/1024364 client protocol 0 2013-04-25 19:05:23.873264 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept: setting up session_security. 2013-04-25 19:05:23.873288 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=0 pgs=0 cs=0 l=1).accept new session 2013-04-25 19:05:23.873292 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept success, connect_seq = 1, sending READY 2013-04-25 19:05:23.873296 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept features 18446744073709551615 2013-04-25 19:05:23.873303 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).register_pipe 2013-04-25 19:05:23.873317 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).discard_requeued_up_to 0 2013-04-25 19:05:23.873324 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept starting writer, state open 2013-04-25 19:05:23.873343 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).accept done 2013-04-25 19:05:23.873351 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 19:05:23.873408 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 19:05:23.873430 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 19:05:23.873440 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 19:05:23.873987 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got MSG 2013-04-25 19:05:23.874014 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got envelope type=17 src client.? front=62 data=0 off 0 2013-04-25 19:05:23.874027 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 bytes from policy throttler 0/104857600 2013-04-25 19:05:23.874036 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader wants 62 from dispatch throttler 22024618/104857600 2013-04-25 19:05:23.874047 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got front 62 2013-04-25 19:05:23.874058 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).aborted = 0 2013-04-25 19:05:23.874065 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got 62 + 0 + 0 byte message 2013-04-25 19:05:23.874085 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader got message 1 0x7fca2000caa0 auth(proto 0 32 bytes epoch 0) v1 2013-04-25 19:05:23.874093 7fccc8402700 20 -- 10.1.0.3:6789/0 queue 0x7fca2000caa0 prio 127 2013-04-25 19:05:23.874099 7fccc8402700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader reading tag... 2013-04-25 19:05:23.874155 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 19:05:23.874177 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).write_ack 1 2013-04-25 19:05:23.874199 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1 2013-04-25 19:05:23.874215 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).writer sleeping 2013-04-25 19:05:26.871703 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success 2013-04-25 19:05:26.871740 7fccc8402700 2 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success 2013-04-25 19:05:26.871768 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).fault on lossy channel, failing 2013-04-25 19:05:26.871776 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=2 pgs=1 cs=1 l=1).stop 2013-04-25 19:05:26.871786 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe 2013-04-25 19:05:26.871799 7fccc8402700 10 -- 10.1.0.3:6789/0 dispatch_throttle_release 62 to dispatch throttler 22024680/104857600 2013-04-25 19:05:26.871809 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 19:05:26.871822 7fccc8402700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=4 pgs=1 cs=1 l=1).reader done 2013-04-25 19:05:26.871829 7fccc8301700 20 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer finishing 2013-04-25 19:05:26.871847 7fccc8301700 10 -- 10.1.0.3:6789/0 queue_reap 0x7fca50002e60 2013-04-25 19:05:26.871857 7fccc8301700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=4 pgs=1 cs=1 l=1).writer done 2013-04-25 19:05:26.871917 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper 2013-04-25 19:05:26.871926 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaping pipe 0x7fca50002e60 10.1.255.22:0/1024364 2013-04-25 19:05:26.871931 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=4 pgs=1 cs=1 l=1).discard_queue 2013-04-25 19:05:26.871939 7fccc14d7700 10 -- 10.1.0.3:6789/0 >> 10.1.255.22:0/1024364 pipe(0x7fca50002e60 sd=24 :6789 s=4 pgs=1 cs=1 l=1).unregister_pipe - not registered 2013-04-25 19:05:26.871957 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper reaped pipe 0x7fca50002e60 10.1.255.22:0/1024364 2013-04-25 19:05:26.871965 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper deleted pipe 0x7fca50002e60 2013-04-25 19:05:26.871968 7fccc14d7700 10 -- 10.1.0.3:6789/0 reaper done