ceph-mon.4.log - Ceph - Ceph

Bug #4811 » ceph-mon.4.log

Denis kaganovich, 04/25/2013 12:25 AM

 -04-25 10:07:40.073976 7fbce674e800  0 ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23), process ceph-mon, pid 1094
 -04-25 10:07:49.773487 7f3459191800  0 ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23), process ceph-mon, pid 1215
 -04-25 10:07:49.773503 7f3459191800 10 needs_conversion
 -04-25 10:07:50.706758 7f3459191800 10 obtain_monmap
 -04-25 10:07:57.109537 7f3459191800 10 obtain_monmap detected aborted sync
 -04-25 10:07:57.109631 7f3459191800 10 obtain_monmap read backup monmap
 -04-25 10:07:57.109660 7f3459191800  0 mon.4 does not exist in monmap, will attempt to join an existing cluster
 -04-25 10:07:57.110644 7f3459191800  1 mon.4@-1(probing) e1 preinit fsid 5eacb654-3204-4ca1-9b16-83bb0140e497
 -04-25 10:07:57.110746 7f3459191800 10 mon.4@-1(probing) e1 check_fsid cluster_uuid contains '5eacb654-3204-4ca1-9b16-83bb0140e497'
 -04-25 10:07:57.110793 7f3459191800 10 mon.4@-1(probing) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?)}
 -04-25 10:07:57.110833 7f3459191800 10 mon.4@-1(probing) e1 has_ever_joined = 1
 -04-25 10:07:57.110863 7f3459191800  1 mon.4@-1(probing) e1 preinit clean up potentially inconsistent store state
 -04-25 10:08:14.912907 7f3459191800 10 mon.4@-1(probing) e1 init_paxos
 -04-25 10:08:20.899994 7f3459191800 10 mon.4@-1(probing).log v0 update_from_paxos
 -04-25 10:08:20.917658 7f3459191800 10 mon.4@-1(probing).log v0 update_from_paxos version 0 summary v 0
 -04-25 10:08:26.867790 7f3459191800 10 mon.4@-1(probing).auth v0 update_from_paxos
 -04-25 10:08:27.077147 7f3459191800 10 mon.4@-1(probing).health(0) init
 -04-25 10:08:27.178455 7f3459191800 10 mon.4@-1(probing) e1 loading initial keyring to bootstrap authentication for mkfs
 -04-25 10:08:27.181914 7f3459191800  2 mon.4@-1(probing) e1 init
 -04-25 10:08:27.182092 7f3459191800 10 mon.4@-1(probing) e1 bootstrap
 -04-25 10:08:27.182098 7f3459191800 10 mon.4@-1(probing) e1 unregister_cluster_logger - not registered
 -04-25 10:08:27.182102 7f3459191800 10 mon.4@-1(probing) e1 cancel_probe_timeout (none scheduled)
 -04-25 10:08:27.182107 7f3459191800  0 mon.4@-1(probing) e1  my rank is now 2 (was -1)
 -04-25 10:08:27.182124 7f3459191800 10 mon.4@2(probing) e1 reset_sync
 -04-25 10:08:27.182130 7f3459191800 10 mon.4@2(probing) e1 reset
 -04-25 10:08:27.182132 7f3459191800 10 mon.4@2(probing) e1 timecheck_finish
 -04-25 10:08:27.182188 7f3459191800 10 mon.4@2(probing) e1 cancel_probe_timeout (none scheduled)
 -04-25 10:08:27.182199 7f3459191800 10 mon.4@2(probing) e1 reset_probe_timeout 0x3992a80 after 2 seconds
 -04-25 10:08:27.182281 7f3459191800 10 mon.4@2(probing) e1 probing other monitors
 -04-25 10:08:27.183091 7f34008f7700 10 mon.4@2(probing) e1 ms_get_authorizer for mon
 -04-25 10:08:27.183217 7f34007f6700 10 mon.4@2(probing) e1 ms_get_authorizer for mon
 -04-25 10:08:27.184084 7f34008f7700 10 mon.4@2(probing) e1 ms_get_authorizer for mon
 -04-25 10:08:27.184099 7f34007f6700 10 mon.4@2(probing) e1 ms_get_authorizer for mon
 -04-25 10:08:27.185769 7f34018f9700 10 mon.4@2(probing) e1 do not have session, making new one
 -04-25 10:08:27.185794 7f34018f9700 10 mon.4@2(probing) e1 ms_dispatch new session MonSession: mon.1 10.227.227.103:6789/0 is open for mon.1 10.227.227.103:6789/0
 -04-25 10:08:27.185802 7f34018f9700  5 mon.4@2(probing) e1 setting monitor caps on this connection
 -04-25 10:08:27.185810 7f34018f9700 10 mon.4@2(probing) e1 handle_probe mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4
 -04-25 10:08:27.185826 7f34018f9700 10 mon.4@2(probing) e1 handle_probe_reply mon.1 10.227.227.103:6789/0mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4
 -04-25 10:08:27.185837 7f34018f9700 10 mon.4@2(probing) e1  monmap is e1: 3 mons at {0=10.227.227.101:6789/0,1=10.227.227.103:6789/0,2=10.227.227.104:6789/0}
 -04-25 10:08:27.185880 7f34018f9700 10 mon.4@2(probing) e1  peer name is 1
 -04-25 10:08:27.185888 7f34018f9700 10 mon.4@2(probing) e1  existing quorum 0,1
 -04-25 10:08:27.185890 7f34018f9700 10 mon.4@2(probing) e1  peer paxos version 11589955 vs my version 0 (too far ahead)
 -04-25 10:08:27.185894 7f34018f9700 10 mon.4@2(probing) e1 cancel_probe_timeout 0x3992a80
 -04-25 10:08:27.185900 7f34018f9700 10 mon.4@2(probing) e1 sync_start entity( mon.1 10.227.227.103:6789/0 )
 -04-25 10:08:45.143820 7f34004f3700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.227.227.104:0/168822515 client protocol 0
 -04-25 10:08:50.650500 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )).monmap v1 get_monmap ver 0
 -04-25 10:08:56.347480 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_store_init backup current monmap
 -04-25 10:08:56.364214 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_handle_reset 0x2fd9a20 10.227.227.104:0/168822515
 -04-25 10:09:02.103859 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 do not have session, making new one
 -04-25 10:09:02.103994 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_dispatch new session MonSession: mon.0 10.227.227.101:6789/0 is open for mon.0 10.227.227.101:6789/0
 -04-25 10:09:02.104027 7f34018f9700  5 mon.4@2(synchronizing sync( requester state start )) e1 setting monitor caps on this connection
 -04-25 10:09:02.104052 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_probe mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4
 -04-25 10:09:02.104085 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_probe_reply mon.0 10.227.227.101:6789/0mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4
 -04-25 10:09:02.104115 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1  monmap is e1: 3 mons at {0=10.227.227.101:6789/0,1=10.227.227.103:6789/0,2=10.227.227.104:6789/0}
 -04-25 10:09:02.104209 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1
 -04-25 10:09:02.104236 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync_start_reply mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1
 -04-25 10:09:02.104263 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync_start_reply leader told us to use mon.1 10.227.227.103:6789/0 as sync provider
 -04-25 10:09:02.104283 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0)
 -04-25 10:09:02.104340 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_start_chunks provider(mon.1 10.227.227.103:6789/0)
 -04-25 10:09:02.132721 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1
 -04-25 10:09:02.132731 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1
 -04-25 10:09:12.843986 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1
 -04-25 10:09:12.844014 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1
 -04-25 10:09:12.934498 7f34004f3700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.227.227.103:0/18850 client protocol 0
 -04-25 10:09:13.056263 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0)
 -04-25 10:09:13.056400 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1
 -04-25 10:09:13.056425 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1
 -04-25 10:09:13.150562 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1
 -04-25 10:09:13.150573 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1
 -04-25 10:09:13.150606 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1
 -04-25 10:09:13.150612 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1
 -04-25 10:09:13.242184 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1
 -04-25 10:09:13.242193 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1
 -04-25 10:09:13.359156 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1
 -04-25 10:09:13.359164 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1
 -04-25 10:09:13.443022 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1
 -04-25 10:09:13.443044 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1
 -04-25 10:09:13.543178 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1
 -04-25 10:09:13.543188 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1
 -04-25 10:09:13.668263 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1
 -04-25 10:09:13.668273 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1
 -04-25 10:09:13.785538 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1
 -04-25 10:09:13.785553 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1
 -04-25 10:09:13.885302 7f34018f9700  5 mon.4@2(synchronizing sync( requester state chunks )) e1 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum
 -04-25 10:09:14.192649 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1
 -04-25 10:09:14.192666 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1
 -04-25 10:09:14.848570 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1
 -04-25 10:09:14.848588 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1
 -04-25 10:09:15.261543 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1
 -04-25 10:09:15.261568 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1
 -04-25 10:09:15.367962 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1
 -04-25 10:09:15.367991 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1
 -04-25 10:09:15.462411 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1
 -04-25 10:09:15.462424 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1
 -04-25 10:09:15.545497 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1
 -04-25 10:09:15.545514 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1
 -04-25 10:09:15.933782 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_handle_reset 0x2fd9b80 10.227.227.103:0/18850
 -04-25 10:09:17.844322 7f34020fa700  1 mon.4@2(synchronizing sync( requester state chunks )) e1 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum
 -04-25 10:09:18.150668 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0)
 -04-25 10:09:18.151927 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1
 -04-25 10:09:18.151943 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1
 -04-25 10:09:18.934981 7f34004f3700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.227.227.103:0/18850 client protocol 0
 -04-25 10:09:18.935425 7f34018f9700  5 mon.4@2(synchronizing sync( requester state chunks )) e1 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum
 -04-25 10:09:21.283812 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1
 -04-25 10:09:21.283829 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1
 -04-25 10:09:21.370114 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1
 -04-25 10:09:21.370124 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1
 -04-25 10:09:21.436523 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1
 -04-25 10:09:21.436537 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1
 -04-25 10:09:21.537153 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1
 -04-25 10:09:21.537167 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1
 -04-25 10:09:21.629180 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1
 -04-25 10:09:21.629197 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1
 -04-25 10:09:21.687218 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1
 -04-25 10:09:21.687229 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1
 -04-25 10:09:21.763104 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1
 -04-25 10:09:21.763116 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1
 -04-25 10:09:21.846705 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1
 -04-25 10:09:21.846717 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1
 -04-25 10:09:21.938639 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_handle_reset 0x6506580 10.227.227.103:0/18850
 -04-25 10:09:21.938700 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1
 -04-25 10:09:21.938709 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1
 -04-25 10:09:22.063707 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1
 -04-25 10:09:22.063719 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1
 -04-25 10:09:22.155552 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1
 -04-25 10:09:22.155570 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1
 -04-25 10:09:22.239128 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1
 -04-25 10:09:22.239140 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1
 -04-25 10:09:22.305924 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1
 -04-25 10:09:22.305936 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1
 -04-25 10:09:22.389566 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1
 -04-25 10:09:22.389578 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1
 -04-25 10:09:22.464955 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1
 -04-25 10:09:22.464970 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1
 -04-25 10:09:22.590523 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1
 -04-25 10:09:22.590534 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1
 -04-25 10:09:22.682374 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1
 -04-25 10:09:22.682383 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1
 -04-25 10:09:22.765907 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1
 -04-25 10:09:22.765917 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1
 -04-25 10:09:22.841936 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1
 -04-25 10:09:22.841953 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1
 -04-25 10:09:22.933991 7f34020fa700  1 mon.4@2(synchronizing sync( requester state chunks )) e1 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum
 -04-25 10:09:22.934100 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1
 -04-25 10:09:22.934123 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1
 -04-25 10:09:23.461787 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0)
 -04-25 10:09:23.461898 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1
 -04-25 10:09:23.461919 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1
 -04-25 10:09:23.650917 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1
 -04-25 10:09:23.650933 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1
 -04-25 10:09:23.650960 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1
 -04-25 10:09:23.650964 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1
 -04-25 10:09:23.878778 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1
 -04-25 10:09:23.878793 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1
 -04-25 10:09:24.062664 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1
 -04-25 10:09:24.062681 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1
 -04-25 10:09:24.256101 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1
 -04-25 10:09:24.256118 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1
 -04-25 10:09:24.431464 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1
 -04-25 10:09:24.431494 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1
 -04-25 10:09:24.573661 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1
 -04-25 10:09:24.573684 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1
 -04-25 10:09:24.742913 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1
 -04-25 10:09:24.742930 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1
 -04-25 10:09:24.916967 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1
 -04-25 10:09:24.917009 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1
 -04-25 10:09:25.073725 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1
 -04-25 10:09:25.073742 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1
 -04-25 10:09:25.249505 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1
 -04-25 10:09:25.249520 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1
 -04-25 10:09:25.449862 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1
 -04-25 10:09:25.449873 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1
 -04-25 10:09:25.600668 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1
 -04-25 10:09:25.600683 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1
 -04-25 10:09:25.819073 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1
 -04-25 10:09:25.819097 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1
 -04-25 10:09:25.917895 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1
 -04-25 10:09:25.917914 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1
 -04-25 10:09:26.102045 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1
 -04-25 10:09:26.102057 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1
 -04-25 10:09:26.202905 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1
 -04-25 10:09:26.202921 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1
 -04-25 10:09:26.296992 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1
 -04-25 10:09:26.297011 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1
 -04-25 10:09:26.454127 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1
 -04-25 10:09:26.454137 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1
 -04-25 10:09:26.629529 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1
 -04-25 10:09:26.629542 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1
 -04-25 10:09:26.872131 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_stop
 -04-25 10:09:26.873069 7f34018f9700 10 mon.4@2(synchronizing sync( requester state stop )) e1 handle_sync mon_sync( finish_reply ) v1
 -04-25 10:09:26.873088 7f34018f9700 10 mon.4@2(synchronizing sync( requester state stop )) e1 handle_sync_finish_reply mon_sync( finish_reply ) v1
 -04-25 10:09:27.073657 7f34018f9700 10 mon.4@2(synchronizing) e1 init_paxos
 -04-25 10:09:27.074136 7f34018f9700  7 mon.4@2(synchronizing).pg v0 update_from_paxos loading latest full pgmap v5896128
 -04-25 10:09:27.082000 7f34018f9700  7 mon.4@2(synchronizing).pg v5896128 update_from_paxos  applying incremental 5896129
 -04-25 10:09:27.082392 7f34018f9700 10 mon.4@2(synchronizing).pg v5896129 v5896129: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 4894KB/s wr, 188op/s
 -04-25 10:09:27.082468 7f34018f9700  7 mon.4@2(synchronizing).pg v5896129 update_from_paxos  applying incremental 5896130
 -04-25 10:09:27.082536 7f34018f9700 10 mon.4@2(synchronizing).pg v5896130 v5896130: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 2929KB/s wr, 114op/s
 -04-25 10:09:27.082576 7f34018f9700  7 mon.4@2(synchronizing).pg v5896130 update_from_paxos  applying incremental 5896131
 -04-25 10:09:27.082679 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 v5896131: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 1963KB/s wr, 90op/s
 -04-25 10:09:27.082711 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 send_pg_creates to 0 pgs
 -04-25 10:09:27.082717 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 update_logger
 -04-25 10:09:27.082803 7f34018f9700 10 mon.4@2(synchronizing).mds e0 update_from_paxos version 3711, my e 0
 -04-25 10:09:27.082854 7f34018f9700 10 mon.4@2(synchronizing).mds e0 update_from_paxos got 3711
 -04-25 10:09:27.082890 7f34018f9700  4 mon.4@2(synchronizing).mds e3711 new map
 -04-25 10:09:27.082895 7f34018f9700  7 mon.4@2(synchronizing).mds e3711 print_map
       epoch	3711
       flags	0
       created	2013-01-28 15:10:54.412329
       modified	2013-04-25 10:04:26.660566
       tableserver	0
       root	0
       session_timeout	60
       session_autoclose	300
       last_failure	3694
       last_failure_osd_epoch	10501
       compat	compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding}
       max_mds	1
       in	0
       up	{0=214699}
       failed
       stopped
       data_pools	0,6
       metadata_pool	1
 :	10.227.227.101:6810/6757 '1' mds.0.260 up:active seq 28
 :	10.227.227.103:6800/7999 '3' mds.-1.0 up:standby seq 1
 :	10.227.227.104:6800/31019 '4' mds.-1.0 up:standby seq 1
 -04-25 10:09:27.082955 7f34018f9700 10 mon.4@2(synchronizing).mds e3711 update_logger
 -04-25 10:09:27.083108 7f34018f9700  7 mon.4@2(synchronizing).osd e0 update_from_paxos loading latest full map e10517
 -04-25 10:09:27.083314 7f34018f9700  7 mon.4@2(synchronizing).osd e10517 update_from_paxos  applying incremental 10518
 -04-25 10:09:27.083402 7f34018f9700  1 mon.4@2(synchronizing).osd e10518 e10518: 12 osds: 10 up, 12 in
 -04-25 10:09:27.083440 7f34018f9700  7 mon.4@2(synchronizing).osd e10518 update_from_paxos  applying incremental 10519
 -04-25 10:09:27.083490 7f34018f9700  1 mon.4@2(synchronizing).osd e10519 e10519: 12 osds: 9 up, 12 in
 -04-25 10:09:27.083519 7f34018f9700  7 mon.4@2(synchronizing).osd e10519 update_from_paxos  applying incremental 10520
 -04-25 10:09:27.083565 7f34018f9700  1 mon.4@2(synchronizing).osd e10520 e10520: 12 osds: 9 up, 12 in
 -04-25 10:09:27.083598 7f34018f9700  7 mon.4@2(synchronizing).osd e10520 update_from_paxos  applying incremental 10521
 -04-25 10:09:27.083643 7f34018f9700  1 mon.4@2(synchronizing).osd e10521 e10521: 12 osds: 9 up, 12 in
 -04-25 10:09:27.083672 7f34018f9700  7 mon.4@2(synchronizing).osd e10521 update_from_paxos  applying incremental 10522
 -04-25 10:09:27.083718 7f34018f9700  1 mon.4@2(synchronizing).osd e10522 e10522: 12 osds: 9 up, 12 in
 -04-25 10:09:27.083761 7f34018f9700  7 mon.4@2(synchronizing).osd e10522 update_from_paxos  applying incremental 10523
 -04-25 10:09:27.083808 7f34018f9700  1 mon.4@2(synchronizing).osd e10523 e10523: 12 osds: 9 up, 12 in
 -04-25 10:09:27.083845 7f34018f9700  7 mon.4@2(synchronizing).osd e10523 update_from_paxos  applying incremental 10524
 -04-25 10:09:27.083899 7f34018f9700  1 mon.4@2(synchronizing).osd e10524 e10524: 12 osds: 10 up, 12 in
 -04-25 10:09:27.083934 7f34018f9700  7 mon.4@2(synchronizing).osd e10524 update_from_paxos  applying incremental 10525
 -04-25 10:09:27.083980 7f34018f9700  1 mon.4@2(synchronizing).osd e10525 e10525: 12 osds: 10 up, 12 in
 -04-25 10:09:27.084026 7f34018f9700  7 mon.4@2(synchronizing).osd e10525 update_from_paxos  applying incremental 10526
 -04-25 10:09:27.084077 7f34018f9700  1 mon.4@2(synchronizing).osd e10526 e10526: 12 osds: 12 up, 12 in
 -04-25 10:09:27.084107 7f34018f9700  7 mon.4@2(synchronizing).osd e10526 update_from_paxos  applying incremental 10527
 -04-25 10:09:27.084152 7f34018f9700  1 mon.4@2(synchronizing).osd e10527 e10527: 12 osds: 12 up, 12 in
 -04-25 10:09:27.211801 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 send_to_waiting 10527
 -04-25 10:09:27.211833 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 share_map_with_random_osd no up osd on our session map
 -04-25 10:09:27.211839 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 update_logger
 -04-25 10:09:27.211873 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos
 -04-25 10:09:27.212051 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos version 5569491 summary v 0
 -04-25 10:09:27.212234 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos latest full 5569490
 -04-25 10:09:27.212347 7f34018f9700  7 mon.4@2(synchronizing).log v5569491 update_from_paxos loading summary e5569490
 -04-25 10:09:27.212446 7f34018f9700  7 mon.4@2(synchronizing).log v5569491 update_from_paxos loaded summary e5569490
 -04-25 10:09:27.212631 7f34018f9700  7 mon.4@2(synchronizing).log v5569491 update_from_paxos applying incremental log 5569491 2013-04-25 10:08:59.980554 mon.0 10.227.227.101:6789/0 1031 : [INF] pgmap v5896131: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 1963KB/s wr, 90op/s
 -04-25 10:09:27.212766 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 check_subs
 -04-25 10:09:27.212873 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos
 -04-25 10:09:27.213007 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos version 4309 keys ver 0 latest 4307
 -04-25 10:09:27.213168 7f34018f9700  7 mon.4@2(synchronizing).auth v4309 update_from_paxos loading summary e 4307
 -04-25 10:09:27.213194 7f34018f9700  7 mon.4@2(synchronizing).auth v4309 update_from_paxos latest length 2392
 -04-25 10:09:27.213337 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos key server version 4307
 -04-25 10:09:27.213445 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos() last_allocated_id=215196 max_global_id=215196
 -04-25 10:09:27.213470 7f34018f9700 10 mon.4@2(synchronizing) e1 bootstrap
 -04-25 10:09:27.213475 7f34018f9700 10 mon.4@2(synchronizing) e1 unregister_cluster_logger - not registered
 -04-25 10:09:27.213479 7f34018f9700 10 mon.4@2(synchronizing) e1 cancel_probe_timeout (none scheduled)
 -04-25 10:09:27.213484 7f34018f9700 10 mon.4@2(synchronizing) e1 reset_sync
 -04-25 10:09:27.213489 7f34018f9700 10 mon.4@2(probing) e1 reset
 -04-25 10:09:27.213494 7f34018f9700 10 mon.4@2(probing) e1 timecheck_finish
 -04-25 10:09:27.213507 7f34018f9700 10 mon.4@2(probing) e1 cancel_probe_timeout (none scheduled)
 -04-25 10:09:27.213515 7f34018f9700 10 mon.4@2(probing) e1 reset_probe_timeout 0x3992f70 after 2 seconds
 -04-25 10:09:27.213531 7f34018f9700 10 mon.4@2(probing) e1 probing other monitors
 -04-25 10:09:27.213615 7f34020fa700 10 mon.4@2(probing).data_health(0) service_tick
 -04-25 10:09:27.214510 7f34020fa700 -1 mon/MonMap.h: In function 'entity_inst_t MonMap::get_inst(const string&)' thread 7f34020fa700 time 2013-04-25 10:09:27.213645
       mon/MonMap.h: 160: FAILED assert(mon_addr.count(n))
        ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23)
 : (MonMap::get_inst(std::string const&)+0x159) [0x4bf079]
 : (DataHealthService::update_stats()+0x67) [0x59a827]
 : (DataHealthService::service_tick()+0x5a) [0x59ba5a]
 : (QuorumService::C_Tick::finish(int)+0x17) [0x59d0a7]
 : (Context::complete(int)+0xa) [0x4bbf2a]
 : (SafeTimer::timer_thread()+0x42e) [0x696e5e]
 : (SafeTimerThread::entry()+0xd) [0x69845d]
 : (()+0x84f8) [0x7f3458d6d4f8]
 : (clone()+0x6d) [0x7f3456ff276d]
        NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
       --- begin dump of recent events ---
         -426> 2013-04-25 10:07:49.771407 7f3459191800  5 asok(0x2fc8000) register_command perfcounters_dump hook 0x2fb2010
         -425> 2013-04-25 10:07:49.771448 7f3459191800  5 asok(0x2fc8000) register_command 1 hook 0x2fb2010
         -424> 2013-04-25 10:07:49.771462 7f3459191800  5 asok(0x2fc8000) register_command perf dump hook 0x2fb2010
         -423> 2013-04-25 10:07:49.771481 7f3459191800  5 asok(0x2fc8000) register_command perfcounters_schema hook 0x2fb2010
         -422> 2013-04-25 10:07:49.771485 7f3459191800  5 asok(0x2fc8000) register_command 2 hook 0x2fb2010
         -421> 2013-04-25 10:07:49.771487 7f3459191800  5 asok(0x2fc8000) register_command perf schema hook 0x2fb2010
         -420> 2013-04-25 10:07:49.771495 7f3459191800  5 asok(0x2fc8000) register_command config show hook 0x2fb2010
         -419> 2013-04-25 10:07:49.771503 7f3459191800  5 asok(0x2fc8000) register_command config set hook 0x2fb2010
         -418> 2013-04-25 10:07:49.771508 7f3459191800  5 asok(0x2fc8000) register_command log flush hook 0x2fb2010
         -417> 2013-04-25 10:07:49.771510 7f3459191800  5 asok(0x2fc8000) register_command log dump hook 0x2fb2010
         -416> 2013-04-25 10:07:49.771515 7f3459191800  5 asok(0x2fc8000) register_command log reopen hook 0x2fb2010
         -415> 2013-04-25 10:07:49.773487 7f3459191800  0 ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23), process ceph-mon, pid 1215
         -414> 2013-04-25 10:07:49.773503 7f3459191800 10 needs_conversion
         -413> 2013-04-25 10:07:50.706758 7f3459191800 10 obtain_monmap
         -412> 2013-04-25 10:07:57.109537 7f3459191800 10 obtain_monmap detected aborted sync
         -411> 2013-04-25 10:07:57.109631 7f3459191800 10 obtain_monmap read backup monmap
         -410> 2013-04-25 10:07:57.109660 7f3459191800  0 mon.4 does not exist in monmap, will attempt to join an existing cluster
         -409> 2013-04-25 10:07:57.110550 7f3459191800  1 -- 10.227.227.104:6789/0 learned my addr 10.227.227.104:6789/0
         -408> 2013-04-25 10:07:57.110566 7f3459191800  1 accepter.accepter.bind my_inst.addr is 10.227.227.104:6789/0 need_addr=0
         -407> 2013-04-25 10:07:57.110586 7f3459191800  5 adding auth protocol: cephx
         -406> 2013-04-25 10:07:57.110591 7f3459191800  5 adding auth protocol: cephx
         -405> 2013-04-25 10:07:57.110644 7f3459191800  1 mon.4@-1(probing) e1 preinit fsid 5eacb654-3204-4ca1-9b16-83bb0140e497
         -404> 2013-04-25 10:07:57.110746 7f3459191800 10 mon.4@-1(probing) e1 check_fsid cluster_uuid contains '5eacb654-3204-4ca1-9b16-83bb0140e497'
         -403> 2013-04-25 10:07:57.110793 7f3459191800 10 mon.4@-1(probing) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?)}
         -402> 2013-04-25 10:07:57.110833 7f3459191800 10 mon.4@-1(probing) e1 has_ever_joined = 1
         -401> 2013-04-25 10:07:57.110863 7f3459191800  1 mon.4@-1(probing) e1 preinit clean up potentially inconsistent store state
         -400> 2013-04-25 10:08:14.912907 7f3459191800 10 mon.4@-1(probing) e1 init_paxos
         -399> 2013-04-25 10:08:20.899994 7f3459191800 10 mon.4@-1(probing).log v0 update_from_paxos
         -398> 2013-04-25 10:08:20.917658 7f3459191800 10 mon.4@-1(probing).log v0 update_from_paxos version 0 summary v 0
         -397> 2013-04-25 10:08:26.867790 7f3459191800 10 mon.4@-1(probing).auth v0 update_from_paxos
         -396> 2013-04-25 10:08:27.077147 7f3459191800 10 mon.4@-1(probing).health(0) init
         -395> 2013-04-25 10:08:27.178455 7f3459191800 10 mon.4@-1(probing) e1 loading initial keyring to bootstrap authentication for mkfs
         -394> 2013-04-25 10:08:27.179210 7f3459191800  2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-4/keyring
         -393> 2013-04-25 10:08:27.179221 7f3459191800  5 asok(0x2fc8000) register_command mon_status hook 0x3992ae0
         -392> 2013-04-25 10:08:27.179229 7f3459191800  5 asok(0x2fc8000) register_command quorum_status hook 0x3992ae0
         -391> 2013-04-25 10:08:27.179232 7f3459191800  5 asok(0x2fc8000) register_command sync_status hook 0x3992ae0
         -390> 2013-04-25 10:08:27.179235 7f3459191800  5 asok(0x2fc8000) register_command add_bootstrap_peer_hint hook 0x3992ae0
         -389> 2013-04-25 10:08:27.181041 7f3459191800  1 finished global_init_daemonize
         -388> 2013-04-25 10:08:27.181475 7f3459191800  5 asok(0x2fc8000) init /var/run/ceph/ceph-mon.4.asok
         -387> 2013-04-25 10:08:27.181539 7f3459191800  5 asok(0x2fc8000) bind_and_listen /var/run/ceph/ceph-mon.4.asok
         -386> 2013-04-25 10:08:27.181642 7f3459191800  5 asok(0x2fc8000) register_command 0 hook 0x2fb15a0
         -385> 2013-04-25 10:08:27.181662 7f3459191800  5 asok(0x2fc8000) register_command version hook 0x2fb15a0
         -384> 2013-04-25 10:08:27.181682 7f3459191800  5 asok(0x2fc8000) register_command git_version hook 0x2fb15a0
         -383> 2013-04-25 10:08:27.181706 7f3459191800  5 asok(0x2fc8000) register_command help hook 0x3992ad0
         -382> 2013-04-25 10:08:27.181738 7f34038fd700  5 asok(0x2fc8000) entry start
         -381> 2013-04-25 10:08:27.181853 7f3459191800  1 -- 10.227.227.104:6789/0 messenger.start
         -380> 2013-04-25 10:08:27.181914 7f3459191800  2 mon.4@-1(probing) e1 init
         -379> 2013-04-25 10:08:27.182049 7f3459191800  1 accepter.accepter.start
         -378> 2013-04-25 10:08:27.182092 7f3459191800 10 mon.4@-1(probing) e1 bootstrap
         -377> 2013-04-25 10:08:27.182098 7f3459191800 10 mon.4@-1(probing) e1 unregister_cluster_logger - not registered
         -376> 2013-04-25 10:08:27.182102 7f3459191800 10 mon.4@-1(probing) e1 cancel_probe_timeout (none scheduled)
         -375> 2013-04-25 10:08:27.182107 7f3459191800  0 mon.4@-1(probing) e1  my rank is now 2 (was -1)
         -374> 2013-04-25 10:08:27.182112 7f3459191800  1 -- 10.227.227.104:6789/0 mark_down_all
         -373> 2013-04-25 10:08:27.182124 7f3459191800 10 mon.4@2(probing) e1 reset_sync
         -372> 2013-04-25 10:08:27.182130 7f3459191800 10 mon.4@2(probing) e1 reset
         -371> 2013-04-25 10:08:27.182132 7f3459191800 10 mon.4@2(probing) e1 timecheck_finish
         -370> 2013-04-25 10:08:27.182188 7f3459191800 10 mon.4@2(probing) e1 cancel_probe_timeout (none scheduled)
         -369> 2013-04-25 10:08:27.182199 7f3459191800 10 mon.4@2(probing) e1 reset_probe_timeout 0x3992a80 after 2 seconds
         -368> 2013-04-25 10:08:27.182281 7f3459191800 10 mon.4@2(probing) e1 probing other monitors
         -367> 2013-04-25 10:08:27.182297 7f3459191800  1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_probe(probe 5eacb654-3204-4ca1-9b16-83bb0140e497 name 4) v4 -- ?+0 0x5a9ec00
         -366> 2013-04-25 10:08:27.182425 7f3459191800  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_probe(probe 5eacb654-3204-4ca1-9b16-83bb0140e497 name 4) v4 -- ?+0 0x5a9e900
         -365> 2013-04-25 10:08:27.183091 7f34008f7700 10 mon.4@2(probing) e1 ms_get_authorizer for mon
         -364> 2013-04-25 10:08:27.183217 7f34007f6700 10 mon.4@2(probing) e1 ms_get_authorizer for mon
         -363> 2013-04-25 10:08:27.184084 7f34008f7700 10 mon.4@2(probing) e1 ms_get_authorizer for mon
         -362> 2013-04-25 10:08:27.184099 7f34007f6700 10 mon.4@2(probing) e1 ms_get_authorizer for mon
         -361> 2013-04-25 10:08:27.185721 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971530 ==== mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 ==== 563+0+0 (2117610832 0 0) 0x5a9e900 con 0x2fd9080
         -360> 2013-04-25 10:08:27.185769 7f34018f9700 10 mon.4@2(probing) e1 do not have session, making new one
         -359> 2013-04-25 10:08:27.185794 7f34018f9700 10 mon.4@2(probing) e1 ms_dispatch new session MonSession: mon.1 10.227.227.103:6789/0 is open for mon.1 10.227.227.103:6789/0
         -358> 2013-04-25 10:08:27.185802 7f34018f9700  5 mon.4@2(probing) e1 setting monitor caps on this connection
         -357> 2013-04-25 10:08:27.185810 7f34018f9700 10 mon.4@2(probing) e1 handle_probe mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4
         -356> 2013-04-25 10:08:27.185826 7f34018f9700 10 mon.4@2(probing) e1 handle_probe_reply mon.1 10.227.227.103:6789/0mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4
         -355> 2013-04-25 10:08:27.185837 7f34018f9700 10 mon.4@2(probing) e1  monmap is e1: 3 mons at {0=10.227.227.101:6789/0,1=10.227.227.103:6789/0,2=10.227.227.104:6789/0}
         -354> 2013-04-25 10:08:27.185880 7f34018f9700 10 mon.4@2(probing) e1  peer name is 1
         -353> 2013-04-25 10:08:27.185888 7f34018f9700 10 mon.4@2(probing) e1  existing quorum 0,1
         -352> 2013-04-25 10:08:27.185890 7f34018f9700 10 mon.4@2(probing) e1  peer paxos version 11589955 vs my version 0 (too far ahead)
         -351> 2013-04-25 10:08:27.185894 7f34018f9700 10 mon.4@2(probing) e1 cancel_probe_timeout 0x3992a80
         -350> 2013-04-25 10:08:27.185900 7f34018f9700 10 mon.4@2(probing) e1 sync_start entity( mon.1 10.227.227.103:6789/0 )
         -349> 2013-04-25 10:08:45.143597 7f34004f3700  1 -- 10.227.227.104:6789/0 >> :/0 pipe(0x5304780 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.227.227.104:33345/0
         -348> 2013-04-25 10:08:45.143820 7f34004f3700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.227.227.104:0/168822515 client protocol 0
         -347> 2013-04-25 10:08:48.143458 7f34004f3700  2 -- 10.227.227.104:6789/0 >> 10.227.227.104:0/168822515 pipe(0x5304780 sd=23 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success
         -346> 2013-04-25 10:08:48.143529 7f34004f3700  2 -- 10.227.227.104:6789/0 >> 10.227.227.104:0/168822515 pipe(0x5304780 sd=23 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success
         -345> 2013-04-25 10:08:50.650500 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )).monmap v1 get_monmap ver 0
         -344> 2013-04-25 10:08:56.347480 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_store_init backup current monmap
         -343> 2013-04-25 10:08:56.364109 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( start ) v1 -- ?+0 0x2fe3340
         -342> 2013-04-25 10:08:56.364214 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_handle_reset 0x2fd9a20 10.227.227.104:0/168822515
         -341> 2013-04-25 10:08:56.364245 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355047 ==== mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 ==== 563+0+0 (3066673873 0 0) 0x5a9ec00 con 0x2fd9340
         -340> 2013-04-25 10:09:02.103859 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 do not have session, making new one
         -339> 2013-04-25 10:09:02.103994 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_dispatch new session MonSession: mon.0 10.227.227.101:6789/0 is open for mon.0 10.227.227.101:6789/0
         -338> 2013-04-25 10:09:02.104027 7f34018f9700  5 mon.4@2(synchronizing sync( requester state start )) e1 setting monitor caps on this connection
         -337> 2013-04-25 10:09:02.104052 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_probe mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4
         -336> 2013-04-25 10:09:02.104085 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_probe_reply mon.0 10.227.227.101:6789/0mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4
         -335> 2013-04-25 10:09:02.104115 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1  monmap is e1: 3 mons at {0=10.227.227.101:6789/0,1=10.227.227.103:6789/0,2=10.227.227.104:6789/0}
         -334> 2013-04-25 10:09:02.104166 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355048 ==== mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1 ==== 174+0+0 (3214969469 0 0) 0x2fe3600 con 0x2fd9340
         -333> 2013-04-25 10:09:02.104209 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1
         -332> 2013-04-25 10:09:02.104236 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync_start_reply mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1
         -331> 2013-04-25 10:09:02.104263 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync_start_reply leader told us to use mon.1 10.227.227.103:6789/0 as sync provider
         -330> 2013-04-25 10:09:02.104283 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0)
         -329> 2013-04-25 10:09:02.104305 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x2fe3b80
         -328> 2013-04-25 10:09:02.104340 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_start_chunks provider(mon.1 10.227.227.103:6789/0)
         -327> 2013-04-25 10:09:02.104369 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( start_chunks ) v1 -- ?+0 0x2fe38c0
         -326> 2013-04-25 10:09:02.132688 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355049 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x2fe3b80 con 0x2fd9340
         -325> 2013-04-25 10:09:02.132721 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1
         -324> 2013-04-25 10:09:02.132731 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1
         -323> 2013-04-25 10:09:08.069875 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971531 ==== mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1 ==== 1044432+0+0 (1482870314 0 0) 0x2fe38c0 con 0x2fd9080
         -322> 2013-04-25 10:09:12.843986 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1
         -321> 2013-04-25 10:09:12.844014 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1
         -320> 2013-04-25 10:09:12.844610 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3b80
         -319> 2013-04-25 10:09:12.934223 7f34004f3700  1 -- 10.227.227.104:6789/0 >> :/0 pipe(0x509c280 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.227.227.103:59710/0
         -318> 2013-04-25 10:09:12.934498 7f34004f3700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.227.227.103:0/18850 client protocol 0
         -317> 2013-04-25 10:09:13.056252 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971532 ==== mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1 ==== 1034782+0+0 (1741418579 0 0) 0x2fe3b80 con 0x2fd9080
         -316> 2013-04-25 10:09:13.056263 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0)
         -315> 2013-04-25 10:09:13.056319 7f34020fa700  1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x43fcdc0
         -314> 2013-04-25 10:09:13.056400 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1
         -313> 2013-04-25 10:09:13.056425 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1
         -312> 2013-04-25 10:09:13.056517 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0
         -311> 2013-04-25 10:09:13.150519 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355050 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x43fcdc0 con 0x2fd9340
         -310> 2013-04-25 10:09:13.150562 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1
         -309> 2013-04-25 10:09:13.150573 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1
         -308> 2013-04-25 10:09:13.150594 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971533 ==== mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1 ==== 1033109+0+0 (2445730747 0 0) 0x2fe38c0 con 0x2fd9080
         -307> 2013-04-25 10:09:13.150606 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1
         -306> 2013-04-25 10:09:13.150612 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1
         -305> 2013-04-25 10:09:13.150669 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -304> 2013-04-25 10:09:13.242148 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971534 ==== mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1 ==== 1040561+0+0 (2240089044 0 0) 0x2fe3600 con 0x2fd9080
         -303> 2013-04-25 10:09:13.242184 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1
         -302> 2013-04-25 10:09:13.242193 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1
         -301> 2013-04-25 10:09:13.242241 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0
         -300> 2013-04-25 10:09:13.359125 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971535 ==== mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1 ==== 1040284+0+0 (2564410500 0 0) 0x2fe38c0 con 0x2fd9080
         -299> 2013-04-25 10:09:13.359156 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1
         -298> 2013-04-25 10:09:13.359164 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1
         -297> 2013-04-25 10:09:13.359209 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -296> 2013-04-25 10:09:13.442973 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971536 ==== mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1 ==== 1048082+0+0 (1045690206 0 0) 0x2fe3600 con 0x2fd9080
         -295> 2013-04-25 10:09:13.443022 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1
         -294> 2013-04-25 10:09:13.443044 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1
         -293> 2013-04-25 10:09:13.443132 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0
         -292> 2013-04-25 10:09:13.543136 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971537 ==== mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1 ==== 1041708+0+0 (2728244868 0 0) 0x2fe38c0 con 0x2fd9080
         -291> 2013-04-25 10:09:13.543178 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1
         -290> 2013-04-25 10:09:13.543188 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1
         -289> 2013-04-25 10:09:13.543253 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -288> 2013-04-25 10:09:13.668228 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971538 ==== mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1 ==== 1038588+0+0 (3496012706 0 0) 0x2fe3600 con 0x2fd9080
         -287> 2013-04-25 10:09:13.668263 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1
         -286> 2013-04-25 10:09:13.668273 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1
         -285> 2013-04-25 10:09:13.668323 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0
         -284> 2013-04-25 10:09:13.785464 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971539 ==== mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1 ==== 1038296+0+0 (2754640597 0 0) 0x2fe38c0 con 0x2fd9080
         -283> 2013-04-25 10:09:13.785538 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1
         -282> 2013-04-25 10:09:13.785553 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1
         -281> 2013-04-25 10:09:13.785642 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -280> 2013-04-25 10:09:13.885235 7f34018f9700  1 -- 10.227.227.104:6789/0 <== client.? 10.227.227.103:0/18850 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x5b7a900 con 0x2fd9b80
         -279> 2013-04-25 10:09:13.885302 7f34018f9700  5 mon.4@2(synchronizing sync( requester state chunks )) e1 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum
         -278> 2013-04-25 10:09:14.192588 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971540 ==== mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1 ==== 1048582+0+0 (2478673006 0 0) 0x2fe3600 con 0x2fd9080
         -277> 2013-04-25 10:09:14.192649 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1
         -276> 2013-04-25 10:09:14.192666 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1
         -275> 2013-04-25 10:09:14.194163 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0
         -274> 2013-04-25 10:09:14.848511 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971541 ==== mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1 ==== 1047810+0+0 (766408052 0 0) 0x2fe38c0 con 0x2fd9080
         -273> 2013-04-25 10:09:14.848570 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1
         -272> 2013-04-25 10:09:14.848588 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1
         -271> 2013-04-25 10:09:14.850593 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -270> 2013-04-25 10:09:15.261469 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971542 ==== mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1 ==== 1042123+0+0 (450065390 0 0) 0x2fe3600 con 0x2fd9080
         -269> 2013-04-25 10:09:15.261543 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1
         -268> 2013-04-25 10:09:15.261568 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1
         -267> 2013-04-25 10:09:15.264252 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0
         -266> 2013-04-25 10:09:15.367892 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971543 ==== mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1 ==== 1047344+0+0 (791022580 0 0) 0x2fe38c0 con 0x2fd9080
         -265> 2013-04-25 10:09:15.367962 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1
         -264> 2013-04-25 10:09:15.367991 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1
         -263> 2013-04-25 10:09:15.368260 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -262> 2013-04-25 10:09:15.462353 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971544 ==== mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1 ==== 1043142+0+0 (1111262443 0 0) 0x2fe3600 con 0x2fd9080
         -261> 2013-04-25 10:09:15.462411 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1
         -260> 2013-04-25 10:09:15.462424 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1
         -259> 2013-04-25 10:09:15.462647 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0
         -258> 2013-04-25 10:09:15.545446 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971545 ==== mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1 ==== 1042501+0+0 (1058205326 0 0) 0x2fe38c0 con 0x2fd9080
         -257> 2013-04-25 10:09:15.545497 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1
         -256> 2013-04-25 10:09:15.545514 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1
         -255> 2013-04-25 10:09:15.545659 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -254> 2013-04-25 10:09:15.933605 7f34004f3700  2 -- 10.227.227.104:6789/0 >> 10.227.227.103:0/18850 pipe(0x509c280 sd=23 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success
         -253> 2013-04-25 10:09:15.933672 7f34004f3700  2 -- 10.227.227.104:6789/0 >> 10.227.227.103:0/18850 pipe(0x509c280 sd=23 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success
         -252> 2013-04-25 10:09:15.933782 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_handle_reset 0x2fd9b80 10.227.227.103:0/18850
         -251> 2013-04-25 10:09:17.844322 7f34020fa700  1 mon.4@2(synchronizing sync( requester state chunks )) e1 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum
         -250> 2013-04-25 10:09:17.844347 7f34020fa700  1 -- 10.227.227.104:6789/0 mark_down 0x2fd9b80 -- pipe dne
         -249> 2013-04-25 10:09:18.150668 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0)
         -248> 2013-04-25 10:09:18.150723 7f34020fa700  1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x81b98c0
         -247> 2013-04-25 10:09:18.151875 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355051 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x81b98c0 con 0x2fd9340
         -246> 2013-04-25 10:09:18.151927 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1
         -245> 2013-04-25 10:09:18.151943 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1
         -244> 2013-04-25 10:09:18.934744 7f34004f3700  1 -- 10.227.227.104:6789/0 >> :/0 pipe(0x509c000 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.227.227.103:59712/0
         -243> 2013-04-25 10:09:18.934981 7f34004f3700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.227.227.103:0/18850 client protocol 0
         -242> 2013-04-25 10:09:18.935384 7f34018f9700  1 -- 10.227.227.104:6789/0 <== client.? 10.227.227.103:0/18850 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x38a4000 con 0x6506580
         -241> 2013-04-25 10:09:18.935425 7f34018f9700  5 mon.4@2(synchronizing sync( requester state chunks )) e1 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum
         -240> 2013-04-25 10:09:21.283756 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971546 ==== mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1 ==== 1026551+0+0 (4095556378 0 0) 0x2fe3600 con 0x2fd9080
         -239> 2013-04-25 10:09:21.283812 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1
         -238> 2013-04-25 10:09:21.283829 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1
         -237> 2013-04-25 10:09:21.283941 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -236> 2013-04-25 10:09:21.370070 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971547 ==== mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1 ==== 268501+0+0 (83636244 0 0) 0x81b98c0 con 0x2fd9080
         -235> 2013-04-25 10:09:21.370114 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1
         -234> 2013-04-25 10:09:21.370124 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1
         -233> 2013-04-25 10:09:21.370150 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -232> 2013-04-25 10:09:21.436481 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971548 ==== mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1 ==== 1048515+0+0 (518301646 0 0) 0x2fe3600 con 0x2fd9080
         -231> 2013-04-25 10:09:21.436523 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1
         -230> 2013-04-25 10:09:21.436537 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1
         -229> 2013-04-25 10:09:21.436566 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -228> 2013-04-25 10:09:21.537106 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971549 ==== mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1 ==== 984553+0+0 (1049056081 0 0) 0x81b98c0 con 0x2fd9080
         -227> 2013-04-25 10:09:21.537153 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1
         -226> 2013-04-25 10:09:21.537167 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1
         -225> 2013-04-25 10:09:21.537204 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -224> 2013-04-25 10:09:21.629137 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971550 ==== mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1 ==== 891521+0+0 (2899987343 0 0) 0x2fe3600 con 0x2fd9080
         -223> 2013-04-25 10:09:21.629180 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1
         -222> 2013-04-25 10:09:21.629197 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1
         -221> 2013-04-25 10:09:21.629238 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -220> 2013-04-25 10:09:21.687180 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971551 ==== mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1 ==== 1047566+0+0 (2939424964 0 0) 0x81b98c0 con 0x2fd9080
         -219> 2013-04-25 10:09:21.687218 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1
         -218> 2013-04-25 10:09:21.687229 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1
         -217> 2013-04-25 10:09:21.687327 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -216> 2013-04-25 10:09:21.763061 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971552 ==== mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1 ==== 1036708+0+0 (3106831870 0 0) 0x2fe3600 con 0x2fd9080
         -215> 2013-04-25 10:09:21.763104 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1
         -214> 2013-04-25 10:09:21.763116 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1
         -213> 2013-04-25 10:09:21.763184 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -212> 2013-04-25 10:09:21.846663 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971553 ==== mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1 ==== 1042318+0+0 (2130255415 0 0) 0x81b98c0 con 0x2fd9080
         -211> 2013-04-25 10:09:21.846705 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1
         -210> 2013-04-25 10:09:21.846717 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1
         -209> 2013-04-25 10:09:21.846784 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -208> 2013-04-25 10:09:21.934957 7f34004f3700  2 -- 10.227.227.104:6789/0 >> 10.227.227.103:0/18850 pipe(0x509c000 sd=23 :6789 s=2 pgs=3 cs=1 l=1).reader couldn't read tag, Success
         -207> 2013-04-25 10:09:21.935022 7f34004f3700  2 -- 10.227.227.104:6789/0 >> 10.227.227.103:0/18850 pipe(0x509c000 sd=23 :6789 s=2 pgs=3 cs=1 l=1).fault 0: Success
         -206> 2013-04-25 10:09:21.938639 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_handle_reset 0x6506580 10.227.227.103:0/18850
         -205> 2013-04-25 10:09:21.938678 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971554 ==== mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1 ==== 970349+0+0 (2769782619 0 0) 0x2fe3600 con 0x2fd9080
         -204> 2013-04-25 10:09:21.938700 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1
         -203> 2013-04-25 10:09:21.938709 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1
         -202> 2013-04-25 10:09:21.938763 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -201> 2013-04-25 10:09:22.063667 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971555 ==== mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1 ==== 1025785+0+0 (2957537854 0 0) 0x81b98c0 con 0x2fd9080
         -200> 2013-04-25 10:09:22.063707 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1
         -199> 2013-04-25 10:09:22.063719 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1
         -198> 2013-04-25 10:09:22.063744 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -197> 2013-04-25 10:09:22.155494 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971556 ==== mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1 ==== 1038617+0+0 (2914685442 0 0) 0x2fe3600 con 0x2fd9080
         -196> 2013-04-25 10:09:22.155552 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1
         -195> 2013-04-25 10:09:22.155570 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1
         -194> 2013-04-25 10:09:22.155617 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -193> 2013-04-25 10:09:22.239085 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971557 ==== mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1 ==== 812850+0+0 (2145814566 0 0) 0x81b98c0 con 0x2fd9080
         -192> 2013-04-25 10:09:22.239128 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1
         -191> 2013-04-25 10:09:22.239140 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1
         -190> 2013-04-25 10:09:22.239173 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -189> 2013-04-25 10:09:22.305883 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971558 ==== mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1 ==== 964205+0+0 (1601467175 0 0) 0x2fe3600 con 0x2fd9080
         -188> 2013-04-25 10:09:22.305924 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1
         -187> 2013-04-25 10:09:22.305936 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1
         -186> 2013-04-25 10:09:22.305962 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -185> 2013-04-25 10:09:22.389528 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971559 ==== mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1 ==== 1046519+0+0 (3493477841 0 0) 0x81b98c0 con 0x2fd9080
         -184> 2013-04-25 10:09:22.389566 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1
         -183> 2013-04-25 10:09:22.389578 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1
         -182> 2013-04-25 10:09:22.389632 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -181> 2013-04-25 10:09:22.464896 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971560 ==== mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1 ==== 1043572+0+0 (83147344 0 0) 0x2fe3600 con 0x2fd9080
         -180> 2013-04-25 10:09:22.464955 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1
         -179> 2013-04-25 10:09:22.464970 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1
         -178> 2013-04-25 10:09:22.465055 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -177> 2013-04-25 10:09:22.590481 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971561 ==== mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1 ==== 1005179+0+0 (3321238167 0 0) 0x81b98c0 con 0x2fd9080
         -176> 2013-04-25 10:09:22.590523 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1
         -175> 2013-04-25 10:09:22.590534 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1
         -174> 2013-04-25 10:09:22.590574 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -173> 2013-04-25 10:09:22.682332 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971562 ==== mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1 ==== 845548+0+0 (2536188459 0 0) 0x2fe3600 con 0x2fd9080
         -172> 2013-04-25 10:09:22.682374 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1
         -171> 2013-04-25 10:09:22.682383 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1
         -170> 2013-04-25 10:09:22.682421 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -169> 2013-04-25 10:09:22.765867 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971563 ==== mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1 ==== 914359+0+0 (140853062 0 0) 0x81b98c0 con 0x2fd9080
         -168> 2013-04-25 10:09:22.765907 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1
         -167> 2013-04-25 10:09:22.765917 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1
         -166> 2013-04-25 10:09:22.765934 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -165> 2013-04-25 10:09:22.841875 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971564 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1 ==== 925511+0+0 (3220308 0 0) 0x2fe3600 con 0x2fd9080
         -164> 2013-04-25 10:09:22.841936 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1
         -163> 2013-04-25 10:09:22.841953 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1
         -162> 2013-04-25 10:09:22.841978 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -161> 2013-04-25 10:09:22.933748 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971565 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1 ==== 925511+0+0 (405716569 0 0) 0x81b98c0 con 0x2fd9080
         -160> 2013-04-25 10:09:22.933991 7f34020fa700  1 mon.4@2(synchronizing sync( requester state chunks )) e1 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum
         -159> 2013-04-25 10:09:22.934028 7f34020fa700  1 -- 10.227.227.104:6789/0 mark_down 0x6506580 -- pipe dne
         -158> 2013-04-25 10:09:22.934100 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1
         -157> 2013-04-25 10:09:22.934123 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1
         -156> 2013-04-25 10:09:22.934142 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600
         -155> 2013-04-25 10:09:23.461787 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0)
         -154> 2013-04-25 10:09:23.461789 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971566 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1 ==== 925511+0+0 (1481054277 0 0) 0x2fe3600 con 0x2fd9080
         -153> 2013-04-25 10:09:23.461847 7f34020fa700  1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x81b9b80
         -152> 2013-04-25 10:09:23.461898 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1
         -151> 2013-04-25 10:09:23.461919 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1
         -150> 2013-04-25 10:09:23.461944 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -149> 2013-04-25 10:09:23.650853 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355052 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x81b9b80 con 0x2fd9340
         -148> 2013-04-25 10:09:23.650917 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1
         -147> 2013-04-25 10:09:23.650933 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1
         -146> 2013-04-25 10:09:23.650950 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971567 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1 ==== 925511+0+0 (4168104339 0 0) 0x81b98c0 con 0x2fd9080
         -145> 2013-04-25 10:09:23.650960 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1
         -144> 2013-04-25 10:09:23.650964 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1
         -143> 2013-04-25 10:09:23.650981 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
         -142> 2013-04-25 10:09:23.878718 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971568 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1 ==== 925511+0+0 (1503552806 0 0) 0x81b9b80 con 0x2fd9080
         -141> 2013-04-25 10:09:23.878778 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1
         -140> 2013-04-25 10:09:23.878793 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1
         -139> 2013-04-25 10:09:23.878817 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -138> 2013-04-25 10:09:24.062604 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971569 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1 ==== 925511+0+0 (2454824331 0 0) 0x81b98c0 con 0x2fd9080
         -137> 2013-04-25 10:09:24.062664 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1
         -136> 2013-04-25 10:09:24.062681 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1
         -135> 2013-04-25 10:09:24.062706 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
         -134> 2013-04-25 10:09:24.256036 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971570 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1 ==== 925511+0+0 (3266358423 0 0) 0x81b9b80 con 0x2fd9080
         -133> 2013-04-25 10:09:24.256101 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1
         -132> 2013-04-25 10:09:24.256118 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1
         -131> 2013-04-25 10:09:24.256143 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -130> 2013-04-25 10:09:24.431386 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971571 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1 ==== 925511+0+0 (210292830 0 0) 0x81b98c0 con 0x2fd9080
         -129> 2013-04-25 10:09:24.431464 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1
         -128> 2013-04-25 10:09:24.431494 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1
         -127> 2013-04-25 10:09:24.431516 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
         -126> 2013-04-25 10:09:24.573611 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971572 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1 ==== 925511+0+0 (2924214099 0 0) 0x81b9b80 con 0x2fd9080
         -125> 2013-04-25 10:09:24.573661 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1
         -124> 2013-04-25 10:09:24.573684 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1
         -123> 2013-04-25 10:09:24.573711 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -122> 2013-04-25 10:09:24.742869 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971573 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1 ==== 925511+0+0 (570858258 0 0) 0x81b98c0 con 0x2fd9080
         -121> 2013-04-25 10:09:24.742913 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1
         -120> 2013-04-25 10:09:24.742930 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1
         -119> 2013-04-25 10:09:24.742948 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
         -118> 2013-04-25 10:09:24.916907 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971574 ==== mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1 ==== 912355+0+0 (1508521429 0 0) 0x81b9b80 con 0x2fd9080
         -117> 2013-04-25 10:09:24.916967 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1
         -116> 2013-04-25 10:09:24.917009 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1
         -115> 2013-04-25 10:09:24.917033 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -114> 2013-04-25 10:09:25.073684 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971575 ==== mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1 ==== 923071+0+0 (596636463 0 0) 0x81b98c0 con 0x2fd9080
         -113> 2013-04-25 10:09:25.073725 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1
         -112> 2013-04-25 10:09:25.073742 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1
         -111> 2013-04-25 10:09:25.073762 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
         -110> 2013-04-25 10:09:25.249451 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971576 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1 ==== 925511+0+0 (2246054051 0 0) 0x81b9b80 con 0x2fd9080
         -109> 2013-04-25 10:09:25.249505 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1
         -108> 2013-04-25 10:09:25.249520 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1
         -107> 2013-04-25 10:09:25.249543 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
         -106> 2013-04-25 10:09:25.449804 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971577 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1 ==== 925511+0+0 (977064735 0 0) 0x81b98c0 con 0x2fd9080
         -105> 2013-04-25 10:09:25.449862 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1
         -104> 2013-04-25 10:09:25.449873 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1
         -103> 2013-04-25 10:09:25.449890 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
         -102> 2013-04-25 10:09:25.600620 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971578 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1 ==== 925511+0+0 (3983611115 0 0) 0x81b9b80 con 0x2fd9080
         -101> 2013-04-25 10:09:25.600668 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1
         -100> 2013-04-25 10:09:25.600683 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1
          -99> 2013-04-25 10:09:25.600707 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
          -98> 2013-04-25 10:09:25.819002 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971579 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1 ==== 925511+0+0 (73955483 0 0) 0x81b98c0 con 0x2fd9080
          -97> 2013-04-25 10:09:25.819073 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1
          -96> 2013-04-25 10:09:25.819097 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1
          -95> 2013-04-25 10:09:25.819122 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
          -94> 2013-04-25 10:09:25.917835 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971580 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1 ==== 925511+0+0 (469562892 0 0) 0x81b9b80 con 0x2fd9080
          -93> 2013-04-25 10:09:25.917895 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1
          -92> 2013-04-25 10:09:25.917914 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1
          -91> 2013-04-25 10:09:25.917939 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
          -90> 2013-04-25 10:09:26.102004 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971581 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1 ==== 925511+0+0 (1611033226 0 0) 0x81b98c0 con 0x2fd9080
          -89> 2013-04-25 10:09:26.102045 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1
          -88> 2013-04-25 10:09:26.102057 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1
          -87> 2013-04-25 10:09:26.102074 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
          -86> 2013-04-25 10:09:26.202867 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971582 ==== mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1 ==== 925602+0+0 (1656611906 0 0) 0x81b9b80 con 0x2fd9080
          -85> 2013-04-25 10:09:26.202905 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1
          -84> 2013-04-25 10:09:26.202921 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1
          -83> 2013-04-25 10:09:26.202940 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
          -82> 2013-04-25 10:09:26.296930 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971583 ==== mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1 ==== 869574+0+0 (742404150 0 0) 0x81b98c0 con 0x2fd9080
          -81> 2013-04-25 10:09:26.296992 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1
          -80> 2013-04-25 10:09:26.297011 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1
          -79> 2013-04-25 10:09:26.297038 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80
          -78> 2013-04-25 10:09:26.454078 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971584 ==== mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1 ==== 396451+0+0 (3256396663 0 0) 0x81b9b80 con 0x2fd9080
          -77> 2013-04-25 10:09:26.454127 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1
          -76> 2013-04-25 10:09:26.454137 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1
          -75> 2013-04-25 10:09:26.454172 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0
          -74> 2013-04-25 10:09:26.629481 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971585 ==== mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1 ==== 1037286+0+0 (659870788 0 0) 0x81b98c0 con 0x2fd9080
          -73> 2013-04-25 10:09:26.629529 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1
          -72> 2013-04-25 10:09:26.629542 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1
          -71> 2013-04-25 10:09:26.629582 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply flags( last ) ) v1 -- ?+0 0x81b9b80
          -70> 2013-04-25 10:09:26.872131 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_stop
          -69> 2013-04-25 10:09:26.872183 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( finish ) v1 -- ?+0 0x2fe3600
          -68> 2013-04-25 10:09:26.873028 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355053 ==== mon_sync( finish_reply ) v1 ==== 174+0+0 (879078845 0 0) 0x2fe3600 con 0x2fd9340
          -67> 2013-04-25 10:09:26.873069 7f34018f9700 10 mon.4@2(synchronizing sync( requester state stop )) e1 handle_sync mon_sync( finish_reply ) v1
          -66> 2013-04-25 10:09:26.873088 7f34018f9700 10 mon.4@2(synchronizing sync( requester state stop )) e1 handle_sync_finish_reply mon_sync( finish_reply ) v1
          -65> 2013-04-25 10:09:27.073657 7f34018f9700 10 mon.4@2(synchronizing) e1 init_paxos
          -64> 2013-04-25 10:09:27.074136 7f34018f9700  7 mon.4@2(synchronizing).pg v0 update_from_paxos loading latest full pgmap v5896128
          -63> 2013-04-25 10:09:27.082000 7f34018f9700  7 mon.4@2(synchronizing).pg v5896128 update_from_paxos  applying incremental 5896129
          -62> 2013-04-25 10:09:27.082392 7f34018f9700 10 mon.4@2(synchronizing).pg v5896129 v5896129: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 4894KB/s wr, 188op/s
          -61> 2013-04-25 10:09:27.082468 7f34018f9700  7 mon.4@2(synchronizing).pg v5896129 update_from_paxos  applying incremental 5896130
          -60> 2013-04-25 10:09:27.082536 7f34018f9700 10 mon.4@2(synchronizing).pg v5896130 v5896130: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 2929KB/s wr, 114op/s
          -59> 2013-04-25 10:09:27.082576 7f34018f9700  7 mon.4@2(synchronizing).pg v5896130 update_from_paxos  applying incremental 5896131
          -58> 2013-04-25 10:09:27.082679 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 v5896131: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 1963KB/s wr, 90op/s
          -57> 2013-04-25 10:09:27.082711 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 send_pg_creates to 0 pgs
          -56> 2013-04-25 10:09:27.082717 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 update_logger
          -55> 2013-04-25 10:09:27.082803 7f34018f9700 10 mon.4@2(synchronizing).mds e0 update_from_paxos version 3711, my e 0
          -54> 2013-04-25 10:09:27.082854 7f34018f9700 10 mon.4@2(synchronizing).mds e0 update_from_paxos got 3711
          -53> 2013-04-25 10:09:27.082890 7f34018f9700  4 mon.4@2(synchronizing).mds e3711 new map
          -52> 2013-04-25 10:09:27.082895 7f34018f9700  7 mon.4@2(synchronizing).mds e3711 print_map
       epoch	3711
       flags	0
       created	2013-01-28 15:10:54.412329
       modified	2013-04-25 10:04:26.660566
       tableserver	0
       root	0
       session_timeout	60
       session_autoclose	300
       last_failure	3694
       last_failure_osd_epoch	10501
       compat	compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding}
       max_mds	1
       in	0
       up	{0=214699}
       failed
       stopped
       data_pools	0,6
       metadata_pool	1
 :	10.227.227.101:6810/6757 '1' mds.0.260 up:active seq 28
 :	10.227.227.103:6800/7999 '3' mds.-1.0 up:standby seq 1
 :	10.227.227.104:6800/31019 '4' mds.-1.0 up:standby seq 1
          -51> 2013-04-25 10:09:27.082955 7f34018f9700 10 mon.4@2(synchronizing).mds e3711 update_logger
          -50> 2013-04-25 10:09:27.083108 7f34018f9700  7 mon.4@2(synchronizing).osd e0 update_from_paxos loading latest full map e10517
          -49> 2013-04-25 10:09:27.083314 7f34018f9700  7 mon.4@2(synchronizing).osd e10517 update_from_paxos  applying incremental 10518
          -48> 2013-04-25 10:09:27.083402 7f34018f9700  1 mon.4@2(synchronizing).osd e10518 e10518: 12 osds: 10 up, 12 in
          -47> 2013-04-25 10:09:27.083440 7f34018f9700  7 mon.4@2(synchronizing).osd e10518 update_from_paxos  applying incremental 10519
          -46> 2013-04-25 10:09:27.083490 7f34018f9700  1 mon.4@2(synchronizing).osd e10519 e10519: 12 osds: 9 up, 12 in
          -45> 2013-04-25 10:09:27.083519 7f34018f9700  7 mon.4@2(synchronizing).osd e10519 update_from_paxos  applying incremental 10520
          -44> 2013-04-25 10:09:27.083565 7f34018f9700  1 mon.4@2(synchronizing).osd e10520 e10520: 12 osds: 9 up, 12 in
          -43> 2013-04-25 10:09:27.083598 7f34018f9700  7 mon.4@2(synchronizing).osd e10520 update_from_paxos  applying incremental 10521
          -42> 2013-04-25 10:09:27.083643 7f34018f9700  1 mon.4@2(synchronizing).osd e10521 e10521: 12 osds: 9 up, 12 in
          -41> 2013-04-25 10:09:27.083672 7f34018f9700  7 mon.4@2(synchronizing).osd e10521 update_from_paxos  applying incremental 10522
          -40> 2013-04-25 10:09:27.083718 7f34018f9700  1 mon.4@2(synchronizing).osd e10522 e10522: 12 osds: 9 up, 12 in
          -39> 2013-04-25 10:09:27.083761 7f34018f9700  7 mon.4@2(synchronizing).osd e10522 update_from_paxos  applying incremental 10523
          -38> 2013-04-25 10:09:27.083808 7f34018f9700  1 mon.4@2(synchronizing).osd e10523 e10523: 12 osds: 9 up, 12 in
          -37> 2013-04-25 10:09:27.083845 7f34018f9700  7 mon.4@2(synchronizing).osd e10523 update_from_paxos  applying incremental 10524
          -36> 2013-04-25 10:09:27.083899 7f34018f9700  1 mon.4@2(synchronizing).osd e10524 e10524: 12 osds: 10 up, 12 in
          -35> 2013-04-25 10:09:27.083934 7f34018f9700  7 mon.4@2(synchronizing).osd e10524 update_from_paxos  applying incremental 10525
          -34> 2013-04-25 10:09:27.083980 7f34018f9700  1 mon.4@2(synchronizing).osd e10525 e10525: 12 osds: 10 up, 12 in
          -33> 2013-04-25 10:09:27.084026 7f34018f9700  7 mon.4@2(synchronizing).osd e10525 update_from_paxos  applying incremental 10526
          -32> 2013-04-25 10:09:27.084077 7f34018f9700  1 mon.4@2(synchronizing).osd e10526 e10526: 12 osds: 12 up, 12 in
          -31> 2013-04-25 10:09:27.084107 7f34018f9700  7 mon.4@2(synchronizing).osd e10526 update_from_paxos  applying incremental 10527
          -30> 2013-04-25 10:09:27.084152 7f34018f9700  1 mon.4@2(synchronizing).osd e10527 e10527: 12 osds: 12 up, 12 in
          -29> 2013-04-25 10:09:27.211801 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 send_to_waiting 10527
          -28> 2013-04-25 10:09:27.211833 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 share_map_with_random_osd no up osd on our session map
          -27> 2013-04-25 10:09:27.211839 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 update_logger
          -26> 2013-04-25 10:09:27.211873 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos
          -25> 2013-04-25 10:09:27.212051 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos version 5569491 summary v 0
          -24> 2013-04-25 10:09:27.212234 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos latest full 5569490
          -23> 2013-04-25 10:09:27.212347 7f34018f9700  7 mon.4@2(synchronizing).log v5569491 update_from_paxos loading summary e5569490
          -22> 2013-04-25 10:09:27.212446 7f34018f9700  7 mon.4@2(synchronizing).log v5569491 update_from_paxos loaded summary e5569490
          -21> 2013-04-25 10:09:27.212631 7f34018f9700  7 mon.4@2(synchronizing).log v5569491 update_from_paxos applying incremental log 5569491 2013-04-25 10:08:59.980554 mon.0 10.227.227.101:6789/0 1031 : [INF] pgmap v5896131: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 1963KB/s wr, 90op/s
          -20> 2013-04-25 10:09:27.212766 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 check_subs
          -19> 2013-04-25 10:09:27.212873 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos
          -18> 2013-04-25 10:09:27.213007 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos version 4309 keys ver 0 latest 4307
          -17> 2013-04-25 10:09:27.213168 7f34018f9700  7 mon.4@2(synchronizing).auth v4309 update_from_paxos loading summary e 4307
          -16> 2013-04-25 10:09:27.213194 7f34018f9700  7 mon.4@2(synchronizing).auth v4309 update_from_paxos latest length 2392
          -15> 2013-04-25 10:09:27.213337 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos key server version 4307
          -14> 2013-04-25 10:09:27.213445 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos() last_allocated_id=215196 max_global_id=215196
          -13> 2013-04-25 10:09:27.213470 7f34018f9700 10 mon.4@2(synchronizing) e1 bootstrap
          -12> 2013-04-25 10:09:27.213475 7f34018f9700 10 mon.4@2(synchronizing) e1 unregister_cluster_logger - not registered
          -11> 2013-04-25 10:09:27.213479 7f34018f9700 10 mon.4@2(synchronizing) e1 cancel_probe_timeout (none scheduled)
          -10> 2013-04-25 10:09:27.213484 7f34018f9700 10 mon.4@2(synchronizing) e1 reset_sync
           -9> 2013-04-25 10:09:27.213489 7f34018f9700 10 mon.4@2(probing) e1 reset
           -8> 2013-04-25 10:09:27.213494 7f34018f9700 10 mon.4@2(probing) e1 timecheck_finish
           -7> 2013-04-25 10:09:27.213507 7f34018f9700 10 mon.4@2(probing) e1 cancel_probe_timeout (none scheduled)
           -6> 2013-04-25 10:09:27.213515 7f34018f9700 10 mon.4@2(probing) e1 reset_probe_timeout 0x3992f70 after 2 seconds
           -5> 2013-04-25 10:09:27.213531 7f34018f9700 10 mon.4@2(probing) e1 probing other monitors
           -4> 2013-04-25 10:09:27.213538 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_probe(probe 5eacb654-3204-4ca1-9b16-83bb0140e497 name 4) v4 -- ?+0 0x8197200
           -3> 2013-04-25 10:09:27.213570 7f34018f9700  1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_probe(probe 5eacb654-3204-4ca1-9b16-83bb0140e497 name 4) v4 -- ?+0 0x8197500
           -2> 2013-04-25 10:09:27.213615 7f34020fa700 10 mon.4@2(probing).data_health(0) service_tick
           -1> 2013-04-25 10:09:27.214449 7f34018f9700  1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355054 ==== mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589981 lc 11590025 )) v4 ==== 563+0+0 (4058502882 0 0) 0x8197200 con 0x2fd9340
 > 2013-04-25 10:09:27.214510 7f34020fa700 -1 mon/MonMap.h: In function 'entity_inst_t MonMap::get_inst(const string&)' thread 7f34020fa700 time 2013-04-25 10:09:27.213645
       mon/MonMap.h: 160: FAILED assert(mon_addr.count(n))
        ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23)
 : (MonMap::get_inst(std::string const&)+0x159) [0x4bf079]
 : (DataHealthService::update_stats()+0x67) [0x59a827]
 : (DataHealthService::service_tick()+0x5a) [0x59ba5a]
 : (QuorumService::C_Tick::finish(int)+0x17) [0x59d0a7]
 : (Context::complete(int)+0xa) [0x4bbf2a]
 : (SafeTimer::timer_thread()+0x42e) [0x696e5e]
 : (SafeTimerThread::entry()+0xd) [0x69845d]
 : (()+0x84f8) [0x7f3458d6d4f8]
 : (clone()+0x6d) [0x7f3456ff276d]
        NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
       --- logging levels ---
 / 5 none
 / 1 lockdep
 / 1 context
 / 1 crush
 / 5 mds
 / 5 mds_balancer
 / 5 mds_locker
 / 5 mds_log
 / 5 mds_log_expire
 / 5 mds_migrator
 / 1 buffer
 / 1 timer
 / 1 filer
 / 1 striper
 / 1 objecter
 / 5 rados
 / 5 rbd
 / 5 journaler
 / 5 objectcacher
 / 5 client
 / 5 osd
 / 5 optracker
 / 5 objclass
 / 3 filestore
 / 3 journal
 / 5 ms
 /10 mon
 /10 monc
 / 5 paxos
 / 5 tp
 / 5 auth
 / 5 crypto
 / 1 finisher
 / 5 heartbeatmap
 / 5 perfcounter
 / 5 rgw
 / 5 hadoop
 / 5 javaclient
 / 5 asok
 / 1 throttle
         -2/-2 (syslog threshold)
         -1/-1 (stderr threshold)
         max_recent     10000
         max_new         1000
         log_file /var/log/ceph/ceph-mon.4.log
       --- end dump of recent events ---
 -04-25 10:09:27.222703 7f34020fa700 -1 *** Caught signal (Aborted) **
        in thread 7f34020fa700
        ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23)
 : /usr/bin/ceph-mon() [0x5b0107]
 : (()+0x10c70) [0x7f3458d75c70]
 : (gsignal()+0x35) [0x7f3456ee6275]
 : (abort()+0x139) [0x7f3456ee7b69]
 : (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f34578a1c9d]
 : (()+0xb9e56) [0x7f345789fe56]
 : (()+0xb9e83) [0x7f345789fe83]
 : (()+0xb9f7e) [0x7f345789ff7e]
 : (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1d3) [0x6a2a33]
 : (MonMap::get_inst(std::string const&)+0x159) [0x4bf079]
 : (DataHealthService::update_stats()+0x67) [0x59a827]
 : (DataHealthService::service_tick()+0x5a) [0x59ba5a]
 : (QuorumService::C_Tick::finish(int)+0x17) [0x59d0a7]
 : (Context::complete(int)+0xa) [0x4bbf2a]
 : (SafeTimer::timer_thread()+0x42e) [0x696e5e]
 : (SafeTimerThread::entry()+0xd) [0x69845d]
 : (()+0x84f8) [0x7f3458d6d4f8]
 : (clone()+0x6d) [0x7f3456ff276d]
        NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
       --- begin dump of recent events ---
 > 2013-04-25 10:09:27.222703 7f34020fa700 -1 *** Caught signal (Aborted) **
        in thread 7f34020fa700
        ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23)
 : /usr/bin/ceph-mon() [0x5b0107]
 : (()+0x10c70) [0x7f3458d75c70]
 : (gsignal()+0x35) [0x7f3456ee6275]
 : (abort()+0x139) [0x7f3456ee7b69]
 : (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f34578a1c9d]
 : (()+0xb9e56) [0x7f345789fe56]
 : (()+0xb9e83) [0x7f345789fe83]
 : (()+0xb9f7e) [0x7f345789ff7e]
 : (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1d3) [0x6a2a33]
 : (MonMap::get_inst(std::string const&)+0x159) [0x4bf079]
 : (DataHealthService::update_stats()+0x67) [0x59a827]
 : (DataHealthService::service_tick()+0x5a) [0x59ba5a]
 : (QuorumService::C_Tick::finish(int)+0x17) [0x59d0a7]
 : (Context::complete(int)+0xa) [0x4bbf2a]
 : (SafeTimer::timer_thread()+0x42e) [0x696e5e]
 : (SafeTimerThread::entry()+0xd) [0x69845d]
 : (()+0x84f8) [0x7f3458d6d4f8]
 : (clone()+0x6d) [0x7f3456ff276d]
        NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
       --- logging levels ---
 / 5 none
 / 1 lockdep
 / 1 context
 / 1 crush
 / 5 mds
 / 5 mds_balancer
 / 5 mds_locker
 / 5 mds_log
 / 5 mds_log_expire
 / 5 mds_migrator
 / 1 buffer
 / 1 timer
 / 1 filer
 / 1 striper
 / 1 objecter
 / 5 rados
 / 5 rbd
 / 5 journaler
 / 5 objectcacher
 / 5 client
 / 5 osd
 / 5 optracker
 / 5 objclass
 / 3 filestore
 / 3 journal
 / 5 ms
 /10 mon
 /10 monc
 / 5 paxos
 / 5 tp
 / 5 auth
 / 5 crypto
 / 1 finisher
 / 5 heartbeatmap
 / 5 perfcounter
 / 5 rgw
 / 5 hadoop
 / 5 javaclient
 / 5 asok
 / 1 throttle
         -2/-2 (syslog threshold)
         -1/-1 (stderr threshold)
         max_recent     10000
         max_new         1000
         log_file /var/log/ceph/ceph-mon.4.log
       --- end dump of recent events ---

« Previous
1
2
3
Next »

(1-1/3)

Project

General

Profile

Ceph

Bug #4811 » ceph-mon.4.log