2013-04-25 10:07:40.073976 7fbce674e800 0 ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23), process ceph-mon, pid 1094 2013-04-25 10:07:49.773487 7f3459191800 0 ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23), process ceph-mon, pid 1215 2013-04-25 10:07:49.773503 7f3459191800 10 needs_conversion 2013-04-25 10:07:50.706758 7f3459191800 10 obtain_monmap 2013-04-25 10:07:57.109537 7f3459191800 10 obtain_monmap detected aborted sync 2013-04-25 10:07:57.109631 7f3459191800 10 obtain_monmap read backup monmap 2013-04-25 10:07:57.109660 7f3459191800 0 mon.4 does not exist in monmap, will attempt to join an existing cluster 2013-04-25 10:07:57.110644 7f3459191800 1 mon.4@-1(probing) e1 preinit fsid 5eacb654-3204-4ca1-9b16-83bb0140e497 2013-04-25 10:07:57.110746 7f3459191800 10 mon.4@-1(probing) e1 check_fsid cluster_uuid contains '5eacb654-3204-4ca1-9b16-83bb0140e497' 2013-04-25 10:07:57.110793 7f3459191800 10 mon.4@-1(probing) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?)} 2013-04-25 10:07:57.110833 7f3459191800 10 mon.4@-1(probing) e1 has_ever_joined = 1 2013-04-25 10:07:57.110863 7f3459191800 1 mon.4@-1(probing) e1 preinit clean up potentially inconsistent store state 2013-04-25 10:08:14.912907 7f3459191800 10 mon.4@-1(probing) e1 init_paxos 2013-04-25 10:08:20.899994 7f3459191800 10 mon.4@-1(probing).log v0 update_from_paxos 2013-04-25 10:08:20.917658 7f3459191800 10 mon.4@-1(probing).log v0 update_from_paxos version 0 summary v 0 2013-04-25 10:08:26.867790 7f3459191800 10 mon.4@-1(probing).auth v0 update_from_paxos 2013-04-25 10:08:27.077147 7f3459191800 10 mon.4@-1(probing).health(0) init 2013-04-25 10:08:27.178455 7f3459191800 10 mon.4@-1(probing) e1 loading initial keyring to bootstrap authentication for mkfs 2013-04-25 10:08:27.181914 7f3459191800 2 mon.4@-1(probing) e1 init 2013-04-25 10:08:27.182092 7f3459191800 10 mon.4@-1(probing) e1 bootstrap 2013-04-25 10:08:27.182098 7f3459191800 10 mon.4@-1(probing) e1 unregister_cluster_logger - not registered 2013-04-25 10:08:27.182102 7f3459191800 10 mon.4@-1(probing) e1 cancel_probe_timeout (none scheduled) 2013-04-25 10:08:27.182107 7f3459191800 0 mon.4@-1(probing) e1 my rank is now 2 (was -1) 2013-04-25 10:08:27.182124 7f3459191800 10 mon.4@2(probing) e1 reset_sync 2013-04-25 10:08:27.182130 7f3459191800 10 mon.4@2(probing) e1 reset 2013-04-25 10:08:27.182132 7f3459191800 10 mon.4@2(probing) e1 timecheck_finish 2013-04-25 10:08:27.182188 7f3459191800 10 mon.4@2(probing) e1 cancel_probe_timeout (none scheduled) 2013-04-25 10:08:27.182199 7f3459191800 10 mon.4@2(probing) e1 reset_probe_timeout 0x3992a80 after 2 seconds 2013-04-25 10:08:27.182281 7f3459191800 10 mon.4@2(probing) e1 probing other monitors 2013-04-25 10:08:27.183091 7f34008f7700 10 mon.4@2(probing) e1 ms_get_authorizer for mon 2013-04-25 10:08:27.183217 7f34007f6700 10 mon.4@2(probing) e1 ms_get_authorizer for mon 2013-04-25 10:08:27.184084 7f34008f7700 10 mon.4@2(probing) e1 ms_get_authorizer for mon 2013-04-25 10:08:27.184099 7f34007f6700 10 mon.4@2(probing) e1 ms_get_authorizer for mon 2013-04-25 10:08:27.185769 7f34018f9700 10 mon.4@2(probing) e1 do not have session, making new one 2013-04-25 10:08:27.185794 7f34018f9700 10 mon.4@2(probing) e1 ms_dispatch new session MonSession: mon.1 10.227.227.103:6789/0 is open for mon.1 10.227.227.103:6789/0 2013-04-25 10:08:27.185802 7f34018f9700 5 mon.4@2(probing) e1 setting monitor caps on this connection 2013-04-25 10:08:27.185810 7f34018f9700 10 mon.4@2(probing) e1 handle_probe mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 2013-04-25 10:08:27.185826 7f34018f9700 10 mon.4@2(probing) e1 handle_probe_reply mon.1 10.227.227.103:6789/0mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 2013-04-25 10:08:27.185837 7f34018f9700 10 mon.4@2(probing) e1 monmap is e1: 3 mons at {0=10.227.227.101:6789/0,1=10.227.227.103:6789/0,2=10.227.227.104:6789/0} 2013-04-25 10:08:27.185880 7f34018f9700 10 mon.4@2(probing) e1 peer name is 1 2013-04-25 10:08:27.185888 7f34018f9700 10 mon.4@2(probing) e1 existing quorum 0,1 2013-04-25 10:08:27.185890 7f34018f9700 10 mon.4@2(probing) e1 peer paxos version 11589955 vs my version 0 (too far ahead) 2013-04-25 10:08:27.185894 7f34018f9700 10 mon.4@2(probing) e1 cancel_probe_timeout 0x3992a80 2013-04-25 10:08:27.185900 7f34018f9700 10 mon.4@2(probing) e1 sync_start entity( mon.1 10.227.227.103:6789/0 ) 2013-04-25 10:08:45.143820 7f34004f3700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.227.227.104:0/168822515 client protocol 0 2013-04-25 10:08:50.650500 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )).monmap v1 get_monmap ver 0 2013-04-25 10:08:56.347480 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_store_init backup current monmap 2013-04-25 10:08:56.364214 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_handle_reset 0x2fd9a20 10.227.227.104:0/168822515 2013-04-25 10:09:02.103859 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 do not have session, making new one 2013-04-25 10:09:02.103994 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_dispatch new session MonSession: mon.0 10.227.227.101:6789/0 is open for mon.0 10.227.227.101:6789/0 2013-04-25 10:09:02.104027 7f34018f9700 5 mon.4@2(synchronizing sync( requester state start )) e1 setting monitor caps on this connection 2013-04-25 10:09:02.104052 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_probe mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 2013-04-25 10:09:02.104085 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_probe_reply mon.0 10.227.227.101:6789/0mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 2013-04-25 10:09:02.104115 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 monmap is e1: 3 mons at {0=10.227.227.101:6789/0,1=10.227.227.103:6789/0,2=10.227.227.104:6789/0} 2013-04-25 10:09:02.104209 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1 2013-04-25 10:09:02.104236 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync_start_reply mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1 2013-04-25 10:09:02.104263 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync_start_reply leader told us to use mon.1 10.227.227.103:6789/0 as sync provider 2013-04-25 10:09:02.104283 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0) 2013-04-25 10:09:02.104340 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_start_chunks provider(mon.1 10.227.227.103:6789/0) 2013-04-25 10:09:02.132721 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 2013-04-25 10:09:02.132731 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 2013-04-25 10:09:12.843986 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1 2013-04-25 10:09:12.844014 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1 2013-04-25 10:09:12.934498 7f34004f3700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.227.227.103:0/18850 client protocol 0 2013-04-25 10:09:13.056263 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0) 2013-04-25 10:09:13.056400 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1 2013-04-25 10:09:13.056425 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1 2013-04-25 10:09:13.150562 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 2013-04-25 10:09:13.150573 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 2013-04-25 10:09:13.150606 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1 2013-04-25 10:09:13.150612 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1 2013-04-25 10:09:13.242184 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1 2013-04-25 10:09:13.242193 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1 2013-04-25 10:09:13.359156 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1 2013-04-25 10:09:13.359164 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1 2013-04-25 10:09:13.443022 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1 2013-04-25 10:09:13.443044 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1 2013-04-25 10:09:13.543178 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1 2013-04-25 10:09:13.543188 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1 2013-04-25 10:09:13.668263 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1 2013-04-25 10:09:13.668273 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1 2013-04-25 10:09:13.785538 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1 2013-04-25 10:09:13.785553 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1 2013-04-25 10:09:13.885302 7f34018f9700 5 mon.4@2(synchronizing sync( requester state chunks )) e1 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum 2013-04-25 10:09:14.192649 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1 2013-04-25 10:09:14.192666 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1 2013-04-25 10:09:14.848570 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1 2013-04-25 10:09:14.848588 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1 2013-04-25 10:09:15.261543 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1 2013-04-25 10:09:15.261568 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1 2013-04-25 10:09:15.367962 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1 2013-04-25 10:09:15.367991 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1 2013-04-25 10:09:15.462411 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1 2013-04-25 10:09:15.462424 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1 2013-04-25 10:09:15.545497 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1 2013-04-25 10:09:15.545514 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1 2013-04-25 10:09:15.933782 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_handle_reset 0x2fd9b80 10.227.227.103:0/18850 2013-04-25 10:09:17.844322 7f34020fa700 1 mon.4@2(synchronizing sync( requester state chunks )) e1 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum 2013-04-25 10:09:18.150668 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0) 2013-04-25 10:09:18.151927 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 2013-04-25 10:09:18.151943 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 2013-04-25 10:09:18.934981 7f34004f3700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.227.227.103:0/18850 client protocol 0 2013-04-25 10:09:18.935425 7f34018f9700 5 mon.4@2(synchronizing sync( requester state chunks )) e1 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum 2013-04-25 10:09:21.283812 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1 2013-04-25 10:09:21.283829 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1 2013-04-25 10:09:21.370114 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1 2013-04-25 10:09:21.370124 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1 2013-04-25 10:09:21.436523 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1 2013-04-25 10:09:21.436537 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1 2013-04-25 10:09:21.537153 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1 2013-04-25 10:09:21.537167 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1 2013-04-25 10:09:21.629180 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1 2013-04-25 10:09:21.629197 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1 2013-04-25 10:09:21.687218 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1 2013-04-25 10:09:21.687229 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1 2013-04-25 10:09:21.763104 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1 2013-04-25 10:09:21.763116 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1 2013-04-25 10:09:21.846705 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1 2013-04-25 10:09:21.846717 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1 2013-04-25 10:09:21.938639 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_handle_reset 0x6506580 10.227.227.103:0/18850 2013-04-25 10:09:21.938700 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1 2013-04-25 10:09:21.938709 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1 2013-04-25 10:09:22.063707 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1 2013-04-25 10:09:22.063719 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1 2013-04-25 10:09:22.155552 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1 2013-04-25 10:09:22.155570 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1 2013-04-25 10:09:22.239128 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1 2013-04-25 10:09:22.239140 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1 2013-04-25 10:09:22.305924 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1 2013-04-25 10:09:22.305936 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1 2013-04-25 10:09:22.389566 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1 2013-04-25 10:09:22.389578 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1 2013-04-25 10:09:22.464955 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1 2013-04-25 10:09:22.464970 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1 2013-04-25 10:09:22.590523 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1 2013-04-25 10:09:22.590534 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1 2013-04-25 10:09:22.682374 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1 2013-04-25 10:09:22.682383 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1 2013-04-25 10:09:22.765907 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1 2013-04-25 10:09:22.765917 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1 2013-04-25 10:09:22.841936 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1 2013-04-25 10:09:22.841953 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1 2013-04-25 10:09:22.933991 7f34020fa700 1 mon.4@2(synchronizing sync( requester state chunks )) e1 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum 2013-04-25 10:09:22.934100 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1 2013-04-25 10:09:22.934123 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1 2013-04-25 10:09:23.461787 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0) 2013-04-25 10:09:23.461898 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1 2013-04-25 10:09:23.461919 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1 2013-04-25 10:09:23.650917 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 2013-04-25 10:09:23.650933 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 2013-04-25 10:09:23.650960 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1 2013-04-25 10:09:23.650964 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1 2013-04-25 10:09:23.878778 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1 2013-04-25 10:09:23.878793 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1 2013-04-25 10:09:24.062664 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1 2013-04-25 10:09:24.062681 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1 2013-04-25 10:09:24.256101 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1 2013-04-25 10:09:24.256118 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1 2013-04-25 10:09:24.431464 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1 2013-04-25 10:09:24.431494 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1 2013-04-25 10:09:24.573661 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1 2013-04-25 10:09:24.573684 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1 2013-04-25 10:09:24.742913 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1 2013-04-25 10:09:24.742930 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1 2013-04-25 10:09:24.916967 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1 2013-04-25 10:09:24.917009 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1 2013-04-25 10:09:25.073725 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1 2013-04-25 10:09:25.073742 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1 2013-04-25 10:09:25.249505 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1 2013-04-25 10:09:25.249520 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1 2013-04-25 10:09:25.449862 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1 2013-04-25 10:09:25.449873 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1 2013-04-25 10:09:25.600668 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1 2013-04-25 10:09:25.600683 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1 2013-04-25 10:09:25.819073 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1 2013-04-25 10:09:25.819097 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1 2013-04-25 10:09:25.917895 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1 2013-04-25 10:09:25.917914 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1 2013-04-25 10:09:26.102045 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1 2013-04-25 10:09:26.102057 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1 2013-04-25 10:09:26.202905 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1 2013-04-25 10:09:26.202921 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1 2013-04-25 10:09:26.296992 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1 2013-04-25 10:09:26.297011 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1 2013-04-25 10:09:26.454127 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1 2013-04-25 10:09:26.454137 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1 2013-04-25 10:09:26.629529 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1 2013-04-25 10:09:26.629542 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1 2013-04-25 10:09:26.872131 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_stop 2013-04-25 10:09:26.873069 7f34018f9700 10 mon.4@2(synchronizing sync( requester state stop )) e1 handle_sync mon_sync( finish_reply ) v1 2013-04-25 10:09:26.873088 7f34018f9700 10 mon.4@2(synchronizing sync( requester state stop )) e1 handle_sync_finish_reply mon_sync( finish_reply ) v1 2013-04-25 10:09:27.073657 7f34018f9700 10 mon.4@2(synchronizing) e1 init_paxos 2013-04-25 10:09:27.074136 7f34018f9700 7 mon.4@2(synchronizing).pg v0 update_from_paxos loading latest full pgmap v5896128 2013-04-25 10:09:27.082000 7f34018f9700 7 mon.4@2(synchronizing).pg v5896128 update_from_paxos applying incremental 5896129 2013-04-25 10:09:27.082392 7f34018f9700 10 mon.4@2(synchronizing).pg v5896129 v5896129: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 4894KB/s wr, 188op/s 2013-04-25 10:09:27.082468 7f34018f9700 7 mon.4@2(synchronizing).pg v5896129 update_from_paxos applying incremental 5896130 2013-04-25 10:09:27.082536 7f34018f9700 10 mon.4@2(synchronizing).pg v5896130 v5896130: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 2929KB/s wr, 114op/s 2013-04-25 10:09:27.082576 7f34018f9700 7 mon.4@2(synchronizing).pg v5896130 update_from_paxos applying incremental 5896131 2013-04-25 10:09:27.082679 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 v5896131: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 1963KB/s wr, 90op/s 2013-04-25 10:09:27.082711 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 send_pg_creates to 0 pgs 2013-04-25 10:09:27.082717 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 update_logger 2013-04-25 10:09:27.082803 7f34018f9700 10 mon.4@2(synchronizing).mds e0 update_from_paxos version 3711, my e 0 2013-04-25 10:09:27.082854 7f34018f9700 10 mon.4@2(synchronizing).mds e0 update_from_paxos got 3711 2013-04-25 10:09:27.082890 7f34018f9700 4 mon.4@2(synchronizing).mds e3711 new map 2013-04-25 10:09:27.082895 7f34018f9700 7 mon.4@2(synchronizing).mds e3711 print_map epoch 3711 flags 0 created 2013-01-28 15:10:54.412329 modified 2013-04-25 10:04:26.660566 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 3694 last_failure_osd_epoch 10501 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=214699} failed stopped data_pools 0,6 metadata_pool 1 214699: 10.227.227.101:6810/6757 '1' mds.0.260 up:active seq 28 214349: 10.227.227.103:6800/7999 '3' mds.-1.0 up:standby seq 1 214977: 10.227.227.104:6800/31019 '4' mds.-1.0 up:standby seq 1 2013-04-25 10:09:27.082955 7f34018f9700 10 mon.4@2(synchronizing).mds e3711 update_logger 2013-04-25 10:09:27.083108 7f34018f9700 7 mon.4@2(synchronizing).osd e0 update_from_paxos loading latest full map e10517 2013-04-25 10:09:27.083314 7f34018f9700 7 mon.4@2(synchronizing).osd e10517 update_from_paxos applying incremental 10518 2013-04-25 10:09:27.083402 7f34018f9700 1 mon.4@2(synchronizing).osd e10518 e10518: 12 osds: 10 up, 12 in 2013-04-25 10:09:27.083440 7f34018f9700 7 mon.4@2(synchronizing).osd e10518 update_from_paxos applying incremental 10519 2013-04-25 10:09:27.083490 7f34018f9700 1 mon.4@2(synchronizing).osd e10519 e10519: 12 osds: 9 up, 12 in 2013-04-25 10:09:27.083519 7f34018f9700 7 mon.4@2(synchronizing).osd e10519 update_from_paxos applying incremental 10520 2013-04-25 10:09:27.083565 7f34018f9700 1 mon.4@2(synchronizing).osd e10520 e10520: 12 osds: 9 up, 12 in 2013-04-25 10:09:27.083598 7f34018f9700 7 mon.4@2(synchronizing).osd e10520 update_from_paxos applying incremental 10521 2013-04-25 10:09:27.083643 7f34018f9700 1 mon.4@2(synchronizing).osd e10521 e10521: 12 osds: 9 up, 12 in 2013-04-25 10:09:27.083672 7f34018f9700 7 mon.4@2(synchronizing).osd e10521 update_from_paxos applying incremental 10522 2013-04-25 10:09:27.083718 7f34018f9700 1 mon.4@2(synchronizing).osd e10522 e10522: 12 osds: 9 up, 12 in 2013-04-25 10:09:27.083761 7f34018f9700 7 mon.4@2(synchronizing).osd e10522 update_from_paxos applying incremental 10523 2013-04-25 10:09:27.083808 7f34018f9700 1 mon.4@2(synchronizing).osd e10523 e10523: 12 osds: 9 up, 12 in 2013-04-25 10:09:27.083845 7f34018f9700 7 mon.4@2(synchronizing).osd e10523 update_from_paxos applying incremental 10524 2013-04-25 10:09:27.083899 7f34018f9700 1 mon.4@2(synchronizing).osd e10524 e10524: 12 osds: 10 up, 12 in 2013-04-25 10:09:27.083934 7f34018f9700 7 mon.4@2(synchronizing).osd e10524 update_from_paxos applying incremental 10525 2013-04-25 10:09:27.083980 7f34018f9700 1 mon.4@2(synchronizing).osd e10525 e10525: 12 osds: 10 up, 12 in 2013-04-25 10:09:27.084026 7f34018f9700 7 mon.4@2(synchronizing).osd e10525 update_from_paxos applying incremental 10526 2013-04-25 10:09:27.084077 7f34018f9700 1 mon.4@2(synchronizing).osd e10526 e10526: 12 osds: 12 up, 12 in 2013-04-25 10:09:27.084107 7f34018f9700 7 mon.4@2(synchronizing).osd e10526 update_from_paxos applying incremental 10527 2013-04-25 10:09:27.084152 7f34018f9700 1 mon.4@2(synchronizing).osd e10527 e10527: 12 osds: 12 up, 12 in 2013-04-25 10:09:27.211801 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 send_to_waiting 10527 2013-04-25 10:09:27.211833 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 share_map_with_random_osd no up osd on our session map 2013-04-25 10:09:27.211839 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 update_logger 2013-04-25 10:09:27.211873 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos 2013-04-25 10:09:27.212051 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos version 5569491 summary v 0 2013-04-25 10:09:27.212234 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos latest full 5569490 2013-04-25 10:09:27.212347 7f34018f9700 7 mon.4@2(synchronizing).log v5569491 update_from_paxos loading summary e5569490 2013-04-25 10:09:27.212446 7f34018f9700 7 mon.4@2(synchronizing).log v5569491 update_from_paxos loaded summary e5569490 2013-04-25 10:09:27.212631 7f34018f9700 7 mon.4@2(synchronizing).log v5569491 update_from_paxos applying incremental log 5569491 2013-04-25 10:08:59.980554 mon.0 10.227.227.101:6789/0 1031 : [INF] pgmap v5896131: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 1963KB/s wr, 90op/s 2013-04-25 10:09:27.212766 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 check_subs 2013-04-25 10:09:27.212873 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos 2013-04-25 10:09:27.213007 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos version 4309 keys ver 0 latest 4307 2013-04-25 10:09:27.213168 7f34018f9700 7 mon.4@2(synchronizing).auth v4309 update_from_paxos loading summary e 4307 2013-04-25 10:09:27.213194 7f34018f9700 7 mon.4@2(synchronizing).auth v4309 update_from_paxos latest length 2392 2013-04-25 10:09:27.213337 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos key server version 4307 2013-04-25 10:09:27.213445 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos() last_allocated_id=215196 max_global_id=215196 2013-04-25 10:09:27.213470 7f34018f9700 10 mon.4@2(synchronizing) e1 bootstrap 2013-04-25 10:09:27.213475 7f34018f9700 10 mon.4@2(synchronizing) e1 unregister_cluster_logger - not registered 2013-04-25 10:09:27.213479 7f34018f9700 10 mon.4@2(synchronizing) e1 cancel_probe_timeout (none scheduled) 2013-04-25 10:09:27.213484 7f34018f9700 10 mon.4@2(synchronizing) e1 reset_sync 2013-04-25 10:09:27.213489 7f34018f9700 10 mon.4@2(probing) e1 reset 2013-04-25 10:09:27.213494 7f34018f9700 10 mon.4@2(probing) e1 timecheck_finish 2013-04-25 10:09:27.213507 7f34018f9700 10 mon.4@2(probing) e1 cancel_probe_timeout (none scheduled) 2013-04-25 10:09:27.213515 7f34018f9700 10 mon.4@2(probing) e1 reset_probe_timeout 0x3992f70 after 2 seconds 2013-04-25 10:09:27.213531 7f34018f9700 10 mon.4@2(probing) e1 probing other monitors 2013-04-25 10:09:27.213615 7f34020fa700 10 mon.4@2(probing).data_health(0) service_tick 2013-04-25 10:09:27.214510 7f34020fa700 -1 mon/MonMap.h: In function 'entity_inst_t MonMap::get_inst(const string&)' thread 7f34020fa700 time 2013-04-25 10:09:27.213645 mon/MonMap.h: 160: FAILED assert(mon_addr.count(n)) ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23) 1: (MonMap::get_inst(std::string const&)+0x159) [0x4bf079] 2: (DataHealthService::update_stats()+0x67) [0x59a827] 3: (DataHealthService::service_tick()+0x5a) [0x59ba5a] 4: (QuorumService::C_Tick::finish(int)+0x17) [0x59d0a7] 5: (Context::complete(int)+0xa) [0x4bbf2a] 6: (SafeTimer::timer_thread()+0x42e) [0x696e5e] 7: (SafeTimerThread::entry()+0xd) [0x69845d] 8: (()+0x84f8) [0x7f3458d6d4f8] 9: (clone()+0x6d) [0x7f3456ff276d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -426> 2013-04-25 10:07:49.771407 7f3459191800 5 asok(0x2fc8000) register_command perfcounters_dump hook 0x2fb2010 -425> 2013-04-25 10:07:49.771448 7f3459191800 5 asok(0x2fc8000) register_command 1 hook 0x2fb2010 -424> 2013-04-25 10:07:49.771462 7f3459191800 5 asok(0x2fc8000) register_command perf dump hook 0x2fb2010 -423> 2013-04-25 10:07:49.771481 7f3459191800 5 asok(0x2fc8000) register_command perfcounters_schema hook 0x2fb2010 -422> 2013-04-25 10:07:49.771485 7f3459191800 5 asok(0x2fc8000) register_command 2 hook 0x2fb2010 -421> 2013-04-25 10:07:49.771487 7f3459191800 5 asok(0x2fc8000) register_command perf schema hook 0x2fb2010 -420> 2013-04-25 10:07:49.771495 7f3459191800 5 asok(0x2fc8000) register_command config show hook 0x2fb2010 -419> 2013-04-25 10:07:49.771503 7f3459191800 5 asok(0x2fc8000) register_command config set hook 0x2fb2010 -418> 2013-04-25 10:07:49.771508 7f3459191800 5 asok(0x2fc8000) register_command log flush hook 0x2fb2010 -417> 2013-04-25 10:07:49.771510 7f3459191800 5 asok(0x2fc8000) register_command log dump hook 0x2fb2010 -416> 2013-04-25 10:07:49.771515 7f3459191800 5 asok(0x2fc8000) register_command log reopen hook 0x2fb2010 -415> 2013-04-25 10:07:49.773487 7f3459191800 0 ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23), process ceph-mon, pid 1215 -414> 2013-04-25 10:07:49.773503 7f3459191800 10 needs_conversion -413> 2013-04-25 10:07:50.706758 7f3459191800 10 obtain_monmap -412> 2013-04-25 10:07:57.109537 7f3459191800 10 obtain_monmap detected aborted sync -411> 2013-04-25 10:07:57.109631 7f3459191800 10 obtain_monmap read backup monmap -410> 2013-04-25 10:07:57.109660 7f3459191800 0 mon.4 does not exist in monmap, will attempt to join an existing cluster -409> 2013-04-25 10:07:57.110550 7f3459191800 1 -- 10.227.227.104:6789/0 learned my addr 10.227.227.104:6789/0 -408> 2013-04-25 10:07:57.110566 7f3459191800 1 accepter.accepter.bind my_inst.addr is 10.227.227.104:6789/0 need_addr=0 -407> 2013-04-25 10:07:57.110586 7f3459191800 5 adding auth protocol: cephx -406> 2013-04-25 10:07:57.110591 7f3459191800 5 adding auth protocol: cephx -405> 2013-04-25 10:07:57.110644 7f3459191800 1 mon.4@-1(probing) e1 preinit fsid 5eacb654-3204-4ca1-9b16-83bb0140e497 -404> 2013-04-25 10:07:57.110746 7f3459191800 10 mon.4@-1(probing) e1 check_fsid cluster_uuid contains '5eacb654-3204-4ca1-9b16-83bb0140e497' -403> 2013-04-25 10:07:57.110793 7f3459191800 10 mon.4@-1(probing) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?)} -402> 2013-04-25 10:07:57.110833 7f3459191800 10 mon.4@-1(probing) e1 has_ever_joined = 1 -401> 2013-04-25 10:07:57.110863 7f3459191800 1 mon.4@-1(probing) e1 preinit clean up potentially inconsistent store state -400> 2013-04-25 10:08:14.912907 7f3459191800 10 mon.4@-1(probing) e1 init_paxos -399> 2013-04-25 10:08:20.899994 7f3459191800 10 mon.4@-1(probing).log v0 update_from_paxos -398> 2013-04-25 10:08:20.917658 7f3459191800 10 mon.4@-1(probing).log v0 update_from_paxos version 0 summary v 0 -397> 2013-04-25 10:08:26.867790 7f3459191800 10 mon.4@-1(probing).auth v0 update_from_paxos -396> 2013-04-25 10:08:27.077147 7f3459191800 10 mon.4@-1(probing).health(0) init -395> 2013-04-25 10:08:27.178455 7f3459191800 10 mon.4@-1(probing) e1 loading initial keyring to bootstrap authentication for mkfs -394> 2013-04-25 10:08:27.179210 7f3459191800 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-4/keyring -393> 2013-04-25 10:08:27.179221 7f3459191800 5 asok(0x2fc8000) register_command mon_status hook 0x3992ae0 -392> 2013-04-25 10:08:27.179229 7f3459191800 5 asok(0x2fc8000) register_command quorum_status hook 0x3992ae0 -391> 2013-04-25 10:08:27.179232 7f3459191800 5 asok(0x2fc8000) register_command sync_status hook 0x3992ae0 -390> 2013-04-25 10:08:27.179235 7f3459191800 5 asok(0x2fc8000) register_command add_bootstrap_peer_hint hook 0x3992ae0 -389> 2013-04-25 10:08:27.181041 7f3459191800 1 finished global_init_daemonize -388> 2013-04-25 10:08:27.181475 7f3459191800 5 asok(0x2fc8000) init /var/run/ceph/ceph-mon.4.asok -387> 2013-04-25 10:08:27.181539 7f3459191800 5 asok(0x2fc8000) bind_and_listen /var/run/ceph/ceph-mon.4.asok -386> 2013-04-25 10:08:27.181642 7f3459191800 5 asok(0x2fc8000) register_command 0 hook 0x2fb15a0 -385> 2013-04-25 10:08:27.181662 7f3459191800 5 asok(0x2fc8000) register_command version hook 0x2fb15a0 -384> 2013-04-25 10:08:27.181682 7f3459191800 5 asok(0x2fc8000) register_command git_version hook 0x2fb15a0 -383> 2013-04-25 10:08:27.181706 7f3459191800 5 asok(0x2fc8000) register_command help hook 0x3992ad0 -382> 2013-04-25 10:08:27.181738 7f34038fd700 5 asok(0x2fc8000) entry start -381> 2013-04-25 10:08:27.181853 7f3459191800 1 -- 10.227.227.104:6789/0 messenger.start -380> 2013-04-25 10:08:27.181914 7f3459191800 2 mon.4@-1(probing) e1 init -379> 2013-04-25 10:08:27.182049 7f3459191800 1 accepter.accepter.start -378> 2013-04-25 10:08:27.182092 7f3459191800 10 mon.4@-1(probing) e1 bootstrap -377> 2013-04-25 10:08:27.182098 7f3459191800 10 mon.4@-1(probing) e1 unregister_cluster_logger - not registered -376> 2013-04-25 10:08:27.182102 7f3459191800 10 mon.4@-1(probing) e1 cancel_probe_timeout (none scheduled) -375> 2013-04-25 10:08:27.182107 7f3459191800 0 mon.4@-1(probing) e1 my rank is now 2 (was -1) -374> 2013-04-25 10:08:27.182112 7f3459191800 1 -- 10.227.227.104:6789/0 mark_down_all -373> 2013-04-25 10:08:27.182124 7f3459191800 10 mon.4@2(probing) e1 reset_sync -372> 2013-04-25 10:08:27.182130 7f3459191800 10 mon.4@2(probing) e1 reset -371> 2013-04-25 10:08:27.182132 7f3459191800 10 mon.4@2(probing) e1 timecheck_finish -370> 2013-04-25 10:08:27.182188 7f3459191800 10 mon.4@2(probing) e1 cancel_probe_timeout (none scheduled) -369> 2013-04-25 10:08:27.182199 7f3459191800 10 mon.4@2(probing) e1 reset_probe_timeout 0x3992a80 after 2 seconds -368> 2013-04-25 10:08:27.182281 7f3459191800 10 mon.4@2(probing) e1 probing other monitors -367> 2013-04-25 10:08:27.182297 7f3459191800 1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_probe(probe 5eacb654-3204-4ca1-9b16-83bb0140e497 name 4) v4 -- ?+0 0x5a9ec00 -366> 2013-04-25 10:08:27.182425 7f3459191800 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_probe(probe 5eacb654-3204-4ca1-9b16-83bb0140e497 name 4) v4 -- ?+0 0x5a9e900 -365> 2013-04-25 10:08:27.183091 7f34008f7700 10 mon.4@2(probing) e1 ms_get_authorizer for mon -364> 2013-04-25 10:08:27.183217 7f34007f6700 10 mon.4@2(probing) e1 ms_get_authorizer for mon -363> 2013-04-25 10:08:27.184084 7f34008f7700 10 mon.4@2(probing) e1 ms_get_authorizer for mon -362> 2013-04-25 10:08:27.184099 7f34007f6700 10 mon.4@2(probing) e1 ms_get_authorizer for mon -361> 2013-04-25 10:08:27.185721 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971530 ==== mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 ==== 563+0+0 (2117610832 0 0) 0x5a9e900 con 0x2fd9080 -360> 2013-04-25 10:08:27.185769 7f34018f9700 10 mon.4@2(probing) e1 do not have session, making new one -359> 2013-04-25 10:08:27.185794 7f34018f9700 10 mon.4@2(probing) e1 ms_dispatch new session MonSession: mon.1 10.227.227.103:6789/0 is open for mon.1 10.227.227.103:6789/0 -358> 2013-04-25 10:08:27.185802 7f34018f9700 5 mon.4@2(probing) e1 setting monitor caps on this connection -357> 2013-04-25 10:08:27.185810 7f34018f9700 10 mon.4@2(probing) e1 handle_probe mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 -356> 2013-04-25 10:08:27.185826 7f34018f9700 10 mon.4@2(probing) e1 handle_probe_reply mon.1 10.227.227.103:6789/0mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 3 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 -355> 2013-04-25 10:08:27.185837 7f34018f9700 10 mon.4@2(probing) e1 monmap is e1: 3 mons at {0=10.227.227.101:6789/0,1=10.227.227.103:6789/0,2=10.227.227.104:6789/0} -354> 2013-04-25 10:08:27.185880 7f34018f9700 10 mon.4@2(probing) e1 peer name is 1 -353> 2013-04-25 10:08:27.185888 7f34018f9700 10 mon.4@2(probing) e1 existing quorum 0,1 -352> 2013-04-25 10:08:27.185890 7f34018f9700 10 mon.4@2(probing) e1 peer paxos version 11589955 vs my version 0 (too far ahead) -351> 2013-04-25 10:08:27.185894 7f34018f9700 10 mon.4@2(probing) e1 cancel_probe_timeout 0x3992a80 -350> 2013-04-25 10:08:27.185900 7f34018f9700 10 mon.4@2(probing) e1 sync_start entity( mon.1 10.227.227.103:6789/0 ) -349> 2013-04-25 10:08:45.143597 7f34004f3700 1 -- 10.227.227.104:6789/0 >> :/0 pipe(0x5304780 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.227.227.104:33345/0 -348> 2013-04-25 10:08:45.143820 7f34004f3700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.227.227.104:0/168822515 client protocol 0 -347> 2013-04-25 10:08:48.143458 7f34004f3700 2 -- 10.227.227.104:6789/0 >> 10.227.227.104:0/168822515 pipe(0x5304780 sd=23 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success -346> 2013-04-25 10:08:48.143529 7f34004f3700 2 -- 10.227.227.104:6789/0 >> 10.227.227.104:0/168822515 pipe(0x5304780 sd=23 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success -345> 2013-04-25 10:08:50.650500 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )).monmap v1 get_monmap ver 0 -344> 2013-04-25 10:08:56.347480 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_store_init backup current monmap -343> 2013-04-25 10:08:56.364109 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( start ) v1 -- ?+0 0x2fe3340 -342> 2013-04-25 10:08:56.364214 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_handle_reset 0x2fd9a20 10.227.227.104:0/168822515 -341> 2013-04-25 10:08:56.364245 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355047 ==== mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 ==== 563+0+0 (3066673873 0 0) 0x5a9ec00 con 0x2fd9340 -340> 2013-04-25 10:09:02.103859 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 do not have session, making new one -339> 2013-04-25 10:09:02.103994 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 ms_dispatch new session MonSession: mon.0 10.227.227.101:6789/0 is open for mon.0 10.227.227.101:6789/0 -338> 2013-04-25 10:09:02.104027 7f34018f9700 5 mon.4@2(synchronizing sync( requester state start )) e1 setting monitor caps on this connection -337> 2013-04-25 10:09:02.104052 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_probe mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 -336> 2013-04-25 10:09:02.104085 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_probe_reply mon.0 10.227.227.101:6789/0mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589919 lc 11589955 )) v4 -335> 2013-04-25 10:09:02.104115 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 monmap is e1: 3 mons at {0=10.227.227.101:6789/0,1=10.227.227.103:6789/0,2=10.227.227.104:6789/0} -334> 2013-04-25 10:09:02.104166 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355048 ==== mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1 ==== 174+0+0 (3214969469 0 0) 0x2fe3600 con 0x2fd9340 -333> 2013-04-25 10:09:02.104209 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1 -332> 2013-04-25 10:09:02.104236 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync_start_reply mon_sync( start_reply flags( reply-to(mon.1 10.227.227.103:6789/0) ) ) v1 -331> 2013-04-25 10:09:02.104263 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 handle_sync_start_reply leader told us to use mon.1 10.227.227.103:6789/0 as sync provider -330> 2013-04-25 10:09:02.104283 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0) -329> 2013-04-25 10:09:02.104305 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x2fe3b80 -328> 2013-04-25 10:09:02.104340 7f34018f9700 10 mon.4@2(synchronizing sync( requester state start )) e1 sync_start_chunks provider(mon.1 10.227.227.103:6789/0) -327> 2013-04-25 10:09:02.104369 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( start_chunks ) v1 -- ?+0 0x2fe38c0 -326> 2013-04-25 10:09:02.132688 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355049 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x2fe3b80 con 0x2fd9340 -325> 2013-04-25 10:09:02.132721 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 -324> 2013-04-25 10:09:02.132731 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 -323> 2013-04-25 10:09:08.069875 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971531 ==== mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1 ==== 1044432+0+0 (1482870314 0 0) 0x2fe38c0 con 0x2fd9080 -322> 2013-04-25 10:09:12.843986 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1 -321> 2013-04-25 10:09:12.844014 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1044242 bytes last_key ( logm,full_5569022 ) ) v1 -320> 2013-04-25 10:09:12.844610 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3b80 -319> 2013-04-25 10:09:12.934223 7f34004f3700 1 -- 10.227.227.104:6789/0 >> :/0 pipe(0x509c280 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.227.227.103:59710/0 -318> 2013-04-25 10:09:12.934498 7f34004f3700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.227.227.103:0/18850 client protocol 0 -317> 2013-04-25 10:09:13.056252 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971532 ==== mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1 ==== 1034782+0+0 (1741418579 0 0) 0x2fe3b80 con 0x2fd9080 -316> 2013-04-25 10:09:13.056263 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0) -315> 2013-04-25 10:09:13.056319 7f34020fa700 1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x43fcdc0 -314> 2013-04-25 10:09:13.056400 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1 -313> 2013-04-25 10:09:13.056425 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1034592 bytes last_key ( logm,full_5569079 ) ) v1 -312> 2013-04-25 10:09:13.056517 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0 -311> 2013-04-25 10:09:13.150519 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355050 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x43fcdc0 con 0x2fd9340 -310> 2013-04-25 10:09:13.150562 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 -309> 2013-04-25 10:09:13.150573 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 -308> 2013-04-25 10:09:13.150594 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971533 ==== mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1 ==== 1033109+0+0 (2445730747 0 0) 0x2fe38c0 con 0x2fd9080 -307> 2013-04-25 10:09:13.150606 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1 -306> 2013-04-25 10:09:13.150612 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1032919 bytes last_key ( logm,full_5569135 ) ) v1 -305> 2013-04-25 10:09:13.150669 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -304> 2013-04-25 10:09:13.242148 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971534 ==== mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1 ==== 1040561+0+0 (2240089044 0 0) 0x2fe3600 con 0x2fd9080 -303> 2013-04-25 10:09:13.242184 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1 -302> 2013-04-25 10:09:13.242193 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1040371 bytes last_key ( logm,full_5569189 ) ) v1 -301> 2013-04-25 10:09:13.242241 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0 -300> 2013-04-25 10:09:13.359125 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971535 ==== mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1 ==== 1040284+0+0 (2564410500 0 0) 0x2fe38c0 con 0x2fd9080 -299> 2013-04-25 10:09:13.359156 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1 -298> 2013-04-25 10:09:13.359164 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1040094 bytes last_key ( logm,full_5569242 ) ) v1 -297> 2013-04-25 10:09:13.359209 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -296> 2013-04-25 10:09:13.442973 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971536 ==== mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1 ==== 1048082+0+0 (1045690206 0 0) 0x2fe3600 con 0x2fd9080 -295> 2013-04-25 10:09:13.443022 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1 -294> 2013-04-25 10:09:13.443044 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047892 bytes last_key ( logm,full_5569306 ) ) v1 -293> 2013-04-25 10:09:13.443132 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0 -292> 2013-04-25 10:09:13.543136 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971537 ==== mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1 ==== 1041708+0+0 (2728244868 0 0) 0x2fe38c0 con 0x2fd9080 -291> 2013-04-25 10:09:13.543178 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1 -290> 2013-04-25 10:09:13.543188 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1041518 bytes last_key ( logm,full_5569368 ) ) v1 -289> 2013-04-25 10:09:13.543253 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -288> 2013-04-25 10:09:13.668228 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971538 ==== mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1 ==== 1038588+0+0 (3496012706 0 0) 0x2fe3600 con 0x2fd9080 -287> 2013-04-25 10:09:13.668263 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1 -286> 2013-04-25 10:09:13.668273 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038398 bytes last_key ( logm,full_5569422 ) ) v1 -285> 2013-04-25 10:09:13.668323 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0 -284> 2013-04-25 10:09:13.785464 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971539 ==== mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1 ==== 1038296+0+0 (2754640597 0 0) 0x2fe38c0 con 0x2fd9080 -283> 2013-04-25 10:09:13.785538 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1 -282> 2013-04-25 10:09:13.785553 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038106 bytes last_key ( logm,full_5569483 ) ) v1 -281> 2013-04-25 10:09:13.785642 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -280> 2013-04-25 10:09:13.885235 7f34018f9700 1 -- 10.227.227.104:6789/0 <== client.? 10.227.227.103:0/18850 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x5b7a900 con 0x2fd9b80 -279> 2013-04-25 10:09:13.885302 7f34018f9700 5 mon.4@2(synchronizing sync( requester state chunks )) e1 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum -278> 2013-04-25 10:09:14.192588 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971540 ==== mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1 ==== 1048582+0+0 (2478673006 0 0) 0x2fe3600 con 0x2fd9080 -277> 2013-04-25 10:09:14.192649 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1 -276> 2013-04-25 10:09:14.192666 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1048398 bytes last_key ( mdsmap,2178 ) ) v1 -275> 2013-04-25 10:09:14.194163 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0 -274> 2013-04-25 10:09:14.848511 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971541 ==== mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1 ==== 1047810+0+0 (766408052 0 0) 0x2fe38c0 con 0x2fd9080 -273> 2013-04-25 10:09:14.848570 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1 -272> 2013-04-25 10:09:14.848588 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047626 bytes last_key ( mdsmap,3671 ) ) v1 -271> 2013-04-25 10:09:14.850593 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -270> 2013-04-25 10:09:15.261469 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971542 ==== mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1 ==== 1042123+0+0 (450065390 0 0) 0x2fe3600 con 0x2fd9080 -269> 2013-04-25 10:09:15.261543 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1 -268> 2013-04-25 10:09:15.261568 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1041933 bytes last_key ( osdmap,full_10017 ) ) v1 -267> 2013-04-25 10:09:15.264252 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0 -266> 2013-04-25 10:09:15.367892 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971543 ==== mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1 ==== 1047344+0+0 (791022580 0 0) 0x2fe38c0 con 0x2fd9080 -265> 2013-04-25 10:09:15.367962 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1 -264> 2013-04-25 10:09:15.367991 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047154 bytes last_key ( osdmap,full_10153 ) ) v1 -263> 2013-04-25 10:09:15.368260 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -262> 2013-04-25 10:09:15.462353 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971544 ==== mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1 ==== 1043142+0+0 (1111262443 0 0) 0x2fe3600 con 0x2fd9080 -261> 2013-04-25 10:09:15.462411 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1 -260> 2013-04-25 10:09:15.462424 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042952 bytes last_key ( osdmap,full_10287 ) ) v1 -259> 2013-04-25 10:09:15.462647 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe38c0 -258> 2013-04-25 10:09:15.545446 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971545 ==== mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1 ==== 1042501+0+0 (1058205326 0 0) 0x2fe38c0 con 0x2fd9080 -257> 2013-04-25 10:09:15.545497 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1 -256> 2013-04-25 10:09:15.545514 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042311 bytes last_key ( osdmap,full_10416 ) ) v1 -255> 2013-04-25 10:09:15.545659 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -254> 2013-04-25 10:09:15.933605 7f34004f3700 2 -- 10.227.227.104:6789/0 >> 10.227.227.103:0/18850 pipe(0x509c280 sd=23 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success -253> 2013-04-25 10:09:15.933672 7f34004f3700 2 -- 10.227.227.104:6789/0 >> 10.227.227.103:0/18850 pipe(0x509c280 sd=23 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success -252> 2013-04-25 10:09:15.933782 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_handle_reset 0x2fd9b80 10.227.227.103:0/18850 -251> 2013-04-25 10:09:17.844322 7f34020fa700 1 mon.4@2(synchronizing sync( requester state chunks )) e1 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum -250> 2013-04-25 10:09:17.844347 7f34020fa700 1 -- 10.227.227.104:6789/0 mark_down 0x2fd9b80 -- pipe dne -249> 2013-04-25 10:09:18.150668 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0) -248> 2013-04-25 10:09:18.150723 7f34020fa700 1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x81b98c0 -247> 2013-04-25 10:09:18.151875 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355051 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x81b98c0 con 0x2fd9340 -246> 2013-04-25 10:09:18.151927 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 -245> 2013-04-25 10:09:18.151943 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 -244> 2013-04-25 10:09:18.934744 7f34004f3700 1 -- 10.227.227.104:6789/0 >> :/0 pipe(0x509c000 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.227.227.103:59712/0 -243> 2013-04-25 10:09:18.934981 7f34004f3700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_verify_authorizer 10.227.227.103:0/18850 client protocol 0 -242> 2013-04-25 10:09:18.935384 7f34018f9700 1 -- 10.227.227.104:6789/0 <== client.? 10.227.227.103:0/18850 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x38a4000 con 0x6506580 -241> 2013-04-25 10:09:18.935425 7f34018f9700 5 mon.4@2(synchronizing sync( requester state chunks )) e1 waitlisting message auth(proto 0 30 bytes epoch 0) v1 until we get in quorum -240> 2013-04-25 10:09:21.283756 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971546 ==== mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1 ==== 1026551+0+0 (4095556378 0 0) 0x2fe3600 con 0x2fd9080 -239> 2013-04-25 10:09:21.283812 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1 -238> 2013-04-25 10:09:21.283829 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1026364 bytes last_key ( paxos,11589986 ) ) v1 -237> 2013-04-25 10:09:21.283941 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -236> 2013-04-25 10:09:21.370070 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971547 ==== mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1 ==== 268501+0+0 (83636244 0 0) 0x81b98c0 con 0x2fd9080 -235> 2013-04-25 10:09:21.370114 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1 -234> 2013-04-25 10:09:21.370124 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 268314 bytes last_key ( paxos,11589999 ) ) v1 -233> 2013-04-25 10:09:21.370150 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -232> 2013-04-25 10:09:21.436481 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971548 ==== mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1 ==== 1048515+0+0 (518301646 0 0) 0x2fe3600 con 0x2fd9080 -231> 2013-04-25 10:09:21.436523 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1 -230> 2013-04-25 10:09:21.436537 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1048329 bytes last_key ( pgmap,5895627 ) ) v1 -229> 2013-04-25 10:09:21.436566 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -228> 2013-04-25 10:09:21.537106 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971549 ==== mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1 ==== 984553+0+0 (1049056081 0 0) 0x81b98c0 con 0x2fd9080 -227> 2013-04-25 10:09:21.537153 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1 -226> 2013-04-25 10:09:21.537167 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 984367 bytes last_key ( pgmap,5895650 ) ) v1 -225> 2013-04-25 10:09:21.537204 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -224> 2013-04-25 10:09:21.629137 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971550 ==== mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1 ==== 891521+0+0 (2899987343 0 0) 0x2fe3600 con 0x2fd9080 -223> 2013-04-25 10:09:21.629180 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1 -222> 2013-04-25 10:09:21.629197 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 891335 bytes last_key ( pgmap,5895663 ) ) v1 -221> 2013-04-25 10:09:21.629238 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -220> 2013-04-25 10:09:21.687180 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971551 ==== mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1 ==== 1047566+0+0 (2939424964 0 0) 0x81b98c0 con 0x2fd9080 -219> 2013-04-25 10:09:21.687218 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1 -218> 2013-04-25 10:09:21.687229 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1047380 bytes last_key ( pgmap,5895761 ) ) v1 -217> 2013-04-25 10:09:21.687327 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -216> 2013-04-25 10:09:21.763061 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971552 ==== mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1 ==== 1036708+0+0 (3106831870 0 0) 0x2fe3600 con 0x2fd9080 -215> 2013-04-25 10:09:21.763104 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1 -214> 2013-04-25 10:09:21.763116 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1036522 bytes last_key ( pgmap,5895842 ) ) v1 -213> 2013-04-25 10:09:21.763184 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -212> 2013-04-25 10:09:21.846663 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971553 ==== mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1 ==== 1042318+0+0 (2130255415 0 0) 0x81b98c0 con 0x2fd9080 -211> 2013-04-25 10:09:21.846705 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1 -210> 2013-04-25 10:09:21.846717 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1042132 bytes last_key ( pgmap,5895882 ) ) v1 -209> 2013-04-25 10:09:21.846784 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -208> 2013-04-25 10:09:21.934957 7f34004f3700 2 -- 10.227.227.104:6789/0 >> 10.227.227.103:0/18850 pipe(0x509c000 sd=23 :6789 s=2 pgs=3 cs=1 l=1).reader couldn't read tag, Success -207> 2013-04-25 10:09:21.935022 7f34004f3700 2 -- 10.227.227.104:6789/0 >> 10.227.227.103:0/18850 pipe(0x509c000 sd=23 :6789 s=2 pgs=3 cs=1 l=1).fault 0: Success -206> 2013-04-25 10:09:21.938639 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 ms_handle_reset 0x6506580 10.227.227.103:0/18850 -205> 2013-04-25 10:09:21.938678 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971554 ==== mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1 ==== 970349+0+0 (2769782619 0 0) 0x2fe3600 con 0x2fd9080 -204> 2013-04-25 10:09:21.938700 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1 -203> 2013-04-25 10:09:21.938709 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 970163 bytes last_key ( pgmap,5895910 ) ) v1 -202> 2013-04-25 10:09:21.938763 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -201> 2013-04-25 10:09:22.063667 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971555 ==== mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1 ==== 1025785+0+0 (2957537854 0 0) 0x81b98c0 con 0x2fd9080 -200> 2013-04-25 10:09:22.063707 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1 -199> 2013-04-25 10:09:22.063719 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1025599 bytes last_key ( pgmap,5895917 ) ) v1 -198> 2013-04-25 10:09:22.063744 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -197> 2013-04-25 10:09:22.155494 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971556 ==== mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1 ==== 1038617+0+0 (2914685442 0 0) 0x2fe3600 con 0x2fd9080 -196> 2013-04-25 10:09:22.155552 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1 -195> 2013-04-25 10:09:22.155570 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1038431 bytes last_key ( pgmap,5895926 ) ) v1 -194> 2013-04-25 10:09:22.155617 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -193> 2013-04-25 10:09:22.239085 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971557 ==== mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1 ==== 812850+0+0 (2145814566 0 0) 0x81b98c0 con 0x2fd9080 -192> 2013-04-25 10:09:22.239128 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1 -191> 2013-04-25 10:09:22.239140 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 812664 bytes last_key ( pgmap,5895945 ) ) v1 -190> 2013-04-25 10:09:22.239173 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -189> 2013-04-25 10:09:22.305883 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971558 ==== mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1 ==== 964205+0+0 (1601467175 0 0) 0x2fe3600 con 0x2fd9080 -188> 2013-04-25 10:09:22.305924 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1 -187> 2013-04-25 10:09:22.305936 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 964019 bytes last_key ( pgmap,5895957 ) ) v1 -186> 2013-04-25 10:09:22.305962 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -185> 2013-04-25 10:09:22.389528 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971559 ==== mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1 ==== 1046519+0+0 (3493477841 0 0) 0x81b98c0 con 0x2fd9080 -184> 2013-04-25 10:09:22.389566 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1 -183> 2013-04-25 10:09:22.389578 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1046333 bytes last_key ( pgmap,5896001 ) ) v1 -182> 2013-04-25 10:09:22.389632 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -181> 2013-04-25 10:09:22.464896 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971560 ==== mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1 ==== 1043572+0+0 (83147344 0 0) 0x2fe3600 con 0x2fd9080 -180> 2013-04-25 10:09:22.464955 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1 -179> 2013-04-25 10:09:22.464970 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1043386 bytes last_key ( pgmap,5896071 ) ) v1 -178> 2013-04-25 10:09:22.465055 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -177> 2013-04-25 10:09:22.590481 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971561 ==== mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1 ==== 1005179+0+0 (3321238167 0 0) 0x81b98c0 con 0x2fd9080 -176> 2013-04-25 10:09:22.590523 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1 -175> 2013-04-25 10:09:22.590534 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 1004993 bytes last_key ( pgmap,5896102 ) ) v1 -174> 2013-04-25 10:09:22.590574 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -173> 2013-04-25 10:09:22.682332 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971562 ==== mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1 ==== 845548+0+0 (2536188459 0 0) 0x2fe3600 con 0x2fd9080 -172> 2013-04-25 10:09:22.682374 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1 -171> 2013-04-25 10:09:22.682383 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 845354 bytes last_key ( pgmap,first_committed ) ) v1 -170> 2013-04-25 10:09:22.682421 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -169> 2013-04-25 10:09:22.765867 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971563 ==== mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1 ==== 914359+0+0 (140853062 0 0) 0x81b98c0 con 0x2fd9080 -168> 2013-04-25 10:09:22.765907 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1 -167> 2013-04-25 10:09:22.765917 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 914168 bytes last_key ( pgmap,full_5895634 ) ) v1 -166> 2013-04-25 10:09:22.765934 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -165> 2013-04-25 10:09:22.841875 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971564 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1 ==== 925511+0+0 (3220308 0 0) 0x2fe3600 con 0x2fd9080 -164> 2013-04-25 10:09:22.841936 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1 -163> 2013-04-25 10:09:22.841953 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895660 ) ) v1 -162> 2013-04-25 10:09:22.841978 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -161> 2013-04-25 10:09:22.933748 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971565 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1 ==== 925511+0+0 (405716569 0 0) 0x81b98c0 con 0x2fd9080 -160> 2013-04-25 10:09:22.933991 7f34020fa700 1 mon.4@2(synchronizing sync( requester state chunks )) e1 discarding message auth(proto 0 30 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum -159> 2013-04-25 10:09:22.934028 7f34020fa700 1 -- 10.227.227.104:6789/0 mark_down 0x6506580 -- pipe dne -158> 2013-04-25 10:09:22.934100 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1 -157> 2013-04-25 10:09:22.934123 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895686 ) ) v1 -156> 2013-04-25 10:09:22.934142 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x2fe3600 -155> 2013-04-25 10:09:23.461787 7f34020fa700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_send_heartbeat mon.0 10.227.227.101:6789/0 reply(0) -154> 2013-04-25 10:09:23.461789 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971566 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1 ==== 925511+0+0 (1481054277 0 0) 0x2fe3600 con 0x2fd9080 -153> 2013-04-25 10:09:23.461847 7f34020fa700 1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x81b9b80 -152> 2013-04-25 10:09:23.461898 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1 -151> 2013-04-25 10:09:23.461919 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895712 ) ) v1 -150> 2013-04-25 10:09:23.461944 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -149> 2013-04-25 10:09:23.650853 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355052 ==== mon_sync( heartbeat_reply ) v1 ==== 174+0+0 (1026169709 0 0) 0x81b9b80 con 0x2fd9340 -148> 2013-04-25 10:09:23.650917 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat_reply ) v1 -147> 2013-04-25 10:09:23.650933 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat_reply mon_sync( heartbeat_reply ) v1 -146> 2013-04-25 10:09:23.650950 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971567 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1 ==== 925511+0+0 (4168104339 0 0) 0x81b98c0 con 0x2fd9080 -145> 2013-04-25 10:09:23.650960 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1 -144> 2013-04-25 10:09:23.650964 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895738 ) ) v1 -143> 2013-04-25 10:09:23.650981 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -142> 2013-04-25 10:09:23.878718 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971568 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1 ==== 925511+0+0 (1503552806 0 0) 0x81b9b80 con 0x2fd9080 -141> 2013-04-25 10:09:23.878778 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1 -140> 2013-04-25 10:09:23.878793 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895764 ) ) v1 -139> 2013-04-25 10:09:23.878817 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -138> 2013-04-25 10:09:24.062604 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971569 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1 ==== 925511+0+0 (2454824331 0 0) 0x81b98c0 con 0x2fd9080 -137> 2013-04-25 10:09:24.062664 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1 -136> 2013-04-25 10:09:24.062681 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895790 ) ) v1 -135> 2013-04-25 10:09:24.062706 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -134> 2013-04-25 10:09:24.256036 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971570 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1 ==== 925511+0+0 (3266358423 0 0) 0x81b9b80 con 0x2fd9080 -133> 2013-04-25 10:09:24.256101 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1 -132> 2013-04-25 10:09:24.256118 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895816 ) ) v1 -131> 2013-04-25 10:09:24.256143 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -130> 2013-04-25 10:09:24.431386 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971571 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1 ==== 925511+0+0 (210292830 0 0) 0x81b98c0 con 0x2fd9080 -129> 2013-04-25 10:09:24.431464 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1 -128> 2013-04-25 10:09:24.431494 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895842 ) ) v1 -127> 2013-04-25 10:09:24.431516 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -126> 2013-04-25 10:09:24.573611 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971572 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1 ==== 925511+0+0 (2924214099 0 0) 0x81b9b80 con 0x2fd9080 -125> 2013-04-25 10:09:24.573661 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1 -124> 2013-04-25 10:09:24.573684 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895868 ) ) v1 -123> 2013-04-25 10:09:24.573711 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -122> 2013-04-25 10:09:24.742869 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971573 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1 ==== 925511+0+0 (570858258 0 0) 0x81b98c0 con 0x2fd9080 -121> 2013-04-25 10:09:24.742913 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1 -120> 2013-04-25 10:09:24.742930 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895894 ) ) v1 -119> 2013-04-25 10:09:24.742948 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -118> 2013-04-25 10:09:24.916907 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971574 ==== mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1 ==== 912355+0+0 (1508521429 0 0) 0x81b9b80 con 0x2fd9080 -117> 2013-04-25 10:09:24.916967 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1 -116> 2013-04-25 10:09:24.917009 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 912164 bytes last_key ( pgmap,full_5895920 ) ) v1 -115> 2013-04-25 10:09:24.917033 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -114> 2013-04-25 10:09:25.073684 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971575 ==== mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1 ==== 923071+0+0 (596636463 0 0) 0x81b98c0 con 0x2fd9080 -113> 2013-04-25 10:09:25.073725 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1 -112> 2013-04-25 10:09:25.073742 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 922880 bytes last_key ( pgmap,full_5895946 ) ) v1 -111> 2013-04-25 10:09:25.073762 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -110> 2013-04-25 10:09:25.249451 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971576 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1 ==== 925511+0+0 (2246054051 0 0) 0x81b9b80 con 0x2fd9080 -109> 2013-04-25 10:09:25.249505 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1 -108> 2013-04-25 10:09:25.249520 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895972 ) ) v1 -107> 2013-04-25 10:09:25.249543 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -106> 2013-04-25 10:09:25.449804 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971577 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1 ==== 925511+0+0 (977064735 0 0) 0x81b98c0 con 0x2fd9080 -105> 2013-04-25 10:09:25.449862 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1 -104> 2013-04-25 10:09:25.449873 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5895998 ) ) v1 -103> 2013-04-25 10:09:25.449890 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -102> 2013-04-25 10:09:25.600620 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971578 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1 ==== 925511+0+0 (3983611115 0 0) 0x81b9b80 con 0x2fd9080 -101> 2013-04-25 10:09:25.600668 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1 -100> 2013-04-25 10:09:25.600683 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896024 ) ) v1 -99> 2013-04-25 10:09:25.600707 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -98> 2013-04-25 10:09:25.819002 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971579 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1 ==== 925511+0+0 (73955483 0 0) 0x81b98c0 con 0x2fd9080 -97> 2013-04-25 10:09:25.819073 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1 -96> 2013-04-25 10:09:25.819097 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896050 ) ) v1 -95> 2013-04-25 10:09:25.819122 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -94> 2013-04-25 10:09:25.917835 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971580 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1 ==== 925511+0+0 (469562892 0 0) 0x81b9b80 con 0x2fd9080 -93> 2013-04-25 10:09:25.917895 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1 -92> 2013-04-25 10:09:25.917914 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896076 ) ) v1 -91> 2013-04-25 10:09:25.917939 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -90> 2013-04-25 10:09:26.102004 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971581 ==== mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1 ==== 925511+0+0 (1611033226 0 0) 0x81b98c0 con 0x2fd9080 -89> 2013-04-25 10:09:26.102045 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1 -88> 2013-04-25 10:09:26.102057 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925320 bytes last_key ( pgmap,full_5896102 ) ) v1 -87> 2013-04-25 10:09:26.102074 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -86> 2013-04-25 10:09:26.202867 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971582 ==== mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1 ==== 925602+0+0 (1656611906 0 0) 0x81b9b80 con 0x2fd9080 -85> 2013-04-25 10:09:26.202905 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1 -84> 2013-04-25 10:09:26.202921 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 925409 bytes last_key ( pgmap,last_committed ) ) v1 -83> 2013-04-25 10:09:26.202940 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -82> 2013-04-25 10:09:26.296930 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971583 ==== mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1 ==== 869574+0+0 (742404150 0 0) 0x81b98c0 con 0x2fd9080 -81> 2013-04-25 10:09:26.296992 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1 -80> 2013-04-25 10:09:26.297011 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 869389 bytes last_key ( pgmap,latest ) ) v1 -79> 2013-04-25 10:09:26.297038 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b9b80 -78> 2013-04-25 10:09:26.454078 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971584 ==== mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1 ==== 396451+0+0 (3256396663 0 0) 0x81b9b80 con 0x2fd9080 -77> 2013-04-25 10:09:26.454127 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1 -76> 2013-04-25 10:09:26.454137 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk bl 396264 bytes last_key ( paxos,11589999 ) ) v1 -75> 2013-04-25 10:09:26.454172 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply ) v1 -- ?+0 0x81b98c0 -74> 2013-04-25 10:09:26.629481 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.1 10.227.227.103:6789/0 1295971585 ==== mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1 ==== 1037286+0+0 (659870788 0 0) 0x81b98c0 con 0x2fd9080 -73> 2013-04-25 10:09:26.629529 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1 -72> 2013-04-25 10:09:26.629542 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 handle_sync_chunk mon_sync( chunk v 11590005 flags( last ) bl 1037100 bytes last_key ( paxos,last_pn ) ) v1 -71> 2013-04-25 10:09:26.629582 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_sync( chunk_reply flags( last ) ) v1 -- ?+0 0x81b9b80 -70> 2013-04-25 10:09:26.872131 7f34018f9700 10 mon.4@2(synchronizing sync( requester state chunks )) e1 sync_stop -69> 2013-04-25 10:09:26.872183 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_sync( finish ) v1 -- ?+0 0x2fe3600 -68> 2013-04-25 10:09:26.873028 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355053 ==== mon_sync( finish_reply ) v1 ==== 174+0+0 (879078845 0 0) 0x2fe3600 con 0x2fd9340 -67> 2013-04-25 10:09:26.873069 7f34018f9700 10 mon.4@2(synchronizing sync( requester state stop )) e1 handle_sync mon_sync( finish_reply ) v1 -66> 2013-04-25 10:09:26.873088 7f34018f9700 10 mon.4@2(synchronizing sync( requester state stop )) e1 handle_sync_finish_reply mon_sync( finish_reply ) v1 -65> 2013-04-25 10:09:27.073657 7f34018f9700 10 mon.4@2(synchronizing) e1 init_paxos -64> 2013-04-25 10:09:27.074136 7f34018f9700 7 mon.4@2(synchronizing).pg v0 update_from_paxos loading latest full pgmap v5896128 -63> 2013-04-25 10:09:27.082000 7f34018f9700 7 mon.4@2(synchronizing).pg v5896128 update_from_paxos applying incremental 5896129 -62> 2013-04-25 10:09:27.082392 7f34018f9700 10 mon.4@2(synchronizing).pg v5896129 v5896129: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 4894KB/s wr, 188op/s -61> 2013-04-25 10:09:27.082468 7f34018f9700 7 mon.4@2(synchronizing).pg v5896129 update_from_paxos applying incremental 5896130 -60> 2013-04-25 10:09:27.082536 7f34018f9700 10 mon.4@2(synchronizing).pg v5896130 v5896130: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 2929KB/s wr, 114op/s -59> 2013-04-25 10:09:27.082576 7f34018f9700 7 mon.4@2(synchronizing).pg v5896130 update_from_paxos applying incremental 5896131 -58> 2013-04-25 10:09:27.082679 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 v5896131: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 1963KB/s wr, 90op/s -57> 2013-04-25 10:09:27.082711 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 send_pg_creates to 0 pgs -56> 2013-04-25 10:09:27.082717 7f34018f9700 10 mon.4@2(synchronizing).pg v5896131 update_logger -55> 2013-04-25 10:09:27.082803 7f34018f9700 10 mon.4@2(synchronizing).mds e0 update_from_paxos version 3711, my e 0 -54> 2013-04-25 10:09:27.082854 7f34018f9700 10 mon.4@2(synchronizing).mds e0 update_from_paxos got 3711 -53> 2013-04-25 10:09:27.082890 7f34018f9700 4 mon.4@2(synchronizing).mds e3711 new map -52> 2013-04-25 10:09:27.082895 7f34018f9700 7 mon.4@2(synchronizing).mds e3711 print_map epoch 3711 flags 0 created 2013-01-28 15:10:54.412329 modified 2013-04-25 10:04:26.660566 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 3694 last_failure_osd_epoch 10501 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=214699} failed stopped data_pools 0,6 metadata_pool 1 214699: 10.227.227.101:6810/6757 '1' mds.0.260 up:active seq 28 214349: 10.227.227.103:6800/7999 '3' mds.-1.0 up:standby seq 1 214977: 10.227.227.104:6800/31019 '4' mds.-1.0 up:standby seq 1 -51> 2013-04-25 10:09:27.082955 7f34018f9700 10 mon.4@2(synchronizing).mds e3711 update_logger -50> 2013-04-25 10:09:27.083108 7f34018f9700 7 mon.4@2(synchronizing).osd e0 update_from_paxos loading latest full map e10517 -49> 2013-04-25 10:09:27.083314 7f34018f9700 7 mon.4@2(synchronizing).osd e10517 update_from_paxos applying incremental 10518 -48> 2013-04-25 10:09:27.083402 7f34018f9700 1 mon.4@2(synchronizing).osd e10518 e10518: 12 osds: 10 up, 12 in -47> 2013-04-25 10:09:27.083440 7f34018f9700 7 mon.4@2(synchronizing).osd e10518 update_from_paxos applying incremental 10519 -46> 2013-04-25 10:09:27.083490 7f34018f9700 1 mon.4@2(synchronizing).osd e10519 e10519: 12 osds: 9 up, 12 in -45> 2013-04-25 10:09:27.083519 7f34018f9700 7 mon.4@2(synchronizing).osd e10519 update_from_paxos applying incremental 10520 -44> 2013-04-25 10:09:27.083565 7f34018f9700 1 mon.4@2(synchronizing).osd e10520 e10520: 12 osds: 9 up, 12 in -43> 2013-04-25 10:09:27.083598 7f34018f9700 7 mon.4@2(synchronizing).osd e10520 update_from_paxos applying incremental 10521 -42> 2013-04-25 10:09:27.083643 7f34018f9700 1 mon.4@2(synchronizing).osd e10521 e10521: 12 osds: 9 up, 12 in -41> 2013-04-25 10:09:27.083672 7f34018f9700 7 mon.4@2(synchronizing).osd e10521 update_from_paxos applying incremental 10522 -40> 2013-04-25 10:09:27.083718 7f34018f9700 1 mon.4@2(synchronizing).osd e10522 e10522: 12 osds: 9 up, 12 in -39> 2013-04-25 10:09:27.083761 7f34018f9700 7 mon.4@2(synchronizing).osd e10522 update_from_paxos applying incremental 10523 -38> 2013-04-25 10:09:27.083808 7f34018f9700 1 mon.4@2(synchronizing).osd e10523 e10523: 12 osds: 9 up, 12 in -37> 2013-04-25 10:09:27.083845 7f34018f9700 7 mon.4@2(synchronizing).osd e10523 update_from_paxos applying incremental 10524 -36> 2013-04-25 10:09:27.083899 7f34018f9700 1 mon.4@2(synchronizing).osd e10524 e10524: 12 osds: 10 up, 12 in -35> 2013-04-25 10:09:27.083934 7f34018f9700 7 mon.4@2(synchronizing).osd e10524 update_from_paxos applying incremental 10525 -34> 2013-04-25 10:09:27.083980 7f34018f9700 1 mon.4@2(synchronizing).osd e10525 e10525: 12 osds: 10 up, 12 in -33> 2013-04-25 10:09:27.084026 7f34018f9700 7 mon.4@2(synchronizing).osd e10525 update_from_paxos applying incremental 10526 -32> 2013-04-25 10:09:27.084077 7f34018f9700 1 mon.4@2(synchronizing).osd e10526 e10526: 12 osds: 12 up, 12 in -31> 2013-04-25 10:09:27.084107 7f34018f9700 7 mon.4@2(synchronizing).osd e10526 update_from_paxos applying incremental 10527 -30> 2013-04-25 10:09:27.084152 7f34018f9700 1 mon.4@2(synchronizing).osd e10527 e10527: 12 osds: 12 up, 12 in -29> 2013-04-25 10:09:27.211801 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 send_to_waiting 10527 -28> 2013-04-25 10:09:27.211833 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 share_map_with_random_osd no up osd on our session map -27> 2013-04-25 10:09:27.211839 7f34018f9700 10 mon.4@2(synchronizing).osd e10527 update_logger -26> 2013-04-25 10:09:27.211873 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos -25> 2013-04-25 10:09:27.212051 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos version 5569491 summary v 0 -24> 2013-04-25 10:09:27.212234 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 update_from_paxos latest full 5569490 -23> 2013-04-25 10:09:27.212347 7f34018f9700 7 mon.4@2(synchronizing).log v5569491 update_from_paxos loading summary e5569490 -22> 2013-04-25 10:09:27.212446 7f34018f9700 7 mon.4@2(synchronizing).log v5569491 update_from_paxos loaded summary e5569490 -21> 2013-04-25 10:09:27.212631 7f34018f9700 7 mon.4@2(synchronizing).log v5569491 update_from_paxos applying incremental log 5569491 2013-04-25 10:08:59.980554 mon.0 10.227.227.101:6789/0 1031 : [INF] pgmap v5896131: 2428 pgs: 2428 active+clean; 1816 GB data, 3160 GB used, 13583 GB / 16744 GB avail; 1963KB/s wr, 90op/s -20> 2013-04-25 10:09:27.212766 7f34018f9700 10 mon.4@2(synchronizing).log v5569491 check_subs -19> 2013-04-25 10:09:27.212873 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos -18> 2013-04-25 10:09:27.213007 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos version 4309 keys ver 0 latest 4307 -17> 2013-04-25 10:09:27.213168 7f34018f9700 7 mon.4@2(synchronizing).auth v4309 update_from_paxos loading summary e 4307 -16> 2013-04-25 10:09:27.213194 7f34018f9700 7 mon.4@2(synchronizing).auth v4309 update_from_paxos latest length 2392 -15> 2013-04-25 10:09:27.213337 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos key server version 4307 -14> 2013-04-25 10:09:27.213445 7f34018f9700 10 mon.4@2(synchronizing).auth v4309 update_from_paxos() last_allocated_id=215196 max_global_id=215196 -13> 2013-04-25 10:09:27.213470 7f34018f9700 10 mon.4@2(synchronizing) e1 bootstrap -12> 2013-04-25 10:09:27.213475 7f34018f9700 10 mon.4@2(synchronizing) e1 unregister_cluster_logger - not registered -11> 2013-04-25 10:09:27.213479 7f34018f9700 10 mon.4@2(synchronizing) e1 cancel_probe_timeout (none scheduled) -10> 2013-04-25 10:09:27.213484 7f34018f9700 10 mon.4@2(synchronizing) e1 reset_sync -9> 2013-04-25 10:09:27.213489 7f34018f9700 10 mon.4@2(probing) e1 reset -8> 2013-04-25 10:09:27.213494 7f34018f9700 10 mon.4@2(probing) e1 timecheck_finish -7> 2013-04-25 10:09:27.213507 7f34018f9700 10 mon.4@2(probing) e1 cancel_probe_timeout (none scheduled) -6> 2013-04-25 10:09:27.213515 7f34018f9700 10 mon.4@2(probing) e1 reset_probe_timeout 0x3992f70 after 2 seconds -5> 2013-04-25 10:09:27.213531 7f34018f9700 10 mon.4@2(probing) e1 probing other monitors -4> 2013-04-25 10:09:27.213538 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.0 10.227.227.101:6789/0 -- mon_probe(probe 5eacb654-3204-4ca1-9b16-83bb0140e497 name 4) v4 -- ?+0 0x8197200 -3> 2013-04-25 10:09:27.213570 7f34018f9700 1 -- 10.227.227.104:6789/0 --> mon.1 10.227.227.103:6789/0 -- mon_probe(probe 5eacb654-3204-4ca1-9b16-83bb0140e497 name 4) v4 -- ?+0 0x8197500 -2> 2013-04-25 10:09:27.213615 7f34020fa700 10 mon.4@2(probing).data_health(0) service_tick -1> 2013-04-25 10:09:27.214449 7f34018f9700 1 -- 10.227.227.104:6789/0 <== mon.0 10.227.227.101:6789/0 46355054 ==== mon_probe(reply 5eacb654-3204-4ca1-9b16-83bb0140e497 name 1 quorum 0,1 paxos( fc 11589981 lc 11590025 )) v4 ==== 563+0+0 (4058502882 0 0) 0x8197200 con 0x2fd9340 0> 2013-04-25 10:09:27.214510 7f34020fa700 -1 mon/MonMap.h: In function 'entity_inst_t MonMap::get_inst(const string&)' thread 7f34020fa700 time 2013-04-25 10:09:27.213645 mon/MonMap.h: 160: FAILED assert(mon_addr.count(n)) ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23) 1: (MonMap::get_inst(std::string const&)+0x159) [0x4bf079] 2: (DataHealthService::update_stats()+0x67) [0x59a827] 3: (DataHealthService::service_tick()+0x5a) [0x59ba5a] 4: (QuorumService::C_Tick::finish(int)+0x17) [0x59d0a7] 5: (Context::complete(int)+0xa) [0x4bbf2a] 6: (SafeTimer::timer_thread()+0x42e) [0x696e5e] 7: (SafeTimerThread::entry()+0xd) [0x69845d] 8: (()+0x84f8) [0x7f3458d6d4f8] 9: (clone()+0x6d) [0x7f3456ff276d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 10/10 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mon.4.log --- end dump of recent events --- 2013-04-25 10:09:27.222703 7f34020fa700 -1 *** Caught signal (Aborted) ** in thread 7f34020fa700 ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23) 1: /usr/bin/ceph-mon() [0x5b0107] 2: (()+0x10c70) [0x7f3458d75c70] 3: (gsignal()+0x35) [0x7f3456ee6275] 4: (abort()+0x139) [0x7f3456ee7b69] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f34578a1c9d] 6: (()+0xb9e56) [0x7f345789fe56] 7: (()+0xb9e83) [0x7f345789fe83] 8: (()+0xb9f7e) [0x7f345789ff7e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1d3) [0x6a2a33] 10: (MonMap::get_inst(std::string const&)+0x159) [0x4bf079] 11: (DataHealthService::update_stats()+0x67) [0x59a827] 12: (DataHealthService::service_tick()+0x5a) [0x59ba5a] 13: (QuorumService::C_Tick::finish(int)+0x17) [0x59d0a7] 14: (Context::complete(int)+0xa) [0x4bbf2a] 15: (SafeTimer::timer_thread()+0x42e) [0x696e5e] 16: (SafeTimerThread::entry()+0xd) [0x69845d] 17: (()+0x84f8) [0x7f3458d6d4f8] 18: (clone()+0x6d) [0x7f3456ff276d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2013-04-25 10:09:27.222703 7f34020fa700 -1 *** Caught signal (Aborted) ** in thread 7f34020fa700 ceph version 0.60-653-gf480484 (f4804849b7644f2c1dfd92404682f510a88e9a23) 1: /usr/bin/ceph-mon() [0x5b0107] 2: (()+0x10c70) [0x7f3458d75c70] 3: (gsignal()+0x35) [0x7f3456ee6275] 4: (abort()+0x139) [0x7f3456ee7b69] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f34578a1c9d] 6: (()+0xb9e56) [0x7f345789fe56] 7: (()+0xb9e83) [0x7f345789fe83] 8: (()+0xb9f7e) [0x7f345789ff7e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1d3) [0x6a2a33] 10: (MonMap::get_inst(std::string const&)+0x159) [0x4bf079] 11: (DataHealthService::update_stats()+0x67) [0x59a827] 12: (DataHealthService::service_tick()+0x5a) [0x59ba5a] 13: (QuorumService::C_Tick::finish(int)+0x17) [0x59d0a7] 14: (Context::complete(int)+0xa) [0x4bbf2a] 15: (SafeTimer::timer_thread()+0x42e) [0x696e5e] 16: (SafeTimerThread::entry()+0xd) [0x69845d] 17: (()+0x84f8) [0x7f3458d6d4f8] 18: (clone()+0x6d) [0x7f3456ff276d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 10/10 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mon.4.log --- end dump of recent events ---