Bug #4793
Updated by Joao Eduardo Luis about 11 years ago
During the process of attempting to sync a new or behind monitor with 0.60, I have seen mon/Monitor.cc: 1126: FAILED assert(!(sync_role & SYNC_ROLE_REQUESTER)). This assert is confirmed by another user (matt_ on irc) with paxos quorum issues in 0.60. <pre> root@node2:/var/log/ceph# tail -300 ceph-mon.a.log.1 -141> 2013-04-20 17:35:04.307290 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9c4f5340 10.1.0.8:0/1004379 -140> 2013-04-20 17:35:04.307294 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9c4f58c0 10.1.0.7:6806/16865 -139> 2013-04-20 17:35:04.307298 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9194a160 10.1.0.71:6806/32767 -138> 2013-04-20 17:35:04.307301 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9194ac60 10.1.0.67:0/1255 -137> 2013-04-20 17:35:04.307305 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9194af20 10.1.0.67:0/1271 -136> 2013-04-20 17:35:04.307309 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9194ba20 10.1.0.130:0/15508 -135> 2013-04-20 17:35:04.307314 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4e000 10.1.0.67:0/1283 -134> 2013-04-20 17:35:04.307319 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4e580 10.1.0.135:0/1028470 -133> 2013-04-20 17:35:04.307323 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4eb00 10.1.0.135:0/2028470 -132> 2013-04-20 17:35:04.307327 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4f340 10.1.0.8:6803/24818 -131> 2013-04-20 17:35:04.307330 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4f8c0 10.1.0.130:0/15518 -130> 2013-04-20 17:35:04.307334 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9de8a6e0 10.1.0.6:6803/30723 -129> 2013-04-20 17:35:04.307338 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9de8a160 10.1.0.137:6803/5123 -128> 2013-04-20 17:35:04.307342 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9de8ac60 10.1.0.130:0/15528 -127> 2013-04-20 17:35:04.307346 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9de8b4a0 10.1.0.68:6806/30592 -126> 2013-04-20 17:35:04.307351 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x91ba82c0 10.1.0.3:0/23812 -125> 2013-04-20 17:35:04.307357 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x91ba8840 10.1.0.134:6806/4204 -124> 2013-04-20 17:35:04.307361 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x91ba8dc0 10.1.0.4:6803/29678 -123> 2013-04-20 17:35:04.307366 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x91ba9340 10.1.0.9:6806/17004 -122> 2013-04-20 17:35:04.307372 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x91ba9b80 10.1.0.6:6806/31726 -121> 2013-04-20 17:35:04.307375 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x91ba9e40 10.1.0.68:6803/30491 -120> 2013-04-20 17:35:04.307379 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4ac60 10.1.0.69:6803/8122 -119> 2013-04-20 17:35:04.307384 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4af20 10.1.0.130:6800/5691 -118> 2013-04-20 17:35:04.307388 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4b1e0 10.1.0.4:6806/29925 -117> 2013-04-20 17:35:04.307392 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f4b760 10.1.0.6:6800/30645 -116> 2013-04-20 17:35:04.307397 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9d6a62c0 10.1.0.4:6800/29454 -115> 2013-04-20 17:35:04.307401 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9d6a6580 10.1.0.136:6806/16387 -114> 2013-04-20 17:35:04.307405 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9d6a6840 10.1.0.70:6806/17398 -113> 2013-04-20 17:35:04.307411 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f61a20 10.1.0.9:6800/16811 -112> 2013-04-20 17:35:04.307415 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9ab86000 10.1.0.72:6803/5685 -111> 2013-04-20 17:35:04.307420 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9ab86b00 10.1.0.3:6803/3002 -110> 2013-04-20 17:35:04.307424 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x9ab87b80 10.1.0.5:6800/14631 -109> 2013-04-20 17:35:04.307428 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x97e0c420 10.1.0.8:6800/24740 -108> 2013-04-20 17:35:04.307432 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x97e0cf20 10.1.0.130:0/15439 -107> 2013-04-20 17:35:04.307436 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x97e0d4a0 10.1.0.69:6806/8200 -106> 2013-04-20 17:35:04.307441 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0x97e0da20 10.1.0.70:6803/17296 -105> 2013-04-20 17:35:04.307445 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f5a2c0 10.1.0.72:6800/5606 -104> 2013-04-20 17:35:04.307450 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 ms_handle_reset 0xa6f5a840 10.1.0.135:6800/15336 -103> 2013-04-20 17:35:04.307455 7f905a33d700 1 -- 10.1.0.3:6789/0 <== mon.1 10.1.0.67:6789/0 393371 ==== mon_health( service 1 op tell e 0 r 0 flags ) v1 ==== 62+0+0 (3701682562 0 0) 0x40543c0 con 0x2a83080 -102> 2013-04-20 17:35:04.307463 7f905a33d700 20 mon.a@0(probing sync( leader state none )) e1 have connection -101> 2013-04-20 17:35:04.307465 7f905a33d700 20 mon.a@0(probing sync( leader state none )) e1 ms_dispatch existing session MonSession: mon.1 10.1.0.67:6789/0 is openallow * for mon.1 10.1.0.67:6789/0 -100> 2013-04-20 17:35:04.307470 7f905a33d700 20 mon.a@0(probing sync( leader state none )) e1 caps allow * -99> 2013-04-20 17:35:04.307474 7f905a33d700 10 mon.a@0(probing sync( leader state none )).data_health(596) service_dispatch mon_health( service 1 op tell e 0 r 0 flags ) v1 -98> 2013-04-20 17:35:04.307477 7f905a33d700 1 mon.a@0(probing sync( leader state none )).data_health(596) service_dispatch not in quorum -- drop message -97> 2013-04-20 17:35:04.307481 7f905a33d700 1 -- 10.1.0.3:6789/0 <== mon.2 10.1.0.130:6789/0 1645580482 ==== mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name c quorum 0,1,2 paxos( fc 429002 lc 429022 )) v4 ==== 567+0+0 (2281760352 0 0) 0x282a600 con 0x2a82dc0 -96> 2013-04-20 17:35:04.307492 7f905a33d700 20 mon.a@0(probing sync( leader state none )) e1 have connection -95> 2013-04-20 17:35:04.307494 7f905a33d700 20 mon.a@0(probing sync( leader state none )) e1 ms_dispatch existing session MonSession: mon.2 10.1.0.130:6789/0 is openallow * for mon.2 10.1.0.130:6789/0 -94> 2013-04-20 17:35:04.307499 7f905a33d700 20 mon.a@0(probing sync( leader state none )) e1 caps allow * -93> 2013-04-20 17:35:04.307502 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 handle_probe mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name c quorum 0,1,2 paxos( fc 429002 lc 429022 )) v4 -92> 2013-04-20 17:35:04.307509 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 handle_probe_reply mon.2 10.1.0.130:6789/0mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name c quorum 0,1,2 paxos( fc 429002 lc 429022 )) v4 -91> 2013-04-20 17:35:04.307515 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 monmap is e1: 3 mons at {a=10.1.0.3:6789/0,b=10.1.0.67:6789/0,c=10.1.0.130:6789/0} -90> 2013-04-20 17:35:04.307531 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 peer name is c -89> 2013-04-20 17:35:04.307535 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 existing quorum 0,1,2 -88> 2013-04-20 17:35:04.307537 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 peer paxos version 429022 vs my version 428984 (too far ahead) -87> 2013-04-20 17:35:04.307541 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 cancel_probe_timeout 0xa6ad1650 -86> 2013-04-20 17:35:04.307544 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 sync_start entity( mon.2 10.1.0.130:6789/0 ) -85> 2013-04-20 17:35:04.307548 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 sync_start we are acting as a leader to someone; destroy their dreams -84> 2013-04-20 17:35:04.307552 7f905a33d700 10 mon.a@0(probing sync( leader state none )) e1 reset_sync -83> 2013-04-20 17:35:04.386956 7f90559bf700 2 -- 10.1.0.3:6789/0 >> 10.1.0.135:6806/15524 pipe(0x1b4c9b80 sd=43 :6789 s=2 pgs=2621 cs=1 l=1).reader couldn't read tag, Success -82> 2013-04-20 17:35:04.387041 7f90559bf700 2 -- 10.1.0.3:6789/0 >> 10.1.0.135:6806/15524 pipe(0x1b4c9b80 sd=43 :6789 s=2 pgs=2621 cs=1 l=1).fault 0: Success -81> 2013-04-20 17:35:04.395992 7f90559bf700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c8c80 sd=34 :6789 s=0 pgs=0 cs=0 l=0).accept sd=34 10.1.0.132:39671/0 -80> 2013-04-20 17:35:04.396254 7f90559bf700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.132:6803/31996 osd protocol 0 -79> 2013-04-20 17:35:04.396759 7f904d31b700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:0/1224 pipe(0x26ff680 sd=44 :6789 s=2 pgs=95 cs=1 l=1).reader couldn't read tag, Success -78> 2013-04-20 17:35:04.396790 7f904d31b700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:0/1224 pipe(0x26ff680 sd=44 :6789 s=2 pgs=95 cs=1 l=1).fault 0: Success -77> 2013-04-20 17:35:04.459945 7f9051793700 2 -- 10.1.0.3:6789/0 >> 10.1.0.135:6803/15419 pipe(0x6d709680 sd=24 :6789 s=2 pgs=2744 cs=1 l=1).reader couldn't read tag, Success -76> 2013-04-20 17:35:04.460035 7f9051793700 2 -- 10.1.0.3:6789/0 >> 10.1.0.135:6803/15419 pipe(0x6d709680 sd=24 :6789 s=2 pgs=2744 cs=1 l=1).fault 0: Success -75> 2013-04-20 17:35:04.472004 7f9051793700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c9400 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.1.0.134:42894/0 -74> 2013-04-20 17:35:04.472324 7f9051793700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.134:6803/4103 osd protocol 0 -73> 2013-04-20 17:35:04.506702 7f904d31b700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c8a00 sd=36 :6789 s=0 pgs=0 cs=0 l=0).accept sd=36 10.1.0.133:51138/0 -72> 2013-04-20 17:35:04.506983 7f904d31b700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.133:6800/18670 osd protocol 0 -71> 2013-04-20 17:35:04.514085 7f905271d700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c8000 sd=37 :6789 s=0 pgs=0 cs=0 l=0).accept sd=37 10.1.0.137:40819/0 -70> 2013-04-20 17:35:04.514316 7f905271d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.137:6800/5044 osd protocol 0 -69> 2013-04-20 17:35:04.535680 7f9050884700 2 -- 10.1.0.3:6789/0 >> 10.1.0.7:6800/16705 pipe(0x26ff900 sd=22 :6789 s=2 pgs=2372 cs=1 l=1).reader couldn't read tag, Success -68> 2013-04-20 17:35:04.535766 7f9050884700 2 -- 10.1.0.3:6789/0 >> 10.1.0.7:6800/16705 pipe(0x26ff900 sd=22 :6789 s=2 pgs=2372 cs=1 l=1).fault 0: Success -67> 2013-04-20 17:35:04.672830 7f9051591700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:6806/23784 pipe(0x26fea00 sd=23 :6789 s=2 pgs=2362 cs=1 l=1).reader couldn't read tag, Success -66> 2013-04-20 17:35:04.672884 7f9051591700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:6806/23784 pipe(0x26fea00 sd=23 :6789 s=2 pgs=2362 cs=1 l=1).fault 0: Success -65> 2013-04-20 17:35:05.071029 7f9054331700 2 -- 10.1.0.3:6789/0 >> 10.1.0.70:6800/17218 pipe(0x6d709b80 sd=5 :6789 s=2 pgs=2533 cs=1 l=1).reader couldn't read tag, Success -64> 2013-04-20 17:35:05.071120 7f9054331700 2 -- 10.1.0.3:6789/0 >> 10.1.0.70:6800/17218 pipe(0x6d709b80 sd=5 :6789 s=2 pgs=2533 cs=1 l=1).fault 0: Success -63> 2013-04-20 17:35:05.190050 7f9054331700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c9900 sd=5 :6789 s=0 pgs=0 cs=0 l=0).accept sd=5 10.1.0.132:39672/0 -62> 2013-04-20 17:35:05.190327 7f9054331700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.132:6800/31916 osd protocol 0 -61> 2013-04-20 17:35:05.374483 7f9051591700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c8f00 sd=22 :6789 s=0 pgs=0 cs=0 l=0).accept sd=22 10.1.0.72:51607/0 -60> 2013-04-20 17:35:05.374750 7f9051591700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.72:6806/5787 osd protocol 0 -59> 2013-04-20 17:35:05.473245 7f904f8b0700 2 -- 10.1.0.3:6789/0 >> 10.1.0.133:6806/18842 pipe(0x4f59b80 sd=25 :6789 s=2 pgs=2451 cs=1 l=1).reader couldn't read tag, Success -58> 2013-04-20 17:35:05.473297 7f904f8b0700 2 -- 10.1.0.3:6789/0 >> 10.1.0.133:6806/18842 pipe(0x4f59b80 sd=25 :6789 s=2 pgs=2451 cs=1 l=1).fault 0: Success -57> 2013-04-20 17:35:05.531613 7f905399f700 2 -- 10.1.0.3:6789/0 >> 10.1.0.133:6803/18749 pipe(0x4f58a00 sd=26 :6789 s=2 pgs=2616 cs=1 l=1).reader couldn't read tag, Success -56> 2013-04-20 17:35:05.531669 7f905399f700 2 -- 10.1.0.3:6789/0 >> 10.1.0.133:6803/18749 pipe(0x4f58a00 sd=26 :6789 s=2 pgs=2616 cs=1 l=1).fault 0: Success -55> 2013-04-20 17:35:05.558114 7f905399f700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c9680 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.1.0.4:39717/0 -54> 2013-04-20 17:35:05.558439 7f905399f700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.4:6806/29925 osd protocol 0 -53> 2013-04-20 17:35:05.622034 7f904f8b0700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c9180 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.1.0.7:41548/0 -52> 2013-04-20 17:35:05.622306 7f904f8b0700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.7:6803/16784 osd protocol 0 -51> 2013-04-20 17:35:05.744325 7f9054533700 2 -- 10.1.0.3:6789/0 >> 10.1.0.8:6806/24896 pipe(0x4f58f00 sd=28 :6789 s=2 pgs=2613 cs=1 l=1).reader couldn't read tag, Success -50> 2013-04-20 17:35:05.744473 7f9054533700 2 -- 10.1.0.3:6789/0 >> 10.1.0.8:6806/24896 pipe(0x4f58f00 sd=28 :6789 s=2 pgs=2613 cs=1 l=1).fault 0: Success -49> 2013-04-20 17:35:05.770280 7f905de48700 2 -- 10.1.0.3:6789/0 >> 10.1.0.9:6803/16890 pipe(0x8e42c280 sd=32 :6789 s=2 pgs=2419 cs=1 l=1).reader couldn't read tag, Success -48> 2013-04-20 17:35:05.770330 7f905de48700 2 -- 10.1.0.3:6789/0 >> 10.1.0.9:6803/16890 pipe(0x8e42c280 sd=32 :6789 s=2 pgs=2419 cs=1 l=1).fault 0: Success -47> 2013-04-20 17:35:05.894104 7f9054735700 2 -- 10.1.0.3:6789/0 >> 10.1.0.8:0/1004379 pipe(0x8e42c780 sd=33 :6789 s=2 pgs=5402 cs=1 l=1).reader couldn't read tag, Success -46> 2013-04-20 17:35:05.894156 7f9054735700 2 -- 10.1.0.3:6789/0 >> 10.1.0.8:0/1004379 pipe(0x8e42c780 sd=33 :6789 s=2 pgs=5402 cs=1 l=1).fault 0: Success -45> 2013-04-20 17:35:05.932337 7f9054735700 1 -- 10.1.0.3:6789/0 >> :/0 pipe(0x1b4c8280 sd=26 :6789 s=0 pgs=0 cs=0 l=0).accept sd=26 10.1.0.7:41549/0 -44> 2013-04-20 17:35:05.932591 7f9054735700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_verify_authorizer 10.1.0.7:6806/16865 osd protocol 0 -43> 2013-04-20 17:35:06.039307 7f905a33d700 1 -- 10.1.0.3:6789/0 --> mon.2 10.1.0.130:6789/0 -- mon_sync( start ) v1 -- ?+0 0x5674dc0 -42> 2013-04-20 17:35:06.039372 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0xa6f5b340 10.1.0.135:6806/15524 -41> 2013-04-20 17:35:06.039396 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0xa6f5b600 10.1.0.67:0/1224 -40> 2013-04-20 17:35:06.039404 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0xa0c35b80 10.1.0.135:6803/15419 -39> 2013-04-20 17:35:06.039413 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0xa6f591e0 10.1.0.7:6800/16705 -38> 2013-04-20 17:35:06.039420 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0xa6f594a0 10.1.0.67:6806/23784 -37> 2013-04-20 17:35:06.039426 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0xa0c35600 10.1.0.70:6800/17218 -36> 2013-04-20 17:35:06.039433 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0x4fb58160 10.1.0.133:6806/18842 -35> 2013-04-20 17:35:06.039444 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0x4fb589a0 10.1.0.133:6803/18749 -34> 2013-04-20 17:35:06.039452 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0x4fb58c60 10.1.0.8:6806/24896 -33> 2013-04-20 17:35:06.039460 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0x4fb594a0 10.1.0.9:6803/16890 -32> 2013-04-20 17:35:06.039466 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 ms_handle_reset 0x4fb59ce0 10.1.0.8:0/1004379 -31> 2013-04-20 17:35:06.039473 7f905a33d700 1 -- 10.1.0.3:6789/0 <== mon.0 10.1.0.3:6789/0 0 ==== mon_sync( start_reply ) v1 ==== 0+0+0 (0 0 0) 0x13f68840 con 0x2652420 -30> 2013-04-20 17:35:06.039480 7f905a33d700 20 mon.a@0(synchronizing sync( requester state start )) e1 have connection -29> 2013-04-20 17:35:06.039483 7f905a33d700 20 mon.a@0(synchronizing sync( requester state start )) e1 ms_dispatch existing session MonSession: mon.0 10.1.0.3:6789/0 is openallow * for mon.0 10.1.0.3:6789/0 -28> 2013-04-20 17:35:06.039490 7f905a33d700 20 mon.a@0(synchronizing sync( requester state start )) e1 caps allow * -27> 2013-04-20 17:35:06.039492 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 handle_sync mon_sync( start_reply ) v1 -26> 2013-04-20 17:35:06.039497 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 handle_sync_start_reply mon_sync( start_reply ) v1 -25> 2013-04-20 17:35:06.039517 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 handle_sync_start_reply synchronizing from leader at mon.0 10.1.0.3:6789/0 -24> 2013-04-20 17:35:06.039523 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 sync_send_heartbeat mon.0 10.1.0.3:6789/0 reply(0) -23> 2013-04-20 17:35:06.039528 7f905a33d700 1 -- 10.1.0.3:6789/0 --> mon.0 10.1.0.3:6789/0 -- mon_sync( heartbeat ) v1 -- ?+0 0x5202b00 -22> 2013-04-20 17:35:06.039536 7f905a33d700 10 mon.a@0(synchronizing sync( requester state start )) e1 sync_start_chunks provider(mon.0 10.1.0.3:6789/0) -21> 2013-04-20 17:35:06.039543 7f905a33d700 1 -- 10.1.0.3:6789/0 --> mon.0 10.1.0.3:6789/0 -- mon_sync( start_chunks ) v1 -- ?+0 0x1b561340 -20> 2013-04-20 17:35:06.039550 7f905a33d700 1 -- 10.1.0.3:6789/0 <== mon.1 10.1.0.67:6789/0 393372 ==== mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name b quorum 0,1,2 paxos( fc 429002 lc 429022 )) v4 ==== 567+0+0 (92848010 0 0) 0x282a900 con 0x2a83080 -19> 2013-04-20 17:35:06.039562 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection -18> 2013-04-20 17:35:06.039565 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.1 10.1.0.67:6789/0 is openallow * for mon.1 10.1.0.67:6789/0 -17> 2013-04-20 17:35:06.039570 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * -16> 2013-04-20 17:35:06.039572 7f905a33d700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_probe mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name b quorum 0,1,2 paxos( fc 429002 lc 429022 )) v4 -15> 2013-04-20 17:35:06.039579 7f905a33d700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_probe_reply mon.1 10.1.0.67:6789/0mon_probe(reply 2f2730c5-0504-4433-ae0b-331dd41d99a4 name b quorum 0,1,2 paxos( fc 429002 lc 429022 )) v4 -14> 2013-04-20 17:35:06.039585 7f905a33d700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 monmap is e1: 3 mons at {a=10.1.0.3:6789/0,b=10.1.0.67:6789/0,c=10.1.0.130:6789/0} -13> 2013-04-20 17:35:06.039594 7f905a33d700 1 -- 10.1.0.3:6789/0 <== mon.0 10.1.0.3:6789/0 0 ==== mon_sync( heartbeat ) v1 ==== 0+0+0 (0 0 0) 0x5202b00 con 0x2652420 -12> 2013-04-20 17:35:06.039600 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection -11> 2013-04-20 17:35:06.039602 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.0 10.1.0.3:6789/0 is openallow * for mon.0 10.1.0.3:6789/0 -10> 2013-04-20 17:35:06.039607 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * -9> 2013-04-20 17:35:06.039609 7f905a33d700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( heartbeat ) v1 -8> 2013-04-20 17:35:06.039614 7f905a33d700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat mon_sync( heartbeat ) v1 -7> 2013-04-20 17:35:06.039617 7f905a33d700 1 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_heartbeat ignored stray message mon_sync( heartbeat ) v1 -6> 2013-04-20 17:35:06.039621 7f905a33d700 1 -- 10.1.0.3:6789/0 <== mon.0 10.1.0.3:6789/0 0 ==== mon_sync( start_chunks ) v1 ==== 0+0+0 (0 0 0) 0x1b561340 con 0x2652420 -5> 2013-04-20 17:35:06.039626 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 have connection -4> 2013-04-20 17:35:06.039628 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 ms_dispatch existing session MonSession: mon.0 10.1.0.3:6789/0 is openallow * for mon.0 10.1.0.3:6789/0 -3> 2013-04-20 17:35:06.039634 7f905a33d700 20 mon.a@0(synchronizing sync( requester state chunks )) e1 caps allow * -2> 2013-04-20 17:35:06.039639 7f905a33d700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync mon_sync( start_chunks ) v1 -1> 2013-04-20 17:35:06.039643 7f905a33d700 10 mon.a@0(synchronizing sync( requester state chunks )) e1 handle_sync_start_chunks mon_sync( start_chunks ) v1 0> 2013-04-20 17:35:06.040463 7f905a33d700 -1 mon/Monitor.cc: In function 'void Monitor::handle_sync_start_chunks(MMonSync*)' thread 7f905a33d700 time 2013-04-20 17:35:06.039649 mon/Monitor.cc: 1126: FAILED assert(!(sync_role & SYNC_ROLE_REQUESTER)) ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c) 1: (Monitor::handle_sync_start_chunks(MMonSync*)+0x58e) [0x4b52fe] 2: (Monitor::handle_sync(MMonSync*)+0x28b) [0x4b652b] 3: (Monitor::_ms_dispatch(Message*)+0xc41) [0x4c1f21] 4: (Monitor::ms_dispatch(Message*)+0x32) [0x4dcc12] 5: (DispatchQueue::entry()+0x35b) [0x693c0b] 6: (DispatchQueue::DispatchThread::entry()+0xd) [0x627d2d] 7: (()+0x7f8e) [0x7f9061318f8e] 8: (clone()+0x6d) [0x7f905ff5ee1d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 20/20 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mon.a.log --- end dump of recent events --- 2013-04-20 17:35:06.108980 7f905a33d700 -1 *** Caught signal (Aborted) ** in thread 7f905a33d700 ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c) 1: /usr/bin/ceph-mon() [0x5869b0] 2: (()+0xfbd0) [0x7f9061320bd0] 3: (gsignal()+0x37) [0x7f905fe9c037] 4: (abort()+0x148) [0x7f905fe9f698] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f90607a8e8d] 6: (()+0x5ef76) [0x7f90607a6f76] 7: (()+0x5efa3) [0x7f90607a6fa3] 8: (()+0x5f1de) [0x7f90607a71de] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x6378cd] 10: (Monitor::handle_sync_start_chunks(MMonSync*)+0x58e) [0x4b52fe] 11: (Monitor::handle_sync(MMonSync*)+0x28b) [0x4b652b] 12: (Monitor::_ms_dispatch(Message*)+0xc41) [0x4c1f21] 13: (Monitor::ms_dispatch(Message*)+0x32) [0x4dcc12] 14: (DispatchQueue::entry()+0x35b) [0x693c0b] 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x627d2d] 16: (()+0x7f8e) [0x7f9061318f8e] 17: (clone()+0x6d) [0x7f905ff5ee1d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -4> 2013-04-20 17:35:06.060714 7f9053f2d700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:0/1255 pipe(0x8e42d900 sd=45 :6789 s=2 pgs=95 cs=1 l=1).reader couldn't read tag, Success -3> 2013-04-20 17:35:06.060742 7f9053f2d700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:0/1255 pipe(0x8e42d900 sd=45 :6789 s=2 pgs=95 cs=1 l=1).fault 0: Success -2> 2013-04-20 17:35:06.078338 7f9057056700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:0/1271 pipe(0x8e42d680 sd=46 :6789 s=2 pgs=95 cs=1 l=1).reader couldn't read tag, Success -1> 2013-04-20 17:35:06.078368 7f9057056700 2 -- 10.1.0.3:6789/0 >> 10.1.0.67:0/1271 pipe(0x8e42d680 sd=46 :6789 s=2 pgs=95 cs=1 l=1).fault 0: Success 0> 2013-04-20 17:35:06.108980 7f905a33d700 -1 *** Caught signal (Aborted) ** in thread 7f905a33d700 ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c) 1: /usr/bin/ceph-mon() [0x5869b0] 2: (()+0xfbd0) [0x7f9061320bd0] 3: (gsignal()+0x37) [0x7f905fe9c037] 4: (abort()+0x148) [0x7f905fe9f698] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f90607a8e8d] 6: (()+0x5ef76) [0x7f90607a6f76] 7: (()+0x5efa3) [0x7f90607a6fa3] 8: (()+0x5f1de) [0x7f90607a71de] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x6378cd] 10: (Monitor::handle_sync_start_chunks(MMonSync*)+0x58e) [0x4b52fe] 11: (Monitor::handle_sync(MMonSync*)+0x28b) [0x4b652b] 12: (Monitor::_ms_dispatch(Message*)+0xc41) [0x4c1f21] 13: (Monitor::ms_dispatch(Message*)+0x32) [0x4dcc12] 14: (DispatchQueue::entry()+0x35b) [0x693c0b] 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x627d2d] 16: (()+0x7f8e) [0x7f9061318f8e] 17: (clone()+0x6d) [0x7f905ff5ee1d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 20/20 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mon.a.log --- end dump of recent events --- root@node2:/var/log/ceph# </pre>