2014-09-18 17:00:30.142858 7f4c8a406700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6810/2873 pipe(0xe252b00 sd=45 :49015 s=2 pgs=61 cs=1 l=0 c=0xdd8d180).fault with nothing to send, going to standby 2014-09-18 17:00:30.144000 7f4c8c426700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6830/33895 pipe(0x4754840 sd=23 :59301 s=2 pgs=3288 cs=1 l=0 c=0xdd8a2c0).fault with nothing to send, going to standby 2014-09-18 17:00:30.146583 7f4c893f6700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6836/43468 pipe(0xe254940 sd=90 :38992 s=2 pgs=756 cs=1 l=0 c=0x49006e0).fault with nothing to send, going to standby 2014-09-18 17:00:30.148823 7f4c88ef1700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6819/2842 pipe(0xe252000 sd=97 :44381 s=2 pgs=63 cs=1 l=0 c=0x4901080).fault with nothing to send, going to standby 2014-09-18 17:00:30.151542 7f4c885e8700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6813/9578 pipe(0xe255700 sd=105 :52300 s=2 pgs=215 cs=1 l=0 c=0x49018c0).fault with nothing to send, going to standby 2014-09-18 17:00:30.153498 7f4c897fa700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6826/32986 pipe(0xe252580 sd=87 :45713 s=2 pgs=3151 cs=1 l=0 c=0xdd8caa0).fault with nothing to send, going to standby 2014-09-18 17:00:30.166704 7f4c891f4700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6804/4303 pipe(0xe2522c0 sd=79 :37109 s=2 pgs=300 cs=1 l=0 c=0x49011e0).fault with nothing to send, going to standby 2014-09-18 17:00:30.169925 7f4c8b315700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6821/11665 pipe(0x4757440 sd=52 :54421 s=2 pgs=4217 cs=1 l=0 c=0xdd8b4a0).fault with nothing to send, going to standby 2014-09-18 17:00:30.178766 7f4c8a80a700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6801/5950 pipe(0x4756ec0 sd=30 :38205 s=2 pgs=5644 cs=1 l=0 c=0xdd8c100).fault with nothing to send, going to standby 2014-09-18 17:00:30.188241 7f4c8c527700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6811/8573 pipe(0x4754b00 sd=32 :45675 s=2 pgs=4241 cs=1 l=0 c=0xdd8a420).fault with nothing to send, going to standby 2014-09-18 17:00:30.237626 7f4c89cff700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6811/41494 pipe(0xe253340 sd=82 :52592 s=2 pgs=1100 cs=1 l=0 c=0xdd8d440).fault with nothing to send, going to standby 2014-09-18 17:00:30.292977 7f4c72a95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6816/2508 pipe(0xe973c80 sd=310 :32815 s=2 pgs=41 cs=1 l=0 c=0xec5f2e0).fault with nothing to send, going to standby 2014-09-18 17:00:30.295379 7f4c8a002700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6813/2441 pipe(0xe253600 sd=56 :34360 s=2 pgs=88 cs=1 l=0 c=0xdd8d9c0).fault with nothing to send, going to standby 2014-09-18 17:00:30.296792 7f4c887ea700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6822/33871 pipe(0xe254100 sd=77 :35493 s=2 pgs=560 cs=1 l=0 c=0x4900000).fault with nothing to send, going to standby 2014-09-18 17:00:30.298593 7f4c8af11700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6808/2833 pipe(0x4757180 sd=57 :35177 s=2 pgs=58 cs=1 l=0 c=0xdd8c260).fault with nothing to send, going to standby 2014-09-18 17:00:30.298641 7f4c72691700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6821/2784 pipe(0xe4ee940 sd=308 :59576 s=2 pgs=55 cs=1 l=0 c=0xec9c160).fault with nothing to send, going to standby 2014-09-18 17:00:30.298637 7f4c877da700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6836/11580 pipe(0xe2e1340 sd=101 :44764 s=2 pgs=4379 cs=1 l=0 c=0x49039c0).fault with nothing to send, going to standby 2014-09-18 17:00:30.302616 7f4c72b96700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6826/2922 pipe(0xe4eec00 sd=141 :60195 s=2 pgs=40 cs=1 l=0 c=0xec9c2c0).fault with nothing to send, going to standby 2014-09-18 17:00:30.308886 7f4c8be20700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6806/7186 pipe(0x4754dc0 sd=38 :37243 s=2 pgs=3781 cs=1 l=0 c=0xdd8ab00).fault with nothing to send, going to standby 2014-09-18 17:00:31.123841 7f4c879dc700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6809/2830 pipe(0xe253e40 sd=98 :47011 s=2 pgs=63 cs=1 l=0 c=0x4901fa0).fault with nothing to send, going to standby 2014-09-18 17:00:31.123850 7f4c888eb700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6816/2834 pipe(0xe2543c0 sd=95 :51741 s=2 pgs=63 cs=1 l=0 c=0x4900160).fault with nothing to send, going to standby 2014-09-18 17:00:31.228457 7f4c8a90b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6802/5612 pipe(0x4757c80 sd=64 :48793 s=2 pgs=251 cs=1 l=0 c=0xdd8c940).fault with nothing to send, going to standby 2014-09-18 17:00:31.230561 7f4c8a608700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6816/9475 pipe(0x47579c0 sd=68 :48952 s=2 pgs=5186 cs=1 l=0 c=0xdd8c680).fault with nothing to send, going to standby 2014-09-18 17:00:31.288569 7f4c8b517700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6825/18664 pipe(0x4756940 sd=50 :48554 s=2 pgs=3132 cs=1 l=0 c=0xdd8ac60).fault with nothing to send, going to standby 2014-09-18 17:00:31.307550 7f4c875d8700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6829/19404 pipe(0xe2e1080 sd=102 :38872 s=2 pgs=1489 cs=1 l=0 c=0xe2d9a20).fault with nothing to send, going to standby 2014-09-18 17:00:31.373822 7f4c8b113700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6818/2484 pipe(0x4756680 sd=54 :35659 s=2 pgs=97 cs=1 l=0 c=0xdd8ba20).fault with nothing to send, going to standby 2014-09-18 17:00:31.374331 7f4c71f8a700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6836/3187 pipe(0xecf8000 sd=318 :6807 s=2 pgs=43 cs=1 l=0 c=0xe414c60).fault with nothing to send, going to standby 2014-09-18 17:00:31.796268 7f4c8ae10700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6817/2504 pipe(0x47563c0 sd=59 :50120 s=2 pgs=98 cs=1 l=0 c=0xdd8b1e0).fault with nothing to send, going to standby 2014-09-18 17:00:36.241273 7f4c898fb700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6831/9454 pipe(0xe253080 sd=86 :50403 s=2 pgs=4171 cs=1 l=0 c=0xdd8d2e0).fault with nothing to send, going to standby 2014-09-18 17:00:39.046594 7f4c8b91b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6808/2445 pipe(0x4755600 sd=42 :59420 s=2 pgs=98 cs=1 l=0 c=0xdd8b760).fault with nothing to send, going to standby 2014-09-18 17:02:05.695891 7f4c8b81a700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6816/10076 pipe(0x4755b80 sd=46 :45622 s=2 pgs=5769 cs=1 l=0 c=0xdd8adc0).fault with nothing to send, going to standby 2014-09-18 17:02:05.714836 7f4c881e4700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6833/41206 pipe(0xe255180 sd=107 :52900 s=2 pgs=456 cs=1 l=0 c=0x49035a0).fault with nothing to send, going to standby 2014-09-18 17:03:40.230075 7f4c71580700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6816/2660 pipe(0xe4b1440 sd=24 :6807 s=0 pgs=0 cs=0 l=0 c=0x4753860).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:03:41.327667 7f4c745b0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6808/2833 pipe(0xe4ef700 sd=78 :6807 s=0 pgs=0 cs=0 l=0 c=0xe4d9180).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:03:42.411513 7f4c8b214700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6816/2834 pipe(0xecd2ec0 sd=85 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8520).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:03:45.216552 7f4c88aed700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6804/4303 pipe(0xe2559c0 sd=132 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8680).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:03:46.297692 7f4c7bf29700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6801/5898 pipe(0xe2e0840 sd=134 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff87e0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:05:35.358964 7f4c83da7700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6801/5950 pipe(0x4757180 sd=74 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8940).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:05:36.288620 7f4c7c832700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6802/5612 pipe(0xe477440 sd=47 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff83c0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.367440 7f4c8ac0e700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6818/2484 pipe(0xe4eeec0 sd=63 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff9c80).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.367749 7f4c7ac16700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6822/33871 pipe(0xe80a680 sd=210 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff9b20).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.367822 7f4c8a305700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6825/18664 pipe(0xe477c80 sd=208 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff9860).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.367881 7f4c7ca34700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6810/2873 pipe(0xe2e0dc0 sd=177 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff9de0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.367925 7f4c7d03a700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6813/9578 pipe(0xe808840 sd=212 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8100).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.367971 7f4c80670700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6811/8573 pipe(0xecf8580 sd=214 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7600).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368008 7f4c8b113700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6821/2784 pipe(0xecfa940 sd=217 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7a20).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368042 7f4c83aa4700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6821/11665 pipe(0xecfb440 sd=215 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7760).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368085 7f4c8ad0f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6819/2842 pipe(0xe2e0b00 sd=190 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8680).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368229 7f4c7c02a700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6816/9475 pipe(0x4757c80 sd=209 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff99c0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368298 7f4c746b1700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6812/2464 pipe(0xe2538c0 sd=54 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7ce0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368331 7f4c7ad17700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6821/10869 pipe(0xe80aec0 sd=211 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7fa0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368377 7f4c85bc5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6830/33895 pipe(0x4756ec0 sd=203 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8520).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368419 7f4c743ae700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6815/2480 pipe(0xe2522c0 sd=218 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7b80).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368463 7f4c892f5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6826/32986 pipe(0xecf9b80 sd=216 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff78c0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368511 7f4c82a94700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6819/11463 pipe(0xe4b03c0 sd=213 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8260).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.368752 7f4c85ec8700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6806/7186 pipe(0xe54b9c0 sd=219 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7e40).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.369072 7f4c88ef1700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6816/10076 pipe(0xecfa100 sd=45 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8100).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.369214 7f4c8ab0d700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6829/19404 pipe(0xe252b00 sd=41 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff83c0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.369255 7f4c8c325700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6814/2831 pipe(0x4756680 sd=52 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff9de0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.369320 7f4c8a608700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6831/9454 pipe(0xe254100 sd=38 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff9c80).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.369351 7f4c87add700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6836/11580 pipe(0xe255700 sd=48 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8940).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.369418 7f4c87cdf700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6836/43468 pipe(0xe4ee940 sd=68 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff87e0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.369454 7f4c8c022700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6818/2832 pipe(0x4757440 sd=32 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff9b20).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:40.369514 7f4c8bb1d700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6813/2441 pipe(0x4756940 sd=23 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff9860).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:08:45.401591 7f4c893f6700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6806/31190 pipe(0xe255c80 sd=56 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7340).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:10:35.223678 7f4c884e7700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6809/2830 pipe(0xe252dc0 sd=86 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff74a0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:10:35.228465 7f4c879dc700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6808/2445 pipe(0xe253080 sd=95 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7fa0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:10:35.234098 7f4c89cff700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6811/41494 pipe(0xe254c00 sd=100 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff7ce0).accept connect_seq 2 vs existing 2 state connecting 2014-09-18 17:16:00.679007 7f4c90a34700 0 log_channel(default) log [INF] : 0.16 scrub ok 2014-09-18 17:16:01.679024 7f4c90a34700 0 log_channel(default) log [INF] : 0.56 scrub ok 2014-09-18 17:16:24.563017 7f4c885e8700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6831/3049 pipe(0xe254940 sd=377 :6807 s=0 pgs=0 cs=0 l=0 c=0xeff8680).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:16:40.977359 7f4c71782700 0 -- 10.242.42.178:6801/2483 >> 10.242.42.212:0/112715683 pipe(0x12978100 sd=336 :6801 s=0 pgs=0 cs=0 l=0 c=0xe2d8840).accept peer addr is really 10.242.42.212:0/112715683 (socket is 10.242.42.212:42970/0) 2014-09-18 17:19:37.175747 7f4c90a34700 0 log_channel(default) log [INF] : 2.649 scrub ok 2014-09-18 17:25:00.242778 7f4c90a34700 0 log_channel(default) log [INF] : 2.3ae scrub ok 2014-09-18 17:25:41.372529 7f4c899fc700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.168:6828/1002350 pipe(0xe253600 sd=319 :6807 s=2 pgs=95 cs=1 l=0 c=0xeff99c0).fault with nothing to send, going to standby 2014-09-18 17:31:27.093352 7f4c71f8a700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6836/3187 pipe(0xecf8000 sd=57 :35438 s=2 pgs=101 cs=3 l=0 c=0xe414c60).fault with nothing to send, going to standby 2014-09-18 17:31:27.525130 7f4c885e8700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6831/3049 pipe(0xe254940 sd=377 :6807 s=2 pgs=98 cs=3 l=0 c=0xede1b20).fault with nothing to send, going to standby 2014-09-18 17:31:40.209439 7f4caae41700 -1 osd.42 10658 heartbeat_check: no reply from osd.2 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:20.209438) 2014-09-18 17:31:40.209455 7f4caae41700 -1 osd.42 10658 heartbeat_check: no reply from osd.9 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:20.209438) 2014-09-18 17:31:40.209475 7f4caae41700 -1 osd.42 10658 heartbeat_check: no reply from osd.21 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:20.209438) 2014-09-18 17:31:40.209481 7f4caae41700 -1 osd.42 10658 heartbeat_check: no reply from osd.48 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:20.209438) 2014-09-18 17:31:40.209486 7f4caae41700 -1 osd.42 10658 heartbeat_check: no reply from osd.49 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:20.209438) 2014-09-18 17:31:40.209490 7f4caae41700 -1 osd.42 10658 heartbeat_check: no reply from osd.56 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:20.209438) 2014-09-18 17:31:40.209495 7f4caae41700 -1 osd.42 10658 heartbeat_check: no reply from osd.62 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:20.209438) 2014-09-18 17:31:41.209607 7f4caae41700 -1 osd.42 10659 heartbeat_check: no reply from osd.2 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:21.209606) 2014-09-18 17:31:41.209617 7f4caae41700 -1 osd.42 10659 heartbeat_check: no reply from osd.9 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:21.209606) 2014-09-18 17:31:41.209621 7f4caae41700 -1 osd.42 10659 heartbeat_check: no reply from osd.21 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:21.209606) 2014-09-18 17:31:41.209626 7f4caae41700 -1 osd.42 10659 heartbeat_check: no reply from osd.48 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:21.209606) 2014-09-18 17:31:41.209630 7f4caae41700 -1 osd.42 10659 heartbeat_check: no reply from osd.49 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:21.209606) 2014-09-18 17:31:41.209634 7f4caae41700 -1 osd.42 10659 heartbeat_check: no reply from osd.56 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:21.209606) 2014-09-18 17:31:41.209638 7f4caae41700 -1 osd.42 10659 heartbeat_check: no reply from osd.62 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:21.209606) 2014-09-18 17:31:42.209905 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.2 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:22.209905) 2014-09-18 17:31:42.209916 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.9 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:22.209905) 2014-09-18 17:31:42.209922 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.21 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:22.209905) 2014-09-18 17:31:42.209928 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.48 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:22.209905) 2014-09-18 17:31:42.209933 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.49 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:22.209905) 2014-09-18 17:31:42.209952 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.56 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:22.209905) 2014-09-18 17:31:42.209957 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.62 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:22.209905) 2014-09-18 17:31:42.922323 7f4c71782700 0 -- 10.242.42.178:6801/2483 >> 10.242.42.212:0/112715683 pipe(0x12b80000 sd=95 :6801 s=0 pgs=0 cs=0 l=0 c=0x104c2160).accept peer addr is really 10.242.42.212:0/112715683 (socket is 10.242.42.212:42973/0) 2014-09-18 17:31:43.210040 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.2 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:23.210039) 2014-09-18 17:31:43.210050 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.9 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:23.210039) 2014-09-18 17:31:43.210055 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.21 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:23.210039) 2014-09-18 17:31:43.210073 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.48 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:23.210039) 2014-09-18 17:31:43.210079 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.49 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:23.210039) 2014-09-18 17:31:43.210083 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.56 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:23.210039) 2014-09-18 17:31:43.210104 7f4caae41700 -1 osd.42 10660 heartbeat_check: no reply from osd.62 since back 2014-09-18 17:31:19.982168 front 2014-09-18 17:31:19.982168 (cutoff 2014-09-18 17:31:23.210039) 2014-09-18 17:36:42.459438 7f4c96a40700 0 log_channel(default) log [INF] : 2.3d5 restarting backfill on osd.9 from (0'0,0'0] MAX to 6827'28556 2014-09-18 17:36:42.460425 7f4c97241700 0 log_channel(default) log [INF] : 2.4f6 restarting backfill on osd.25 from (0'0,0'0] MAX to 6873'32776 2014-09-18 17:36:42.464367 7f4c96a40700 0 log_channel(default) log [INF] : 2.172 restarting backfill on osd.9 from (0'0,0'0] MAX to 6739'20886 2014-09-18 17:36:42.467658 7f4c96a40700 0 log_channel(default) log [INF] : 2.172 restarting backfill on osd.25 from (0'0,0'0] MAX to 6739'20886 2014-09-18 17:36:42.468676 7f4c97241700 0 log_channel(default) log [INF] : 2.71c restarting backfill on osd.4 from (0'0,0'0] MAX to 6811'28514 2014-09-18 17:36:42.476416 7f4c96a40700 0 log_channel(default) log [INF] : 2.179 restarting backfill on osd.6 from (0'0,0'0] MAX to 6752'29654 2014-09-18 17:36:42.477080 7f4c97241700 0 log_channel(default) log [INF] : 2.5b1 restarting backfill on osd.30 from (0'0,0'0] MAX to 6873'22940 2014-09-18 17:36:42.482275 7f4c96a40700 0 log_channel(default) log [INF] : 2.17e restarting backfill on osd.30 from (0'0,0'0] MAX to 6789'26580 2014-09-18 17:36:42.484315 7f4c97241700 0 log_channel(default) log [INF] : 2.5b4 restarting backfill on osd.25 from (0'0,0'0] MAX to 6792'28596 2014-09-18 17:36:42.485931 7f4c96a40700 0 log_channel(default) log [INF] : 2.2d8 restarting backfill on osd.9 from (0'0,0'0] MAX to 6825'27712 2014-09-18 17:36:42.486769 7f4c97241700 0 log_channel(default) log [INF] : 2.62 restarting backfill on osd.6 from (0'0,0'0] MAX to 6775'23328 2014-09-18 17:36:42.490092 7f4c96a40700 0 log_channel(default) log [INF] : 2.71 restarting backfill on osd.41 from (0'0,0'0] MAX to 6845'29618 2014-09-18 17:36:42.494516 7f4c96a40700 0 log_channel(default) log [INF] : 2.491 restarting backfill on osd.32 from (0'0,0'0] MAX to 6842'30456 2014-09-18 17:36:42.495225 7f4c97241700 0 log_channel(default) log [INF] : 2.654 restarting backfill on osd.25 from (0'0,0'0] MAX to 6778'24660 2014-09-18 17:36:42.496429 7f4c96a40700 0 log_channel(default) log [INF] : 2.646 restarting backfill on osd.35 from (0'0,0'0] MAX to 6825'27460 2014-09-18 17:36:42.497973 7f4c96a40700 0 log_channel(default) log [INF] : 2.646 restarting backfill on osd.58 from (0'0,0'0] MAX to 6825'27460 2014-09-18 17:36:42.499345 7f4c97241700 0 log_channel(default) log [INF] : 2.526 restarting backfill on osd.32 from (0'0,0'0] MAX to 6749'22622 2014-09-18 17:36:42.501866 7f4c97241700 0 log_channel(default) log [INF] : 2.4b0 restarting backfill on osd.32 from (0'0,0'0] MAX to 6832'25336 2014-09-18 17:36:42.502222 7f4c96a40700 0 log_channel(default) log [INF] : 2.1bb restarting backfill on osd.35 from (0'0,0'0] MAX to 6791'24086 2014-09-18 17:36:42.508697 7f4c97241700 0 log_channel(default) log [INF] : 2.4b1 restarting backfill on osd.9 from (0'0,0'0] MAX to 6745'24816 2014-09-18 17:36:42.509166 7f4c96a40700 0 log_channel(default) log [INF] : 2.22d restarting backfill on osd.27 from (0'0,0'0] MAX to 6789'28280 2014-09-18 17:36:42.513088 7f4c96a40700 0 log_channel(default) log [INF] : 2.22d restarting backfill on osd.35 from (0'0,0'0] MAX to 6789'28280 2014-09-18 17:36:42.514584 7f4c97241700 0 log_channel(default) log [INF] : 2.799 restarting backfill on osd.25 from (0'0,0'0] MAX to 6761'29900 2014-09-18 17:36:42.520937 7f4c97241700 0 log_channel(default) log [INF] : 2.79c restarting backfill on osd.32 from (0'0,0'0] MAX to 6856'26450 2014-09-18 17:36:42.521181 7f4c96a40700 0 log_channel(default) log [INF] : 2.403 restarting backfill on osd.25 from (0'0,0'0] MAX to 6774'31098 2014-09-18 17:36:42.525048 7f4c97241700 0 log_channel(default) log [INF] : 2.4c6 restarting backfill on osd.32 from (0'0,0'0] MAX to 6754'24942 2014-09-18 17:36:42.527579 7f4c96a40700 0 log_channel(default) log [INF] : 2.4c9 restarting backfill on osd.41 from (0'0,0'0] MAX to 6835'28020 2014-09-18 17:36:42.534795 7f4c96a40700 0 log_channel(default) log [INF] : 2.67b restarting backfill on osd.41 from (0'0,0'0] MAX to 6745'24728 2014-09-18 17:36:42.540071 7f4c96a40700 0 log_channel(default) log [INF] : 2.40a restarting backfill on osd.32 from (0'0,0'0] MAX to 6791'27848 2014-09-18 17:36:42.543857 7f4c96a40700 0 log_channel(default) log [INF] : 2.4d0 restarting backfill on osd.30 from (0'0,0'0] MAX to 6699'27238 2014-09-18 17:36:42.546758 7f4c96a40700 0 log_channel(default) log [INF] : 2.26d restarting backfill on osd.35 from (0'0,0'0] MAX to 6840'34196 2014-09-18 17:36:46.448647 7f4c97241700 0 log_channel(default) log [INF] : 2.4f6 restarting backfill on osd.4 from (0'0,0'0] MAX to 6873'32776 2014-09-18 17:36:46.459485 7f4c97241700 0 log_channel(default) log [INF] : 2.491 restarting backfill on osd.20 from (0'0,0'0] MAX to 6842'30456 2014-09-18 17:36:46.461914 7f4c97241700 0 log_channel(default) log [INF] : 2.1bb restarting backfill on osd.27 from (0'0,0'0] MAX to 6791'24086 2014-09-18 17:36:46.560077 7f4c96a40700 0 log_channel(default) log [INF] : 2.654 restarting backfill on osd.35 from (0'0,0'0] MAX to 6778'24660 2014-09-18 17:36:46.562987 7f4c97241700 0 log_channel(default) log [INF] : 2.799 restarting backfill on osd.35 from (0'0,0'0] MAX to 6761'29900 2014-09-18 17:36:46.570162 7f4c96a40700 0 log_channel(default) log [INF] : 2.40a restarting backfill on osd.35 from (0'0,0'0] MAX to 6791'27848 2014-09-18 17:36:46.920714 7f4c96a40700 0 log_channel(default) log [INF] : 2.17e restarting backfill on osd.6 from (0'0,0'0] MAX to 6789'26580 2014-09-18 17:36:47.225388 7f4c96a40700 0 log_channel(default) log [INF] : 2.71 restarting backfill on osd.35 from (0'0,0'0] MAX to 6845'29618 2014-09-18 17:36:47.225871 7f4c96a40700 0 log_channel(default) log [INF] : 2.71 restarting backfill on osd.41 from (0'0,0'0] MAX to 6845'29618 2014-09-18 17:36:48.209536 7f4c97241700 0 log_channel(default) log [INF] : 2.62 restarting backfill on osd.25 from (0'0,0'0] MAX to 6775'23328 2014-09-18 17:36:48.457426 7f4c97241700 0 log_channel(default) log [INF] : 2.6f8 restarting backfill on osd.6 from (0'0,0'0] MAX to 6788'24342 2014-09-18 17:36:48.470457 7f4c97241700 0 log_channel(default) log [INF] : 2.2d8 restarting backfill on osd.25 from (0'0,0'0] MAX to 6825'27712 2014-09-18 17:36:48.557261 7f4c96a40700 0 log_channel(default) log [INF] : 2.186 restarting backfill on osd.4 from (0'0,0'0] MAX to 6871'27280 2014-09-18 17:36:48.771315 7f4c97241700 0 log_channel(default) log [INF] : 2.6e2 restarting backfill on osd.35 from (0'0,0'0] MAX to 6730'25922 2014-09-18 17:36:49.308895 7f4c96a40700 0 log_channel(default) log [INF] : 2.719 restarting backfill on osd.4 from (0'0,0'0] MAX to 6842'29696 2014-09-18 17:36:49.309682 7f4c96a40700 0 log_channel(default) log [INF] : 2.719 restarting backfill on osd.27 from (0'0,0'0] MAX to 6842'29696 2014-09-18 17:36:50.204435 7f4c97241700 0 log_channel(default) log [INF] : 2.204 restarting backfill on osd.27 from (0'0,0'0] MAX to 6810'26722 2014-09-18 17:36:50.325491 7f4c97241700 0 log_channel(default) log [INF] : 2.519 restarting backfill on osd.6 from (0'0,0'0] MAX to 6868'25546 2014-09-18 17:36:50.327697 7f4c96a40700 0 log_channel(default) log [INF] : 2.178 restarting backfill on osd.9 from (0'0,0'0] MAX to 6874'18440 2014-09-18 17:36:50.857521 7f4c97241700 0 log_channel(default) log [INF] : 2.457 restarting backfill on osd.35 from (0'0,0'0] MAX to 6761'29864 2014-09-18 17:36:53.237298 7f4c97241700 0 log_channel(default) log [INF] : 2.7fa restarting backfill on osd.25 from (0'0,0'0] MAX to 6761'32700 2014-09-18 17:36:54.812954 7f4c97241700 0 log_channel(default) log [INF] : 2.4c9 restarting backfill on osd.20 from (0'0,0'0] MAX to 6835'28020 2014-09-18 17:36:54.813500 7f4c97241700 0 log_channel(default) log [INF] : 2.4c9 restarting backfill on osd.41 from (0'0,0'0] MAX to 6835'28020 2014-09-18 17:36:54.816415 7f4c97241700 0 log_channel(default) log [INF] : 2.67b restarting backfill on osd.9 from (0'0,0'0] MAX to 6745'24728 2014-09-18 17:36:54.816928 7f4c97241700 0 log_channel(default) log [INF] : 2.67b restarting backfill on osd.41 from (0'0,0'0] MAX to 6745'24728 2014-09-18 17:36:55.449711 7f4c96a40700 0 log_channel(default) log [INF] : 2.e0 restarting backfill on osd.4 from (0'0,0'0] MAX to 6854'28782 2014-09-18 17:36:55.450375 7f4c96a40700 0 log_channel(default) log [INF] : 2.e0 restarting backfill on osd.27 from (0'0,0'0] MAX to 6854'28782 2014-09-18 17:36:55.832448 7f4c97241700 0 log_channel(default) log [INF] : 2.13 restarting backfill on osd.9 from (0'0,0'0] MAX to 6725'30144 2014-09-18 17:36:55.833079 7f4c97241700 0 log_channel(default) log [INF] : 2.13 restarting backfill on osd.25 from (0'0,0'0] MAX to 6725'30144 2014-09-18 17:36:56.433886 7f4c97241700 0 log_channel(default) log [INF] : 2.410 restarting backfill on osd.9 from (0'0,0'0] MAX to 6840'31228 2014-09-18 17:36:56.434555 7f4c97241700 0 log_channel(default) log [INF] : 2.410 restarting backfill on osd.25 from (0'0,0'0] MAX to 6840'31228 2014-09-18 17:36:57.467110 7f4c96a40700 0 log_channel(default) log [INF] : 2.2c7 restarting backfill on osd.4 from (0'0,0'0] MAX to 6805'24696 2014-09-18 17:36:57.467999 7f4c96a40700 0 log_channel(default) log [INF] : 2.2c7 restarting backfill on osd.27 from (0'0,0'0] MAX to 6805'24696 2014-09-18 17:36:59.281176 7f4c97241700 0 log_channel(default) log [INF] : 2.4c1 restarting backfill on osd.9 from (0'0,0'0] MAX to 6761'26972 2014-09-18 17:36:59.281781 7f4c97241700 0 log_channel(default) log [INF] : 2.4c1 restarting backfill on osd.27 from (0'0,0'0] MAX to 6761'26972 2014-09-18 17:37:02.752688 7f4c96a40700 0 log_channel(default) log [INF] : 2.3b6 restarting backfill on osd.25 from (0'0,0'0] MAX to 6856'30926 2014-09-18 17:37:02.753417 7f4c96a40700 0 log_channel(default) log [INF] : 2.3b6 restarting backfill on osd.35 from (0'0,0'0] MAX to 6856'30926 2014-09-18 17:44:41.354578 7f4c7a30d700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6827/21534 pipe(0x123c2c00 sd=23 :6807 s=2 pgs=29 cs=1 l=0 c=0x107375a0).fault, initiating reconnect 2014-09-18 17:44:41.354611 7f4c7a10b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6827/21534 pipe(0x123c2c00 sd=23 :6807 s=1 pgs=29 cs=2 l=0 c=0x107375a0).fault 2014-09-18 17:44:41.355349 7f4c8258f700 0 -- 10.242.42.178:0/2483 >> 10.242.42.166:6841/21534 pipe(0xecd0000 sd=23 :0 s=1 pgs=0 cs=0 l=1 c=0x12aeac00).fault 2014-09-18 17:44:41.355367 7f4c81b85700 0 -- 10.242.42.178:0/2483 >> 10.242.42.166:6843/21534 pipe(0xecd18c0 sd=31 :0 s=1 pgs=0 cs=0 l=1 c=0x12ae8c60).fault 2014-09-18 17:45:02.277350 7f4c74eb9700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6826/21533 pipe(0x123c2ec0 sd=87 :6807 s=2 pgs=15 cs=1 l=0 c=0x10737860).fault with nothing to send, going to standby 2014-09-18 17:45:02.278763 7f4c74db8700 0 -- 10.242.42.178:0/2483 >> 10.242.42.166:6840/21533 pipe(0xe970580 sd=23 :0 s=1 pgs=0 cs=0 l=1 c=0x10af8d60).fault 2014-09-18 17:45:02.278786 7f4c882e5700 0 -- 10.242.42.178:0/2483 >> 10.242.42.166:6842/21533 pipe(0xe971b80 sd=46 :0 s=1 pgs=0 cs=0 l=1 c=0x109e4260).fault 2014-09-18 17:45:07.048335 7f4c7a20c700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.172:6838/36459 pipe(0x123c0dc0 sd=100 :6807 s=2 pgs=34 cs=1 l=0 c=0x10737700).fault with nothing to send, going to standby 2014-09-18 17:45:07.049578 7f4c791fc700 0 -- 10.242.42.178:0/2483 >> 10.242.42.172:6841/36459 pipe(0xe809600 sd=39 :0 s=1 pgs=0 cs=0 l=1 c=0x114b9fa0).fault 2014-09-18 17:45:07.049664 7f4c71782700 0 -- 10.242.42.178:0/2483 >> 10.242.42.172:6843/36459 pipe(0x141a9700 sd=103 :0 s=1 pgs=0 cs=0 l=1 c=0x114b8c60).fault 2014-09-18 17:45:08.281845 7f4c8be20700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6814/2485 pipe(0xe252840 sd=50 :38915 s=2 pgs=121 cs=3 l=0 c=0xdd8cc00).fault, initiating reconnect 2014-09-18 17:45:08.282107 7f4c790fb700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6825/2485 pipe(0x12b82100 sd=137 :0 s=1 pgs=0 cs=0 l=1 c=0x10ae5020).fault 2014-09-18 17:45:08.282121 7f4c71984700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6832/2485 pipe(0xe808840 sd=139 :0 s=1 pgs=0 cs=0 l=1 c=0x10ae5180).fault 2014-09-18 17:45:08.282200 7f4c89e00700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6814/2485 pipe(0xe252840 sd=50 :38915 s=1 pgs=121 cs=4 l=0 c=0xdd8cc00).fault 2014-09-18 17:46:21.148812 7f4ca12d8700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.168:6831/1010800 pipe(0x123c0b00 sd=67 :6807 s=2 pgs=3895 cs=1 l=0 c=0xfc71440).fault with nothing to send, going to standby 2014-09-18 17:51:04.057399 7f4c97241700 0 log_channel(default) log [INF] : 2.2c7 restarting backfill on osd.35 from (0'0,0'0] MAX to 6805'24696 2014-09-18 17:51:04.065070 7f4c97241700 0 log_channel(default) log [INF] : 2.40a restarting backfill on osd.4 from (0'0,0'0] MAX to 6791'27848 2014-09-18 17:51:04.069205 7f4c97241700 0 log_channel(default) log [INF] : 2.3b6 restarting backfill on osd.6 from (0'0,0'0] MAX to 6856'30926 2014-09-18 17:51:05.061800 7f4c97241700 0 log_channel(default) log [INF] : 2.22d restarting backfill on osd.4 from (0'0,0'0] MAX to 6789'28280 2014-09-18 17:51:10.534170 7f4c96a40700 0 log_channel(default) log [INF] : 2.457 restarting backfill on osd.4 from (0'0,0'0] MAX to 6761'29864 2014-09-18 17:51:10.534440 7f4c97241700 0 log_channel(default) log [INF] : 2.2c7 restarting backfill on osd.24 from (0'0,0'0] MAX to 6805'24696 2014-09-18 17:51:10.540517 7f4c96a40700 0 log_channel(default) log [INF] : 2.186 restarting backfill on osd.35 from (0'0,0'0] MAX to 6871'27280 2014-09-18 17:51:16.847546 7f4c7319c700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6804/2488 pipe(0x11e02c00 sd=115 :6807 s=0 pgs=0 cs=0 l=0 c=0x1088b700).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:51:16.850386 7f4c72994700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6826/2922 pipe(0x11e00840 sd=201 :6807 s=0 pgs=0 cs=0 l=0 c=0x1088b2e0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:51:17.850714 7f4c96a40700 0 log_channel(default) log [INF] : 2.e0 restarting backfill on osd.20 from (0'0,0'0] MAX to 6854'28782 2014-09-18 17:51:18.892578 7f4c96a40700 0 log_channel(default) log [INF] : 2.1a2 restarting backfill on osd.6 from (0'0,0'0] MAX to 6869'33524 2014-09-18 17:51:19.642106 7f4c97241700 0 log_channel(default) log [INF] : 2.67b restarting backfill on osd.10 from (0'0,0'0] MAX to 6745'24728 2014-09-18 17:51:19.642934 7f4c97241700 0 log_channel(default) log [INF] : 2.67b restarting backfill on osd.25 from (0'0,0'0] MAX to 6745'24728 2014-09-18 17:51:24.015211 7f4c85dc7700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6831/3049 pipe(0x11e00000 sd=190 :6807 s=0 pgs=0 cs=0 l=0 c=0x1088bde0).accept connect_seq 4 vs existing 3 state standby 2014-09-18 17:51:25.016102 7f4c97241700 0 log_channel(default) log [INF] : 2.186 restarting backfill on osd.9 from (0'0,0'0] MAX to 6871'27280 2014-09-18 17:51:25.016482 7f4c96a40700 0 log_channel(default) log [INF] : 2.719 restarting backfill on osd.35 from (0'0,0'0] MAX to 6842'29696 2014-09-18 17:51:25.025365 7f4c96a40700 0 log_channel(default) log [INF] : 2.71c restarting backfill on osd.35 from (0'0,0'0] MAX to 6811'28514 2014-09-18 17:51:25.031994 7f4c97241700 0 log_channel(default) log [INF] : 2.1bb restarting backfill on osd.23 from (0'0,0'0] MAX to 6791'24086 2014-09-18 17:51:25.558760 7f4c97241700 0 log_channel(default) log [INF] : 2.26d restarting backfill on osd.55 from (0'0,0'0] MAX to 6840'34196 2014-09-18 17:51:33.644518 7f4c96a40700 0 log_channel(default) log [INF] : 2.71c restarting backfill on osd.39 from (0'0,0'0] MAX to 6811'28514 2014-09-18 17:51:33.651033 7f4c96a40700 0 log_channel(default) log [INF] : 2.4c6 restarting backfill on osd.9 from (0'0,0'0] MAX to 6754'24942 2014-09-18 17:51:33.655290 7f4c96a40700 0 log_channel(default) log [INF] : 2.40a restarting backfill on osd.6 from (0'0,0'0] MAX to 6791'27848 2014-09-18 17:51:34.650415 7f4c97241700 0 log_channel(default) log [INF] : 2.517 restarting backfill on osd.6 from (0'0,0'0] MAX to 6819'24342 2014-09-18 17:51:36.549503 7f4c96a40700 0 log_channel(default) log [INF] : 2.646 restarting backfill on osd.22 from (0'0,0'0] MAX to 6825'27460 2014-09-18 17:51:39.723944 7f4c74bb6700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6816/2508 pipe(0x11e01e40 sd=93 :6807 s=0 pgs=0 cs=0 l=0 c=0x1088fb20).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:51:40.548622 7f4c97241700 0 log_channel(default) log [INF] : 2.403 restarting backfill on osd.56 from (0'0,0'0] MAX to 6774'31098 2014-09-18 17:51:40.549129 7f4c96a40700 0 log_channel(default) log [INF] : 2.4d0 restarting backfill on osd.9 from (0'0,0'0] MAX to 6699'27238 2014-09-18 17:51:40.549548 7f4c97241700 0 log_channel(default) log [INF] : 2.403 restarting backfill on osd.58 from (0'0,0'0] MAX to 6774'31098 2014-09-18 17:51:40.550055 7f4c96a40700 0 log_channel(default) log [INF] : 2.4d0 restarting backfill on osd.32 from (0'0,0'0] MAX to 6699'27238 2014-09-18 17:51:40.682812 7f4c96a40700 0 log_channel(default) log [INF] : 2.2d8 restarting backfill on osd.22 from (0'0,0'0] MAX to 6825'27712 2014-09-18 17:51:40.683601 7f4c96a40700 0 log_channel(default) log [INF] : 2.2d8 restarting backfill on osd.53 from (0'0,0'0] MAX to 6825'27712 2014-09-18 17:51:40.740468 7f4c97241700 0 log_channel(default) log [INF] : 2.17e restarting backfill on osd.23 from (0'0,0'0] MAX to 6789'26580 2014-09-18 17:51:40.741190 7f4c97241700 0 log_channel(default) log [INF] : 2.17e restarting backfill on osd.25 from (0'0,0'0] MAX to 6789'26580 2014-09-18 17:51:40.741703 7f4c97241700 0 log_channel(default) log [INF] : 2.17e restarting backfill on osd.63 from (0'0,0'0] MAX to 6789'26580 2014-09-18 17:51:42.112495 7f4caae41700 -1 osd.42 11150 heartbeat_check: no reply from osd.41 since back 2014-09-18 17:51:21.717709 front 2014-09-18 17:51:21.717709 (cutoff 2014-09-18 17:51:22.112492) 2014-09-18 17:51:42.623821 7f4c8fa32700 -1 osd.42 11151 heartbeat_check: no reply from osd.41 since back 2014-09-18 17:51:21.717709 front 2014-09-18 17:51:21.717709 (cutoff 2014-09-18 17:51:22.623819) 2014-09-18 17:51:43.112680 7f4caae41700 -1 osd.42 11151 heartbeat_check: no reply from osd.41 since back 2014-09-18 17:51:21.717709 front 2014-09-18 17:51:21.717709 (cutoff 2014-09-18 17:51:23.112675) 2014-09-18 17:51:44.112863 7f4caae41700 -1 osd.42 11151 heartbeat_check: no reply from osd.41 since back 2014-09-18 17:51:21.717709 front 2014-09-18 17:51:21.717709 (cutoff 2014-09-18 17:51:24.112859) 2014-09-18 17:51:45.112981 7f4caae41700 -1 osd.42 11152 heartbeat_check: no reply from osd.41 since back 2014-09-18 17:51:21.717709 front 2014-09-18 17:51:21.717709 (cutoff 2014-09-18 17:51:25.112977) 2014-09-18 17:51:46.552983 7f4c96a40700 0 log_channel(default) log [INF] : 2.457 restarting backfill on osd.24 from (0'0,0'0] MAX to 6761'29864 2014-09-18 17:51:47.555071 7f4c97241700 0 log_channel(default) log [INF] : 2.719 restarting backfill on osd.56 from (0'0,0'0] MAX to 6842'29696 2014-09-18 17:51:49.571039 7f4c892f5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.168:6831/1010800 pipe(0x11e03700 sd=68 :6807 s=0 pgs=0 cs=0 l=0 c=0x1088fde0).accept connect_seq 2 vs existing 1 state standby 2014-09-18 17:51:50.158592 7f4c97241700 0 log_channel(default) log [INF] : 2.22d restarting backfill on osd.22 from (0'0,0'0] MAX to 6789'28280 2014-09-18 17:56:36.346501 7f4c97241700 0 log_channel(default) log [INF] : 2.4e8 restarting backfill on osd.35 from (0'0,0'0] MAX to 6852'28108 2014-09-18 17:56:44.528032 7f4c96a40700 0 log_channel(default) log [INF] : 2.4e8 restarting backfill on osd.0 from (0'0,0'0] MAX to 6852'28108 2014-09-18 17:56:45.541515 7f4c97241700 0 log_channel(default) log [INF] : 2.6e2 restarting backfill on osd.0 from (0'0,0'0] MAX to 6730'25922 2014-09-18 17:56:45.541912 7f4c96a40700 0 log_channel(default) log [INF] : 2.22d restarting backfill on osd.0 from (0'0,0'0] MAX to 6789'28280 2014-09-18 17:57:04.326674 7f4c7bf29700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6801/5898 pipe(0xe2e0840 sd=134 :6807 s=2 pgs=3419 cs=3 l=0 c=0xdd8c3c0).fault with nothing to send, going to standby 2014-09-18 17:57:04.326751 7f4c759c4700 0 -- 10.242.42.178:0/2483 >> 10.242.42.174:6802/5898 pipe(0x1e287c80 sd=23 :0 s=1 pgs=0 cs=0 l=1 c=0xcc9dde0).fault 2014-09-18 17:57:04.344287 7f4c8117b700 0 -- 10.242.42.178:0/2483 >> 10.242.42.174:6803/5898 pipe(0x1e2879c0 sd=253 :0 s=1 pgs=0 cs=0 l=1 c=0xcc9dc80).fault 2014-09-18 17:57:05.063329 7f4c8a204700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6801/5898 pipe(0xe2e0840 sd=134 :6807 s=1 pgs=3419 cs=4 l=0 c=0xdd8c3c0).fault 2014-09-18 17:57:48.046841 7f4c96a40700 0 log_channel(default) log [INF] : 2.22d restarting backfill on osd.22 from (0'0,0'0] MAX to 6789'28280 2014-09-18 17:57:56.604430 7f4caae41700 -1 osd.42 11280 heartbeat_check: no reply from osd.43 since back 2014-09-18 17:57:40.322858 front 2014-09-18 17:57:36.221983 (cutoff 2014-09-18 17:57:36.604426) 2014-09-18 17:57:57.604748 7f4caae41700 -1 osd.42 11281 heartbeat_check: no reply from osd.43 since back 2014-09-18 17:57:40.322858 front 2014-09-18 17:57:36.221983 (cutoff 2014-09-18 17:57:37.604744) 2014-09-18 17:57:58.604895 7f4caae41700 -1 osd.42 11281 heartbeat_check: no reply from osd.43 since back 2014-09-18 17:57:40.322858 front 2014-09-18 17:57:36.221983 (cutoff 2014-09-18 17:57:38.604892) 2014-09-18 17:58:41.747076 7f4c97241700 0 log_channel(default) log [INF] : 2.4e8 restarting backfill on osd.35 from (0'0,0'0] MAX to 6852'28108 2014-09-18 17:58:49.693169 7f4c96a40700 0 log_channel(default) log [INF] : 2.4e8 restarting backfill on osd.0 from (0'0,0'0] MAX to 6852'28108 2014-09-18 17:58:51.278226 7f4c97241700 0 log_channel(default) log [INF] : 2.22d restarting backfill on osd.0 from (0'0,0'0] MAX to 6789'28280 2014-09-18 17:58:51.278795 7f4c96a40700 0 log_channel(default) log [INF] : 2.6e2 restarting backfill on osd.0 from (0'0,0'0] MAX to 6730'25922 2014-09-18 17:58:51.338396 7f4c8b113700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6836/3187 pipe(0x11e018c0 sd=146 :6807 s=0 pgs=0 cs=0 l=0 c=0xcbed020).accept connect_seq 4 vs existing 3 state standby 2014-09-18 17:58:59.782229 7f4c746b1700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6812/2464 pipe(0xe2538c0 sd=54 :6807 s=2 pgs=108 cs=3 l=0 c=0xdd8af20).fault with nothing to send, going to standby 2014-09-18 17:58:59.783434 7f4c71e89700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6819/2464 pipe(0xecd3440 sd=89 :0 s=1 pgs=0 cs=0 l=1 c=0x114a09a0).fault 2014-09-18 17:58:59.783471 7f4c88cef700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6826/2464 pipe(0xecd1080 sd=152 :0 s=1 pgs=0 cs=0 l=1 c=0x114a0c60).fault 2014-09-18 17:59:15.621512 7f4c96a40700 0 log_channel(default) log [INF] : 2.71c restarting backfill on osd.0 from (0'0,0'0] MAX to 6811'28514 2014-09-18 17:59:15.625549 7f4c97241700 0 log_channel(default) log [INF] : 2.646 restarting backfill on osd.1 from (0'0,0'0] MAX to 6825'27460 2014-09-18 17:59:15.626397 7f4c96a40700 0 log_channel(default) log [INF] : 2.4c6 restarting backfill on osd.0 from (0'0,0'0] MAX to 6754'24942 2014-09-18 17:59:15.630740 7f4c97241700 0 log_channel(default) log [INF] : 2.517 restarting backfill on osd.0 from (0'0,0'0] MAX to 6819'24342 2014-09-18 17:59:15.632160 7f4c96a40700 0 log_channel(default) log [INF] : 2.40a restarting backfill on osd.1 from (0'0,0'0] MAX to 6791'27848 2014-09-18 17:59:19.971933 7f4c97241700 0 log_channel(default) log [INF] : 2.1d8 restarting backfill on osd.22 from (0'0,0'0] MAX to 6865'30228 2014-09-18 17:59:19.972564 7f4c97241700 0 log_channel(default) log [INF] : 2.1d8 restarting backfill on osd.27 from (0'0,0'0] MAX to 6865'30228 2014-09-18 17:59:28.913584 7f4c96a40700 0 log_channel(default) log [INF] : 2.457 restarting backfill on osd.2 from (0'0,0'0] MAX to 6761'29864 2014-09-18 17:59:28.916996 7f4c97241700 0 log_channel(default) log [INF] : 2.517 restarting backfill on osd.2 from (0'0,0'0] MAX to 6819'24342 2014-09-18 17:59:28.917428 7f4c96a40700 0 log_channel(default) log [INF] : 2.40a restarting backfill on osd.2 from (0'0,0'0] MAX to 6791'27848 2014-09-18 17:59:37.606731 7f4c96a40700 0 log_channel(default) log [INF] : 2.4f6 restarting backfill on osd.2 from (0'0,0'0] MAX to 6873'32776 2014-09-18 17:59:37.608015 7f4c97241700 0 log_channel(default) log [INF] : 2.186 restarting backfill on osd.0 from (0'0,0'0] MAX to 6871'27280 2014-09-18 17:59:37.614586 7f4c97241700 0 log_channel(default) log [INF] : 2.1bb restarting backfill on osd.0 from (0'0,0'0] MAX to 6791'24086 2014-09-18 17:59:37.614937 7f4c96a40700 0 log_channel(default) log [INF] : 2.4c6 restarting backfill on osd.1 from (0'0,0'0] MAX to 6754'24942 2014-09-18 17:59:37.617681 7f4c97241700 0 log_channel(default) log [INF] : 2.26d restarting backfill on osd.0 from (0'0,0'0] MAX to 6840'34196 2014-09-18 17:59:50.603061 7f4c97241700 0 log_channel(default) log [INF] : 2.4c6 restarting backfill on osd.3 from (0'0,0'0] MAX to 6754'24942 2014-09-18 17:59:50.603302 7f4c96a40700 0 log_channel(default) log [INF] : 2.4e8 restarting backfill on osd.3 from (0'0,0'0] MAX to 6852'28108 2014-09-18 18:00:05.606579 7f4c96a40700 0 log_channel(default) log [INF] : 2.29c restarting backfill on osd.0 from (0'0,0'0] MAX to 6805'31672 2014-09-18 18:00:05.606778 7f4c97241700 0 log_channel(default) log [INF] : 2.e0 restarting backfill on osd.0 from (0'0,0'0] MAX to 6854'28782 2014-09-18 18:00:05.612186 7f4c97241700 0 log_channel(default) log [INF] : 2.1a2 restarting backfill on osd.1 from (0'0,0'0] MAX to 6869'33524 2014-09-18 18:00:05.616126 7f4c96a40700 0 log_channel(default) log [INF] : 2.120 restarting backfill on osd.1 from (0'0,0'0] MAX to 6776'27398 2014-09-18 18:00:05.723779 7f4c96a40700 0 log_channel(default) log [INF] : 2.78f restarting backfill on osd.1 from (0'0,0'0] MAX to 6789'22964 2014-09-18 18:00:07.825127 7f4c866d0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6802/44336 pipe(0x11e02680 sd=193 :6807 s=2 pgs=17 cs=1 l=0 c=0x108f5de0).fault with nothing to send, going to standby 2014-09-18 18:00:07.825952 7f4c867d1700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6803/44336 pipe(0xecf9080 sd=43 :0 s=1 pgs=0 cs=0 l=1 c=0x1e0b87e0).fault 2014-09-18 18:00:07.825987 7f4c8b315700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6829/44336 pipe(0xecf8840 sd=151 :0 s=1 pgs=0 cs=0 l=1 c=0x1e0b8940).fault 2014-09-18 18:00:12.860243 7f4c868d2700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6825/45232 pipe(0x21f0f700 sd=275 :6807 s=2 pgs=10 cs=1 l=0 c=0x13a4fe40).fault, initiating reconnect 2014-09-18 18:00:12.860353 7f4c7aa14700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6825/45232 pipe(0x21f0f700 sd=271 :6807 s=1 pgs=10 cs=2 l=0 c=0x13a4fe40).fault 2014-09-18 18:00:12.860905 7f4c866d0700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6826/45232 pipe(0x123c1e40 sd=151 :0 s=1 pgs=0 cs=0 l=1 c=0x1d6edc80).fault 2014-09-18 18:00:12.860935 7f4c867d1700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6827/45232 pipe(0xe254680 sd=193 :0 s=1 pgs=0 cs=0 l=1 c=0x1d6edde0).fault 2014-09-18 18:00:14.701003 7f4c96a40700 0 log_channel(default) log [INF] : 2.517 restarting backfill on osd.4 from (0'0,0'0] MAX to 6819'24342 2014-09-18 18:00:15.612467 7f4c97241700 0 log_channel(default) log [INF] : 2.4c6 restarting backfill on osd.4 from (0'0,0'0] MAX to 6754'24942 2014-09-18 18:00:15.616374 7f4c97241700 0 log_channel(default) log [INF] : 2.120 restarting backfill on osd.4 from (0'0,0'0] MAX to 6776'27398 2014-09-18 18:00:17.773618 7f4c884e7700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6828/45709 pipe(0x21f0d600 sd=134 :6807 s=2 pgs=16 cs=1 l=0 c=0x21e80f20).fault, initiating reconnect 2014-09-18 18:00:17.773750 7f4c8016b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6828/45709 pipe(0x21f0d600 sd=134 :6807 s=1 pgs=16 cs=2 l=0 c=0x21e80f20).fault 2014-09-18 18:00:17.773968 7f4c75fca700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6843/45709 pipe(0x2a5ce3c0 sd=202 :0 s=1 pgs=0 cs=0 l=1 c=0x101ce840).fault 2014-09-18 18:00:17.773996 7f4c893f6700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6844/45709 pipe(0x2a5cfc80 sd=283 :0 s=1 pgs=0 cs=0 l=1 c=0x101ce6e0).fault 2014-09-18 18:00:18.623518 7f4c96a40700 0 log_channel(default) log [INF] : 2.1a2 restarting backfill on osd.4 from (0'0,0'0] MAX to 6869'33524 2014-09-18 18:00:19.786561 7f4c891f4700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6833/18678 pipe(0x123c0000 sd=143 :6807 s=2 pgs=9 cs=1 l=0 c=0x107372e0).fault with nothing to send, going to standby 2014-09-18 18:00:19.788665 7f4c8b618700 0 -- 10.242.42.178:0/2483 >> 10.242.42.174:6840/18678 pipe(0x2a5ce680 sd=28 :0 s=1 pgs=0 cs=0 l=1 c=0x1db61b80).fault 2014-09-18 18:00:19.788670 7f4c8ba1c700 0 -- 10.242.42.178:0/2483 >> 10.242.42.174:6841/18678 pipe(0x2a5cd600 sd=64 :0 s=1 pgs=0 cs=0 l=1 c=0x101ce580).fault 2014-09-18 18:00:19.868977 7f4c8b81a700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.178:6841/4511 pipe(0x21f0e940 sd=147 :6807 s=0 pgs=0 cs=0 l=0 c=0x10f492e0).accept connect_seq 0 vs existing 0 state connecting 2014-09-18 18:00:19.873474 7f4c8bb1d700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6838/11580 pipe(0x125d4dc0 sd=202 :0 s=1 pgs=0 cs=0 l=1 c=0x101ce420).fault 2014-09-18 18:00:19.873619 7f4c8c123700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6837/11580 pipe(0xe970dc0 sd=193 :0 s=1 pgs=0 cs=0 l=1 c=0x101e3a20).fault 2014-09-18 18:00:19.873648 7f4c87add700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6836/11580 pipe(0xe255700 sd=48 :6807 s=2 pgs=4411 cs=3 l=0 c=0x49039c0).fault with nothing to send, going to standby 2014-09-18 18:00:19.896479 7f4c8238d700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6833/9454 pipe(0x1d62c100 sd=195 :0 s=1 pgs=0 cs=0 l=1 c=0x101ce160).fault 2014-09-18 18:00:19.896506 7f4c8a608700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6831/9454 pipe(0xe254100 sd=38 :6807 s=2 pgs=4204 cs=3 l=0 c=0xdd8d2e0).fault with nothing to send, going to standby 2014-09-18 18:00:19.896619 7f4c7d842700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6832/9454 pipe(0x125d58c0 sd=195 :0 s=1 pgs=0 cs=0 l=1 c=0x101ce2c0).fault 2014-09-18 18:00:19.903274 7f4c743ae700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6804/38919 pipe(0x123c3c80 sd=52 :6807 s=2 pgs=19 cs=1 l=0 c=0x10736940).fault with nothing to send, going to standby 2014-09-18 18:00:19.904519 7f4c8c628700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6841/38919 pipe(0xe2538c0 sd=74 :0 s=1 pgs=0 cs=0 l=1 c=0x101cf080).fault 2014-09-18 18:00:19.904524 7f4c8c325700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6840/38919 pipe(0xe2559c0 sd=142 :0 s=1 pgs=0 cs=0 l=1 c=0x101cfce0).fault 2014-09-18 18:00:20.605692 7f4c889ec700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6804/38919 pipe(0x123c3c80 sd=40 :6807 s=1 pgs=19 cs=2 l=0 c=0x10736940).fault 2014-09-18 18:00:22.634101 7f4c853bd700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.174:6833/18678 pipe(0x123c0000 sd=48 :6807 s=1 pgs=9 cs=2 l=0 c=0x107372e0).fault 2014-09-18 18:00:23.159366 7f4c8c426700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6845/46347 pipe(0x21f0c840 sd=144 :6807 s=2 pgs=16 cs=1 l=0 c=0x142f7600).fault, initiating reconnect 2014-09-18 18:00:23.159623 7f4c87cdf700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6845/46347 pipe(0x21f0c840 sd=64 :6807 s=1 pgs=16 cs=2 l=0 c=0x142f7600).fault 2014-09-18 18:00:23.159733 7f4c891f4700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6846/46347 pipe(0xe970b00 sd=74 :0 s=1 pgs=0 cs=0 l=1 c=0x101d07e0).fault 2014-09-18 18:00:23.159773 7f4c87add700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6847/46347 pipe(0x13d0eb00 sd=85 :0 s=1 pgs=0 cs=0 l=1 c=0x12836c60).fault 2014-09-18 18:00:30.748333 7f4c7d53f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6848/47577 pipe(0x11074840 sd=161 :6807 s=2 pgs=18 cs=1 l=0 c=0x1395fa20).fault, initiating reconnect 2014-09-18 18:00:30.748495 7f4ca11d7700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6848/47577 pipe(0x11074840 sd=40 :6807 s=1 pgs=18 cs=2 l=0 c=0x1395fa20).fault 2014-09-18 18:00:30.748698 7f4c893f6700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6860/47577 pipe(0x140f0940 sd=48 :0 s=1 pgs=0 cs=0 l=1 c=0x1e1c4160).fault 2014-09-18 18:00:30.748705 7f4c8ba1c700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6861/47577 pipe(0x140ee000 sd=52 :0 s=1 pgs=0 cs=0 l=1 c=0x23a9fde0).fault 2014-09-18 18:00:30.836695 7f4c71681700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.168:6836/13436 pipe(0x123c3700 sd=53 :6807 s=2 pgs=41 cs=1 l=0 c=0x14aa78c0).fault with nothing to send, going to standby 2014-09-18 18:00:37.137591 7f4ca8339700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6862/49029 pipe(0x11077440 sd=222 :6807 s=2 pgs=17 cs=1 l=0 c=0x1395f760).fault with nothing to send, going to standby 2014-09-18 18:00:37.137981 7f4c72b96700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6864/49029 pipe(0x2a5ce3c0 sd=287 :0 s=1 pgs=0 cs=0 l=1 c=0x1395ffa0).fault 2014-09-18 18:00:37.138001 7f4c83ca6700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6863/49029 pipe(0x2a5cfc80 sd=195 :0 s=1 pgs=0 cs=0 l=1 c=0x1395fe40).fault 2014-09-18 18:00:37.520880 7f4ca14da700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6862/49029 pipe(0x11077440 sd=195 :6807 s=1 pgs=17 cs=2 l=0 c=0x1395f760).fault 2014-09-18 18:00:55.813980 7f4caae41700 -1 osd.42 11426 heartbeat_check: no reply from osd.41 since back 2014-09-18 18:00:35.473662 front 2014-09-18 18:00:35.473662 (cutoff 2014-09-18 18:00:35.813962) 2014-09-18 18:00:56.480531 7f4c8fa32700 -1 osd.42 11426 heartbeat_check: no reply from osd.41 since back 2014-09-18 18:00:35.473662 front 2014-09-18 18:00:35.473662 (cutoff 2014-09-18 18:00:36.480528) 2014-09-18 18:00:56.814088 7f4caae41700 -1 osd.42 11426 heartbeat_check: no reply from osd.41 since back 2014-09-18 18:00:35.473662 front 2014-09-18 18:00:35.473662 (cutoff 2014-09-18 18:00:36.814084) 2014-09-18 18:00:57.814196 7f4caae41700 -1 osd.42 11426 heartbeat_check: no reply from osd.41 since back 2014-09-18 18:00:35.473662 front 2014-09-18 18:00:35.473662 (cutoff 2014-09-18 18:00:37.814191) 2014-09-18 18:01:00.614797 7f4c97241700 0 log_channel(default) log [INF] : 2.2c7 restarting backfill on osd.0 from (0'0,0'0] MAX to 6805'24696 2014-09-18 18:01:01.729264 7f4c97241700 0 log_channel(default) log [INF] : 2.74e restarting backfill on osd.4 from (0'0,0'0] MAX to 6775'35284 2014-09-18 18:01:02.098621 7f4c97241700 0 log_channel(default) log [INF] : 2.71 restarting backfill on osd.2 from (0'0,0'0] MAX to 6845'29618 2014-09-18 18:01:02.103370 7f4c97241700 0 log_channel(default) log [INF] : 2.1d8 restarting backfill on osd.2 from (0'0,0'0] MAX to 6865'30228 2014-09-18 18:01:02.476415 7f4c97241700 0 log_channel(default) log [INF] : 2.799 restarting backfill on osd.3 from (0'0,0'0] MAX to 6761'29900 2014-09-18 18:01:05.734697 7f4c892f5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.168:6831/1010800 pipe(0x11e03700 sd=68 :6807 s=2 pgs=6207 cs=3 l=0 c=0xfc71440).fault with nothing to send, going to standby 2014-09-18 18:01:29.877196 7f4c96a40700 0 log_channel(default) log [INF] : 2.1d8 restarting backfill on osd.11 from (0'0,0'0] MAX to 6865'30228 2014-09-18 18:01:31.628117 7f4c97241700 0 log_channel(default) log [INF] : 2.799 restarting backfill on osd.47 from (0'0,0'0] MAX to 6761'29900 2014-09-18 18:01:32.640103 7f4c96a40700 0 log_channel(default) log [INF] : 2.646 restarting backfill on osd.47 from (0'0,0'0] MAX to 6825'27460 2014-09-18 18:01:58.794830 7f4c78bf6700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6851/46765 pipe(0x11075b80 sd=267 :6807 s=2 pgs=29 cs=1 l=0 c=0x1395fb80).fault with nothing to send, going to standby 2014-09-18 18:01:58.795480 7f4c8a608700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6852/46765 pipe(0x11e00000 sd=99 :0 s=1 pgs=0 cs=0 l=1 c=0x10387fa0).fault 2014-09-18 18:01:58.795587 7f4c8c628700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6853/46765 pipe(0x11e03700 sd=141 :0 s=1 pgs=0 cs=0 l=1 c=0x10a5d020).fault 2014-09-18 18:01:59.366535 7f4c80670700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6811/8573 pipe(0xecf8580 sd=214 :6807 s=2 pgs=4272 cs=3 l=0 c=0xdd8a420).fault with nothing to send, going to standby 2014-09-18 18:01:59.366570 7f4c898fb700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6812/8573 pipe(0x13d51440 sd=64 :0 s=1 pgs=0 cs=0 l=1 c=0x10a5cc00).fault 2014-09-18 18:01:59.366574 7f4c8c325700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6813/8573 pipe(0x13d51c80 sd=202 :0 s=1 pgs=0 cs=0 l=1 c=0x10a5bfa0).fault 2014-09-18 18:01:59.645855 7f4c7c731700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6811/8573 pipe(0xecf8580 sd=212 :6807 s=1 pgs=4272 cs=4 l=0 c=0xdd8a420).fault 2014-09-18 18:02:00.438269 7f4c78cf7700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6856/46772 pipe(0x12591340 sd=260 :6807 s=2 pgs=14 cs=1 l=0 c=0x139619c0).fault with nothing to send, going to standby 2014-09-18 18:02:00.438978 7f4c878db700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6857/46772 pipe(0x13d50940 sd=218 :0 s=1 pgs=0 cs=0 l=1 c=0x10a5d5a0).fault 2014-09-18 18:02:00.439017 7f4c7d842700 0 -- 10.242.42.178:0/2483 >> 10.242.42.176:6858/46772 pipe(0x13d51180 sd=222 :0 s=1 pgs=0 cs=0 l=1 c=0x10a5d860).fault 2014-09-18 18:02:00.913819 7f4c78df8700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.176:6856/46772 pipe(0x12591340 sd=95 :6807 s=1 pgs=14 cs=2 l=0 c=0x139619c0).fault 2014-09-18 18:02:01.543228 7f4c762cd700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6820/22971 pipe(0x123c2680 sd=30 :6807 s=2 pgs=20 cs=1 l=0 c=0x1088bc80).fault with nothing to send, going to standby 2014-09-18 18:02:01.544604 7f4c854be700 0 -- 10.242.42.178:0/2483 >> 10.242.42.166:6844/22971 pipe(0x140ee840 sd=111 :0 s=1 pgs=0 cs=0 l=1 c=0x10a5d2e0).fault 2014-09-18 18:02:01.544609 7f4c8b517700 0 -- 10.242.42.178:0/2483 >> 10.242.42.166:6821/22971 pipe(0x13d50100 sd=72 :0 s=1 pgs=0 cs=0 l=1 c=0x10a5bce0).fault 2014-09-18 18:02:01.957319 7f4c761cc700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.166:6820/22971 pipe(0x123c2680 sd=30 :6807 s=1 pgs=20 cs=2 l=0 c=0x1088bc80).fault 2014-09-18 18:02:26.652023 7f4c96a40700 0 log_channel(default) log [INF] : 2.4f6 restarting backfill on osd.27 from (0'0,0'0] MAX to 6873'32776 2014-09-18 18:02:26.659272 7f4c97241700 0 log_channel(default) log [INF] : 2.74e restarting backfill on osd.27 from (0'0,0'0] MAX to 6775'35284 2014-09-18 18:02:26.664367 7f4c96a40700 0 log_channel(default) log [INF] : 2.2d8 restarting backfill on osd.27 from (0'0,0'0] MAX to 6825'27712 2014-09-18 18:02:26.669106 7f4c96a40700 0 log_channel(default) log [INF] : 2.410 restarting backfill on osd.27 from (0'0,0'0] MAX to 6840'31228 2014-09-18 18:02:27.510390 7f4c96a40700 0 log_channel(default) log [INF] : 2.799 restarting backfill on osd.27 from (0'0,0'0] MAX to 6761'29900 2014-09-18 18:02:27.657871 7f4c97241700 0 log_channel(default) log [INF] : 2.7fa restarting backfill on osd.27 from (0'0,0'0] MAX to 6761'32700 2014-09-18 18:02:27.663992 7f4c96a40700 0 log_channel(default) log [INF] : 2.5b4 restarting backfill on osd.27 from (0'0,0'0] MAX to 6792'28596 2014-09-18 18:02:27.665005 7f4c97241700 0 log_channel(default) log [INF] : 2.403 restarting backfill on osd.27 from (0'0,0'0] MAX to 6774'31098 2014-09-18 18:02:27.827551 7f4c97241700 0 log_channel(default) log [INF] : 2.78f restarting backfill on osd.27 from (0'0,0'0] MAX to 6789'22964 2014-09-18 18:02:28.553199 7f4c97241700 0 log_channel(default) log [INF] : 2.3b6 restarting backfill on osd.27 from (0'0,0'0] MAX to 6856'30926 2014-09-18 18:02:28.553601 7f4c96a40700 0 log_channel(default) log [INF] : 2.62 restarting backfill on osd.27 from (0'0,0'0] MAX to 6775'23328 2014-09-18 18:02:29.137174 7f4c97241700 0 log_channel(default) log [INF] : 2.11b restarting backfill on osd.27 from (0'0,0'0] MAX to 6775'28078 2014-09-18 18:02:35.632591 7f4c97241700 0 log_channel(default) log [INF] : 2.4f6 restarting backfill on osd.3 from (0'0,0'0] MAX to 6873'32776 2014-09-18 18:02:35.638187 7f4c96a40700 0 log_channel(default) log [INF] : 2.3b6 restarting backfill on osd.3 from (0'0,0'0] MAX to 6856'30926 2014-09-18 18:02:35.864902 7f4c97241700 0 log_channel(default) log [INF] : 2.719 restarting backfill on osd.47 from (0'0,0'0] MAX to 6842'29696 2014-09-18 18:02:37.909333 7f4ca16dc700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.168:6806/10347 pipe(0x123c0840 sd=279 :6807 s=2 pgs=49 cs=1 l=0 c=0x111e5c80).fault with nothing to send, going to standby 2014-09-18 18:03:11.530999 7f4c97241700 0 log_channel(default) log [INF] : 2.719 restarting backfill on osd.48 from (0'0,0'0] MAX to 6842'29696 2014-09-18 18:06:01.663049 7f4c96a40700 0 log_channel(default) log [INF] : 2.457 restarting backfill on osd.32 from (0'0,0'0] MAX to 6761'29864 2014-09-18 18:06:01.670575 7f4c97241700 0 log_channel(default) log [INF] : 2.71 restarting backfill on osd.58 from (0'0,0'0] MAX to 6845'29618 2014-09-18 18:06:02.663900 7f4c96a40700 0 log_channel(default) log [INF] : 2.178 restarting backfill on osd.32 from (0'0,0'0] MAX to 6874'18440 2014-09-18 18:06:02.667965 7f4c96a40700 0 log_channel(default) log [INF] : 2.519 restarting backfill on osd.57 from (0'0,0'0] MAX to 6868'25546 2014-09-18 18:06:02.668813 7f4c97241700 0 log_channel(default) log [INF] : 2.4c9 restarting backfill on osd.32 from (0'0,0'0] MAX to 6835'28020 2014-09-18 18:06:03.181932 7f4c775e0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6807/17207 pipe(0x21f0db80 sd=246 :6807 s=2 pgs=29 cs=1 l=0 c=0x13a4ffa0).fault with nothing to send, going to standby 2014-09-18 18:06:03.182588 7f4c88aed700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6808/17207 pipe(0x1443a580 sd=114 :0 s=1 pgs=0 cs=0 l=1 c=0x1d55f8c0).fault 2014-09-18 18:06:03.182607 7f4c84bb5700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6809/17207 pipe(0x1443b8c0 sd=287 :0 s=1 pgs=0 cs=0 l=1 c=0x1d560aa0).fault 2014-09-18 18:06:03.447186 7f4c96a40700 0 log_channel(default) log [INF] : 2.1a2 restarting backfill on osd.58 from (0'0,0'0] MAX to 6869'33524 2014-09-18 18:06:15.882213 7f4c71984700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6810/28634 pipe(0xecf9080 sd=102 :6807 s=2 pgs=32 cs=1 l=0 c=0x1395e9a0).fault, initiating reconnect 2014-09-18 18:06:15.882360 7f4c885e8700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6810/28634 pipe(0xecf9080 sd=102 :6807 s=1 pgs=32 cs=2 l=0 c=0x1395e9a0).fault 2014-09-18 18:06:15.882928 7f4c790fb700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6840/28634 pipe(0x125d79c0 sd=167 :0 s=1 pgs=0 cs=0 l=1 c=0x1d7c92e0).fault 2014-09-18 18:06:15.882956 7f4c740ab700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6841/28634 pipe(0x125d7c80 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x1d7c9180).fault 2014-09-18 18:06:21.886402 7f4c88aed700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6842/29102 pipe(0xecf8840 sd=167 :6807 s=2 pgs=25 cs=1 l=0 c=0x1395e840).fault, initiating reconnect 2014-09-18 18:06:21.886606 7f4c775e0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6842/29102 pipe(0xecf8840 sd=36 :6807 s=1 pgs=25 cs=2 l=0 c=0x1395e840).fault 2014-09-18 18:06:21.886811 7f4c84bb5700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6843/29102 pipe(0xe80b440 sd=167 :0 s=1 pgs=0 cs=0 l=1 c=0x141b83c0).fault 2014-09-18 18:06:21.886881 7f4c773de700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6844/29102 pipe(0xe808dc0 sd=287 :0 s=1 pgs=0 cs=0 l=1 c=0x141b6840).fault 2014-09-18 18:06:28.530618 7f4c82b95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6845/29579 pipe(0xecf8dc0 sd=117 :6807 s=2 pgs=32 cs=1 l=0 c=0x1395e6e0).fault with nothing to send, going to standby 2014-09-18 18:06:28.531367 7f4c775e0700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6846/29579 pipe(0xe808840 sd=114 :0 s=1 pgs=0 cs=0 l=1 c=0x141b7760).fault 2014-09-18 18:06:28.531381 7f4c88aed700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6847/29579 pipe(0xe80bc80 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x141b7ce0).fault 2014-09-18 18:06:34.670406 7f4c773de700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6848/30058 pipe(0xecfb180 sd=114 :6807 s=2 pgs=28 cs=1 l=0 c=0x1395e580).fault with nothing to send, going to standby 2014-09-18 18:06:34.671032 7f4c7f55f700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6849/30058 pipe(0xe8098c0 sd=205 :0 s=1 pgs=0 cs=0 l=1 c=0x1d662840).fault 2014-09-18 18:06:34.671045 7f4c80b75700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6850/30058 pipe(0xe809600 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x1da6e000).fault 2014-09-18 18:06:40.695562 7f4c82b95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6851/30534 pipe(0xecfa3c0 sd=117 :6807 s=2 pgs=33 cs=1 l=0 c=0x1395e420).fault, initiating reconnect 2014-09-18 18:06:40.695658 7f4c775e0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6851/30534 pipe(0xecfa3c0 sd=117 :6807 s=1 pgs=33 cs=2 l=0 c=0x1395e420).fault 2014-09-18 18:06:40.696205 7f4ca853b700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6852/30534 pipe(0x123c2680 sd=140 :0 s=1 pgs=0 cs=0 l=1 c=0x120fb440).fault 2014-09-18 18:06:40.696234 7f4c88aed700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6853/30534 pipe(0xfd04dc0 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x1395e6e0).fault 2014-09-18 18:06:45.907725 7f4c84bb5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/31029 pipe(0xecf9340 sd=117 :6807 s=2 pgs=33 cs=1 l=0 c=0x1395e2c0).fault, initiating reconnect 2014-09-18 18:06:45.907877 7f4c7f55f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/31029 pipe(0xecf9340 sd=117 :6807 s=1 pgs=33 cs=2 l=0 c=0x1395e2c0).fault 2014-09-18 18:06:45.908315 7f4c80b75700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/31029 pipe(0xfd07700 sd=167 :0 s=1 pgs=0 cs=0 l=1 c=0x1395e580).fault 2014-09-18 18:06:45.908355 7f4c773de700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/31029 pipe(0xfd05080 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x1277c7e0).fault 2014-09-18 18:06:51.908751 7f4c775e0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/31492 pipe(0xecfb440 sd=117 :6807 s=2 pgs=32 cs=1 l=0 c=0x1395e160).fault, initiating reconnect 2014-09-18 18:06:51.908849 7f4c88aed700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/31492 pipe(0xecfb440 sd=117 :6807 s=1 pgs=32 cs=2 l=0 c=0x1395e160).fault 2014-09-18 18:06:51.909212 7f4c80b75700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/31492 pipe(0x1443bb80 sd=174 :0 s=1 pgs=0 cs=0 l=1 c=0x1d5a7b80).fault 2014-09-18 18:06:51.909220 7f4c84bb5700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/31492 pipe(0xfd04dc0 sd=114 :0 s=1 pgs=0 cs=0 l=1 c=0xe567600).fault 2014-09-18 18:07:00.893201 7f4c7f55f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6860/31971 pipe(0xecfac00 sd=117 :6807 s=2 pgs=33 cs=1 l=0 c=0x1395e000).fault with nothing to send, going to standby 2014-09-18 18:07:00.893609 7f4c82b95700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6861/31971 pipe(0x1443b340 sd=205 :0 s=1 pgs=0 cs=0 l=1 c=0x1217a580).fault 2014-09-18 18:07:00.893636 7f4ca853b700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6862/31971 pipe(0x146b6dc0 sd=142 :0 s=1 pgs=0 cs=0 l=1 c=0x1db34000).fault 2014-09-18 18:07:06.902178 7f4c84bb5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6807/32449 pipe(0xecf82c0 sd=39 :6807 s=2 pgs=20 cs=1 l=0 c=0x10b262c0).fault, initiating reconnect 2014-09-18 18:07:06.902264 7f4c773de700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6808/32449 pipe(0x1443c3c0 sd=30 :0 s=1 pgs=0 cs=0 l=1 c=0x14f4db20).fault 2014-09-18 18:07:06.902276 7f4c88aed700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6807/32449 pipe(0xecf82c0 sd=39 :6807 s=1 pgs=20 cs=2 l=0 c=0x10b262c0).fault 2014-09-18 18:07:06.902316 7f4ca853b700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6809/32449 pipe(0x1443d180 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x14f4cd60).fault 2014-09-18 18:07:13.002558 7f4c7f55f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6863/32993 pipe(0x150159c0 sd=140 :6807 s=2 pgs=16 cs=1 l=0 c=0x10b269a0).fault, initiating reconnect 2014-09-18 18:07:13.002679 7f4c82b95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6863/32993 pipe(0x150159c0 sd=102 :6807 s=1 pgs=16 cs=2 l=0 c=0x10b269a0).fault 2014-09-18 18:07:13.003326 7f4c88aed700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6864/32993 pipe(0x1443bb80 sd=140 :0 s=1 pgs=0 cs=0 l=1 c=0x14f4ba20).fault 2014-09-18 18:07:13.003351 7f4ca853b700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6865/32993 pipe(0x1443b340 sd=82 :0 s=1 pgs=0 cs=0 l=1 c=0x14f4d700).fault 2014-09-18 18:07:19.106836 7f4c84bb5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6866/33469 pipe(0xe973180 sd=114 :6807 s=2 pgs=24 cs=1 l=0 c=0x141b6000).fault, initiating reconnect 2014-09-18 18:07:19.106944 7f4c773de700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6866/33469 pipe(0xe973180 sd=59 :6807 s=1 pgs=24 cs=2 l=0 c=0x141b6000).fault 2014-09-18 18:07:19.107671 7f4c80b75700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6867/33469 pipe(0x125d6680 sd=114 :0 s=1 pgs=0 cs=0 l=1 c=0x110e63c0).fault 2014-09-18 18:07:19.107698 7f4c775e0700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6868/33469 pipe(0xfd05b80 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x12df5080).fault 2014-09-18 18:07:25.207619 7f4c82b95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6869/33945 pipe(0xe9739c0 sd=117 :6807 s=2 pgs=32 cs=1 l=0 c=0x141b6b00).fault, initiating reconnect 2014-09-18 18:07:25.207753 7f4ca853b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6869/33945 pipe(0xe9739c0 sd=117 :6807 s=1 pgs=32 cs=2 l=0 c=0x141b6b00).fault 2014-09-18 18:07:25.208318 7f4c7f55f700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6870/33945 pipe(0x1443a2c0 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x11177a20).fault 2014-09-18 18:07:25.208342 7f4c88aed700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6871/33945 pipe(0x1443a840 sd=114 :0 s=1 pgs=0 cs=0 l=1 c=0x10dbc680).fault 2014-09-18 18:07:30.871741 7f4c84bb5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6810/34424 pipe(0xe971340 sd=36 :6807 s=2 pgs=32 cs=1 l=0 c=0x141b7ce0).fault, initiating reconnect 2014-09-18 18:07:30.871839 7f4c80b75700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6810/34424 pipe(0xe971340 sd=36 :6807 s=1 pgs=32 cs=2 l=0 c=0x141b7ce0).fault 2014-09-18 18:07:30.872095 7f4ca853b700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6840/34424 pipe(0xfd04dc0 sd=174 :0 s=1 pgs=0 cs=0 l=1 c=0xf151c80).fault 2014-09-18 18:07:30.872152 7f4c7f55f700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6841/34424 pipe(0xfd05b80 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0xf151860).fault 2014-09-18 18:07:36.923383 7f4c82b95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6842/34895 pipe(0x102e6dc0 sd=140 :6807 s=2 pgs=33 cs=1 l=0 c=0x141b7760).fault with nothing to send, going to standby 2014-09-18 18:07:36.923823 7f4c773de700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6843/34895 pipe(0x1443adc0 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x1e1c42c0).fault 2014-09-18 18:07:36.923866 7f4c775e0700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6844/34895 pipe(0x1443c940 sd=39 :0 s=1 pgs=0 cs=0 l=1 c=0x1e1c4420).fault 2014-09-18 18:07:43.034566 7f4c84bb5700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6872/35399 pipe(0x1d62cec0 sd=114 :6807 s=2 pgs=32 cs=1 l=0 c=0x141b7080).fault with nothing to send, going to standby 2014-09-18 18:07:43.034707 7f4c80b75700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6873/35399 pipe(0x1443a840 sd=205 :0 s=1 pgs=0 cs=0 l=1 c=0x1e1c4000).fault 2014-09-18 18:07:43.034720 7f4c7f55f700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6874/35399 pipe(0x1443a2c0 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x1e1c6940).fault 2014-09-18 18:07:49.047399 7f4c8e22f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6845/35878 pipe(0x140eeb00 sd=142 :6807 s=2 pgs=28 cs=1 l=0 c=0x141b7340).fault, initiating reconnect 2014-09-18 18:07:49.047654 7f4c82b95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6845/35878 pipe(0x140eeb00 sd=127 :6807 s=1 pgs=28 cs=2 l=0 c=0x141b7340).fault 2014-09-18 18:07:49.048205 7f4c775e0700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6846/35878 pipe(0x10b63600 sd=142 :0 s=1 pgs=0 cs=0 l=1 c=0x12df49a0).fault 2014-09-18 18:07:49.048272 7f4c773de700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6847/35878 pipe(0xe4b0100 sd=117 :0 s=1 pgs=0 cs=0 l=1 c=0x12df46e0).fault 2014-09-18 18:07:55.167177 7f4ca853b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6875/36362 pipe(0x14b519c0 sd=127 :6807 s=2 pgs=32 cs=1 l=0 c=0x141b8100).fault, initiating reconnect 2014-09-18 18:07:55.167365 7f4c7f55f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6875/36362 pipe(0x14b519c0 sd=29 :6807 s=1 pgs=32 cs=2 l=0 c=0x141b8100).fault 2014-09-18 18:07:55.167962 7f4c80b75700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6876/36362 pipe(0x1443c940 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x141b9c80).fault 2014-09-18 18:07:55.167980 7f4c8e22f700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6877/36362 pipe(0x1443adc0 sd=140 :0 s=1 pgs=0 cs=0 l=1 c=0x141b92e0).fault 2014-09-18 18:08:00.931638 7f4c82b95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6848/36851 pipe(0x14b4e000 sd=82 :6807 s=2 pgs=32 cs=1 l=0 c=0x13aaf600).fault, initiating reconnect 2014-09-18 18:08:00.931770 7f4c773de700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6848/36851 pipe(0x14b4e000 sd=22 :6807 s=1 pgs=32 cs=2 l=0 c=0x13aaf600).fault 2014-09-18 18:08:00.932375 7f4c7f55f700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6849/36851 pipe(0x102e8100 sd=82 :0 s=1 pgs=0 cs=0 l=1 c=0x110e5ce0).fault 2014-09-18 18:08:00.932402 7f4c80b75700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6850/36851 pipe(0x102e9700 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x110e6100).fault 2014-09-18 18:08:06.936809 7f4ca853b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6878/37327 pipe(0x10b65c80 sd=117 :6807 s=2 pgs=24 cs=1 l=0 c=0x10387e40).fault with nothing to send, going to standby 2014-09-18 18:08:06.937463 7f4c82b95700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6879/37327 pipe(0x12b82100 sd=31 :0 s=1 pgs=0 cs=0 l=1 c=0x10b271e0).fault 2014-09-18 18:08:06.937567 7f4c80b75700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6880/37327 pipe(0x12b802c0 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x111e4940).fault 2014-09-18 18:08:13.018829 7f4c773de700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6881/37802 pipe(0x10b64680 sd=114 :6807 s=2 pgs=32 cs=1 l=0 c=0x103862c0).fault, initiating reconnect 2014-09-18 18:08:13.018936 7f4c7f55f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6881/37802 pipe(0x10b64680 sd=70 :6807 s=1 pgs=32 cs=2 l=0 c=0x103862c0).fault 2014-09-18 18:08:13.019543 7f4c775e0700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6882/37802 pipe(0x47563c0 sd=114 :0 s=1 pgs=0 cs=0 l=1 c=0x13aae160).fault 2014-09-18 18:08:13.019593 7f4c84bb5700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6883/37802 pipe(0x4754580 sd=246 :0 s=1 pgs=0 cs=0 l=1 c=0x13ab0940).fault 2014-09-18 18:08:16.842340 7f4c96a40700 0 log_channel(default) log [INF] : 2.4f6 restarting backfill on osd.48 from (0'0,0'0] MAX to 6873'32776 2014-09-18 18:08:16.843944 7f4c97241700 0 log_channel(default) log [INF] : 2.62 restarting backfill on osd.47 from (0'0,0'0] MAX to 6775'23328 2014-09-18 18:08:16.852641 7f4c97241700 0 log_channel(default) log [INF] : 2.4b1 restarting backfill on osd.48 from (0'0,0'0] MAX to 6745'24816 2014-09-18 18:08:16.855520 7f4c97241700 0 log_channel(default) log [INF] : 2.4c1 restarting backfill on osd.6 from (0'0,0'0] MAX to 6761'26972 2014-09-18 18:08:16.992095 7f4c97241700 0 log_channel(default) log [INF] : 2.519 restarting backfill on osd.9 from (0'0,0'0] MAX to 6868'25546 2014-09-18 18:08:19.335041 7f4c82b95700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6851/38278 pipe(0x10b65700 sd=127 :6807 s=2 pgs=33 cs=1 l=0 c=0x12b3ac00).fault with nothing to send, going to standby 2014-09-18 18:08:19.335812 7f4c773de700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6853/38278 pipe(0x24d143c0 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x100c3600).fault 2014-09-18 18:08:19.335832 7f4c84bb5700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6852/38278 pipe(0x24d13340 sd=31 :0 s=1 pgs=0 cs=0 l=1 c=0x100c4d60).fault 2014-09-18 18:08:23.923764 7f4c96a40700 0 log_channel(default) log [INF] : 2.4c1 restarting backfill on osd.48 from (0'0,0'0] MAX to 6761'26972 2014-09-18 18:08:25.144743 7f4c7f55f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3a680).fault, initiating reconnect 2014-09-18 18:08:25.144892 7f4c775e0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).fault 2014-09-18 18:08:25.145416 7f4c84bb5700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=167 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault 2014-09-18 18:08:25.145417 7f4c82b95700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault 2014-09-18 18:08:28.330059 7f4c97241700 0 log_channel(default) log [INF] : 2.410 restarting backfill on osd.6 from (0'0,0'0] MAX to 6840'31228 2014-09-18 18:08:30.680342 7f4c97241700 0 log_channel(default) log [INF] : 2.178 restarting backfill on osd.20 from (0'0,0'0] MAX to 6874'18440 2014-09-18 18:08:30.681038 7f4c96a40700 0 log_channel(default) log [INF] : 2.6f8 restarting backfill on osd.35 from (0'0,0'0] MAX to 6788'24342 2014-09-18 18:08:30.684715 7f4c97241700 0 log_channel(default) log [INF] : 2.179 restarting backfill on osd.35 from (0'0,0'0] MAX to 6752'29654 2014-09-18 18:08:30.688104 7f4c97241700 0 log_channel(default) log [INF] : 2.410 restarting backfill on osd.56 from (0'0,0'0] MAX to 6840'31228 2014-09-18 18:08:30.688533 7f4c96a40700 0 log_channel(default) log [INF] : 2.11b restarting backfill on osd.9 from (0'0,0'0] MAX to 6775'28078 2014-09-18 18:08:30.975106 7f4ca853b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=127 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3aec0).fault with nothing to send, going to standby 2014-09-18 18:08:30.975610 7f4c71984700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=199 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).fault 2014-09-18 18:08:30.975635 7f4c790fb700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).fault 2014-09-18 18:08:31.338216 7f4c773de700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=93 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).fault 2014-09-18 18:08:32.435228 7f4c97241700 0 log_channel(default) log [INF] : 2.519 restarting backfill on osd.4 from (0'0,0'0] MAX to 6868'25546 2014-09-18 18:08:36.959506 7f4c8e22f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3b2e0).fault with nothing to send, going to standby 2014-09-18 18:08:36.959913 7f4c773de700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).fault 2014-09-18 18:08:36.959917 7f4c71984700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).fault 2014-09-18 18:08:37.077672 7f4c775e0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).fault 2014-09-18 18:08:37.691722 7f4c97241700 0 log_channel(default) log [INF] : 2.2d8 restarting backfill on osd.2 from (0'0,0'0] MAX to 6825'27712 2014-09-18 18:08:37.691823 7f4c96a40700 0 log_channel(default) log [INF] : 2.204 restarting backfill on osd.0 from (0'0,0'0] MAX to 6810'26722 2014-09-18 18:08:37.717934 7f4c96a40700 -1 osd/PGLog.h: In function 'void PGLog::IndexedLog::claim_log_and_clear_rollback_info(const pg_log_t&)' thread 7f4c96a40700 time 2014-09-18 18:08:37.701817 osd/PGLog.h: 87: FAILED assert(rollback_info_trimmed_to == head) ceph version 0.84-sd-sprint4 (3215c520e1306f50d0094b5646636c02456c9df4) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xb776db] 2: (PG::RecoveryState::Stray::react(PG::MLogRec const&)+0x1f9) [0x7e1d19] 3: (boost::statechart::simple_state, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x1f4) [0x81a944] 4: (boost::statechart::state_machine, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x805d6b] 5: (PG::handle_peering_event(std::tr1::shared_ptr, PG::RecoveryCtx*)+0x1ce) [0x7b5b8e] 6: (OSD::process_peering_events(std::list > const&, ThreadPool::TPHandle&)+0x2b0) [0x6a3140] 7: (OSD::PeeringWQ::_process(std::list > const&, ThreadPool::TPHandle&)+0x18) [0x6f83c8] 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa46) [0xb68866] 9: (ThreadPool::WorkThread::entry()+0x10) [0xb69910] 10: (()+0x8182) [0x7f4cb21da182] 11: (clone()+0x6d) [0x7f4cb05c030d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -10000> 2014-09-18 18:08:22.915663 7f4c9f251700 5 -- op tracker -- , seq: 85539, time: 2014-09-18 18:08:22.915661, event: started, op: pg_notify(2.1bb(9) epoch 11701) -9999> 2014-09-18 18:08:22.915684 7f4c9f251700 5 -- op tracker -- , seq: 85539, time: 2014-09-18 18:08:22.915682, event: done, op: pg_notify(2.1bb(9) epoch 11701) -9998> 2014-09-18 18:08:22.915711 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11701..11701 src has 1..11701) v3 -- ?+0 0x24d1d180 con 0x1088aec0 -9997> 2014-09-18 18:08:22.915736 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_query(2.410 epoch 11701) v3 -- ?+0 0xf6dfa40 con 0x1088aec0 -9996> 2014-09-18 18:08:22.915751 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.410 epoch 11701) v3 -- ?+0 0x2b93c940 con 0x1395f8c0 -9995> 2014-09-18 18:08:22.915875 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3570 ==== pg_notify(2.2c7(139) epoch 11701) v5 ==== 8651+0+0 (1542592925 0 0) 0x11bc1de0 con 0xcbecec0 -9994> 2014-09-18 18:08:22.915899 7f4c9f251700 5 -- op tracker -- , seq: 85540, time: 2014-09-18 18:08:22.915694, event: header_read, op: pg_notify(2.2c7(139) epoch 11701) -9993> 2014-09-18 18:08:22.915911 7f4c9f251700 5 -- op tracker -- , seq: 85540, time: 2014-09-18 18:08:22.915697, event: throttled, op: pg_notify(2.2c7(139) epoch 11701) -9992> 2014-09-18 18:08:22.915918 7f4c9f251700 5 -- op tracker -- , seq: 85540, time: 2014-09-18 18:08:22.915852, event: all_read, op: pg_notify(2.2c7(139) epoch 11701) -9991> 2014-09-18 18:08:22.915925 7f4c9f251700 5 -- op tracker -- , seq: 85540, time: 2014-09-18 18:08:22.915893, event: dispatched, op: pg_notify(2.2c7(139) epoch 11701) -9990> 2014-09-18 18:08:22.915933 7f4c9f251700 5 -- op tracker -- , seq: 85540, time: 2014-09-18 18:08:22.915930, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11701) -9989> 2014-09-18 18:08:22.915943 7f4c9f251700 5 -- op tracker -- , seq: 85540, time: 2014-09-18 18:08:22.915940, event: started, op: pg_notify(2.2c7(139) epoch 11701) -9988> 2014-09-18 18:08:22.915969 7f4c9f251700 5 -- op tracker -- , seq: 85540, time: 2014-09-18 18:08:22.915966, event: done, op: pg_notify(2.2c7(139) epoch 11701) -9987> 2014-09-18 18:08:22.916053 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1179 ==== pg_notify(2.1d8(9) epoch 11701) v5 ==== 1261+0+0 (1225975860 0 0) 0x11bbed20 con 0x1088b440 -9986> 2014-09-18 18:08:22.916067 7f4c9f251700 5 -- op tracker -- , seq: 85541, time: 2014-09-18 18:08:22.915900, event: header_read, op: pg_notify(2.1d8(9) epoch 11701) -9985> 2014-09-18 18:08:22.916076 7f4c9f251700 5 -- op tracker -- , seq: 85541, time: 2014-09-18 18:08:22.915901, event: throttled, op: pg_notify(2.1d8(9) epoch 11701) -9984> 2014-09-18 18:08:22.916082 7f4c9f251700 5 -- op tracker -- , seq: 85541, time: 2014-09-18 18:08:22.915938, event: all_read, op: pg_notify(2.1d8(9) epoch 11701) -9983> 2014-09-18 18:08:22.916088 7f4c9f251700 5 -- op tracker -- , seq: 85541, time: 2014-09-18 18:08:22.916063, event: dispatched, op: pg_notify(2.1d8(9) epoch 11701) -9982> 2014-09-18 18:08:22.916095 7f4c9f251700 5 -- op tracker -- , seq: 85541, time: 2014-09-18 18:08:22.916093, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11701) -9981> 2014-09-18 18:08:22.916103 7f4c9f251700 5 -- op tracker -- , seq: 85541, time: 2014-09-18 18:08:22.916101, event: started, op: pg_notify(2.1d8(9) epoch 11701) -9980> 2014-09-18 18:08:22.916125 7f4c9f251700 5 -- op tracker -- , seq: 85541, time: 2014-09-18 18:08:22.916123, event: done, op: pg_notify(2.1d8(9) epoch 11701) -9979> 2014-09-18 18:08:22.918087 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 101 ==== osd_map(11701..11701 src has 1..11701) v3 ==== 2801+0+0 (2938961775 0 0) 0x24d1b200 con 0x1395edc0 -9978> 2014-09-18 18:08:22.918104 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11701,11701], i have 11701, src has [1,11701] -9977> 2014-09-18 18:08:22.918113 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 102 ==== pg_query(2.6e3 epoch 11701) v3 ==== 144+0+0 (3264362380 0 0) 0xf91da20 con 0x1395edc0 -9976> 2014-09-18 18:08:22.918123 7f4c9f251700 5 -- op tracker -- , seq: 85542, time: 2014-09-18 18:08:22.918087, event: header_read, op: pg_query(2.6e3 epoch 11701) -9975> 2014-09-18 18:08:22.918132 7f4c9f251700 5 -- op tracker -- , seq: 85542, time: 2014-09-18 18:08:22.918088, event: throttled, op: pg_query(2.6e3 epoch 11701) -9974> 2014-09-18 18:08:22.918139 7f4c9f251700 5 -- op tracker -- , seq: 85542, time: 2014-09-18 18:08:22.918107, event: all_read, op: pg_query(2.6e3 epoch 11701) -9973> 2014-09-18 18:08:22.918145 7f4c9f251700 5 -- op tracker -- , seq: 85542, time: 2014-09-18 18:08:22.918120, event: dispatched, op: pg_query(2.6e3 epoch 11701) -9972> 2014-09-18 18:08:22.918152 7f4c9f251700 5 -- op tracker -- , seq: 85542, time: 2014-09-18 18:08:22.918150, event: waiting_for_osdmap, op: pg_query(2.6e3 epoch 11701) -9971> 2014-09-18 18:08:22.918160 7f4c9f251700 5 -- op tracker -- , seq: 85542, time: 2014-09-18 18:08:22.918158, event: started, op: pg_query(2.6e3 epoch 11701) -9970> 2014-09-18 18:08:22.918174 7f4c9f251700 5 -- op tracker -- , seq: 85542, time: 2014-09-18 18:08:22.918172, event: done, op: pg_query(2.6e3 epoch 11701) -9969> 2014-09-18 18:08:22.918203 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6e3(27) epoch 11701) v5 -- ?+0 0xe1ac760 con 0x1395edc0 -9968> 2014-09-18 18:08:22.918383 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 509 ==== osd_map(11701..11701 src has 1..11701) v3 ==== 2801+0+0 (2938961775 0 0) 0x2fba0640 con 0x1395ef20 -9967> 2014-09-18 18:08:22.918397 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11701,11701], i have 11701, src has [1,11701] -9966> 2014-09-18 18:08:22.918441 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 510 ==== pg_notify(2.1d8(20) epoch 11701) v5 ==== 1878+0+0 (3574000922 0 0) 0x11bc10c0 con 0x1395ef20 -9965> 2014-09-18 18:08:22.918451 7f4c9f251700 5 -- op tracker -- , seq: 85543, time: 2014-09-18 18:08:22.918398, event: header_read, op: pg_notify(2.1d8(20) epoch 11701) -9964> 2014-09-18 18:08:22.918460 7f4c9f251700 5 -- op tracker -- , seq: 85543, time: 2014-09-18 18:08:22.918398, event: throttled, op: pg_notify(2.1d8(20) epoch 11701) -9963> 2014-09-18 18:08:22.918466 7f4c9f251700 5 -- op tracker -- , seq: 85543, time: 2014-09-18 18:08:22.918431, event: all_read, op: pg_notify(2.1d8(20) epoch 11701) -9962> 2014-09-18 18:08:22.918471 7f4c9f251700 5 -- op tracker -- , seq: 85543, time: 2014-09-18 18:08:22.918449, event: dispatched, op: pg_notify(2.1d8(20) epoch 11701) -9961> 2014-09-18 18:08:22.918479 7f4c9f251700 5 -- op tracker -- , seq: 85543, time: 2014-09-18 18:08:22.918477, event: waiting_for_osdmap, op: pg_notify(2.1d8(20) epoch 11701) -9960> 2014-09-18 18:08:22.918492 7f4c9f251700 5 -- op tracker -- , seq: 85543, time: 2014-09-18 18:08:22.918490, event: started, op: pg_notify(2.1d8(20) epoch 11701) -9959> 2014-09-18 18:08:22.918509 7f4c9f251700 5 -- op tracker -- , seq: 85543, time: 2014-09-18 18:08:22.918508, event: done, op: pg_notify(2.1d8(20) epoch 11701) -9958> 2014-09-18 18:08:22.918551 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 1993 ==== osd_map(11701..11701 src has 1..11701) v3 ==== 2801+0+0 (2938961775 0 0) 0x24d1c880 con 0xe2d9a20 -9957> 2014-09-18 18:08:22.918563 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11701,11701], i have 11701, src has [1,11701] -9956> 2014-09-18 18:08:22.918630 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 1994 ==== pg_notify(2.1d8(15),2.26d(9) epoch 11701) v5 ==== 2848+0+0 (1408539259 0 0) 0xe76dc20 con 0xe2d9a20 -9955> 2014-09-18 18:08:22.918652 7f4c9f251700 5 -- op tracker -- , seq: 85544, time: 2014-09-18 18:08:22.918560, event: header_read, op: pg_notify(2.1d8(15),2.26d(9) epoch 11701) -9954> 2014-09-18 18:08:22.918665 7f4c9f251700 5 -- op tracker -- , seq: 85544, time: 2014-09-18 18:08:22.918560, event: throttled, op: pg_notify(2.1d8(15),2.26d(9) epoch 11701) -9953> 2014-09-18 18:08:22.918676 7f4c9f251700 5 -- op tracker -- , seq: 85544, time: 2014-09-18 18:08:22.918613, event: all_read, op: pg_notify(2.1d8(15),2.26d(9) epoch 11701) -9952> 2014-09-18 18:08:22.918683 7f4c9f251700 5 -- op tracker -- , seq: 85544, time: 2014-09-18 18:08:22.918646, event: dispatched, op: pg_notify(2.1d8(15),2.26d(9) epoch 11701) -9951> 2014-09-18 18:08:22.918690 7f4c9f251700 5 -- op tracker -- , seq: 85544, time: 2014-09-18 18:08:22.918688, event: waiting_for_osdmap, op: pg_notify(2.1d8(15),2.26d(9) epoch 11701) -9950> 2014-09-18 18:08:22.918702 7f4c9f251700 5 -- op tracker -- , seq: 85544, time: 2014-09-18 18:08:22.918700, event: started, op: pg_notify(2.1d8(15),2.26d(9) epoch 11701) -9949> 2014-09-18 18:08:22.918730 7f4c9f251700 5 -- op tracker -- , seq: 85544, time: 2014-09-18 18:08:22.918729, event: done, op: pg_notify(2.1d8(15),2.26d(9) epoch 11701) -9948> 2014-09-18 18:08:22.918732 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 crt=6865'30228 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.005008 7 0.000142 -9947> 2014-09-18 18:08:22.918776 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 crt=6865'30228 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -9946> 2014-09-18 18:08:22.918849 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 1995 ==== pg_notify(2.1d8(15) epoch 11701) v5 ==== 1603+0+0 (175927692 0 0) 0x21cea3c0 con 0xe2d9a20 -9945> 2014-09-18 18:08:22.918849 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000074 0 0.000000 -9944> 2014-09-18 18:08:22.918866 7f4c9f251700 5 -- op tracker -- , seq: 85545, time: 2014-09-18 18:08:22.918802, event: header_read, op: pg_notify(2.1d8(15) epoch 11701) -9943> 2014-09-18 18:08:22.918868 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -9942> 2014-09-18 18:08:22.918875 7f4c9f251700 5 -- op tracker -- , seq: 85545, time: 2014-09-18 18:08:22.918803, event: throttled, op: pg_notify(2.1d8(15) epoch 11701) -9941> 2014-09-18 18:08:22.918881 7f4c9f251700 5 -- op tracker -- , seq: 85545, time: 2014-09-18 18:08:22.918838, event: all_read, op: pg_notify(2.1d8(15) epoch 11701) -9940> 2014-09-18 18:08:22.918886 7f4c9f251700 5 -- op tracker -- , seq: 85545, time: 2014-09-18 18:08:22.918863, event: dispatched, op: pg_notify(2.1d8(15) epoch 11701) -9939> 2014-09-18 18:08:22.918893 7f4c9f251700 5 -- op tracker -- , seq: 85545, time: 2014-09-18 18:08:22.918892, event: waiting_for_osdmap, op: pg_notify(2.1d8(15) epoch 11701) -9938> 2014-09-18 18:08:22.918901 7f4c9f251700 5 -- op tracker -- , seq: 85545, time: 2014-09-18 18:08:22.918899, event: started, op: pg_notify(2.1d8(15) epoch 11701) -9937> 2014-09-18 18:08:22.918941 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000073 0 0.000000 -9936> 2014-09-18 18:08:22.918955 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -9935> 2014-09-18 18:08:22.919031 7f4c9f251700 5 -- op tracker -- , seq: 85545, time: 2014-09-18 18:08:22.919029, event: done, op: pg_notify(2.1d8(15) epoch 11701) -9934> 2014-09-18 18:08:22.919804 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 511 ==== pg_notify(2.4d0(10) epoch 11701) v5 ==== 1292+0+0 (1707998440 0 0) 0xf099c00 con 0x1395ef20 -9933> 2014-09-18 18:08:22.919818 7f4c9f251700 5 -- op tracker -- , seq: 85546, time: 2014-09-18 18:08:22.919761, event: header_read, op: pg_notify(2.4d0(10) epoch 11701) -9932> 2014-09-18 18:08:22.919826 7f4c9f251700 5 -- op tracker -- , seq: 85546, time: 2014-09-18 18:08:22.919762, event: throttled, op: pg_notify(2.4d0(10) epoch 11701) -9931> 2014-09-18 18:08:22.919832 7f4c9f251700 5 -- op tracker -- , seq: 85546, time: 2014-09-18 18:08:22.919792, event: all_read, op: pg_notify(2.4d0(10) epoch 11701) -9930> 2014-09-18 18:08:22.919836 7f4c9f251700 5 -- op tracker -- , seq: 85546, time: 2014-09-18 18:08:22.919815, event: dispatched, op: pg_notify(2.4d0(10) epoch 11701) -9929> 2014-09-18 18:08:22.919841 7f4c9f251700 5 -- op tracker -- , seq: 85546, time: 2014-09-18 18:08:22.919839, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11701) -9928> 2014-09-18 18:08:22.919846 7f4c9f251700 5 -- op tracker -- , seq: 85546, time: 2014-09-18 18:08:22.919845, event: started, op: pg_notify(2.4d0(10) epoch 11701) -9927> 2014-09-18 18:08:22.919861 7f4c9f251700 5 -- op tracker -- , seq: 85546, time: 2014-09-18 18:08:22.919860, event: done, op: pg_notify(2.4d0(10) epoch 11701) -9926> 2014-09-18 18:08:22.922131 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 512 ==== pg_notify(2.1d8(20) epoch 11701) v5 ==== 1878+0+0 (3574000922 0 0) 0x14221de0 con 0x1395ef20 -9925> 2014-09-18 18:08:22.922147 7f4c9f251700 5 -- op tracker -- , seq: 85547, time: 2014-09-18 18:08:22.922067, event: header_read, op: pg_notify(2.1d8(20) epoch 11701) -9924> 2014-09-18 18:08:22.922159 7f4c9f251700 5 -- op tracker -- , seq: 85547, time: 2014-09-18 18:08:22.922068, event: throttled, op: pg_notify(2.1d8(20) epoch 11701) -9923> 2014-09-18 18:08:22.922169 7f4c9f251700 5 -- op tracker -- , seq: 85547, time: 2014-09-18 18:08:22.922117, event: all_read, op: pg_notify(2.1d8(20) epoch 11701) -9922> 2014-09-18 18:08:22.922179 7f4c9f251700 5 -- op tracker -- , seq: 85547, time: 2014-09-18 18:08:22.922143, event: dispatched, op: pg_notify(2.1d8(20) epoch 11701) -9921> 2014-09-18 18:08:22.922187 7f4c9f251700 5 -- op tracker -- , seq: 85547, time: 2014-09-18 18:08:22.922185, event: waiting_for_osdmap, op: pg_notify(2.1d8(20) epoch 11701) -9920> 2014-09-18 18:08:22.922197 7f4c9f251700 5 -- op tracker -- , seq: 85547, time: 2014-09-18 18:08:22.922195, event: started, op: pg_notify(2.1d8(20) epoch 11701) -9919> 2014-09-18 18:08:22.922214 7f4c9f251700 5 -- op tracker -- , seq: 85547, time: 2014-09-18 18:08:22.922212, event: done, op: pg_notify(2.1d8(20) epoch 11701) -9918> 2014-09-18 18:08:22.925732 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 895 ==== pg_notify(2.410(11) epoch 11701) v5 ==== 1287+0+0 (781577358 0 0) 0x2b93c940 con 0x1395f8c0 -9917> 2014-09-18 18:08:22.925748 7f4c9f251700 5 -- op tracker -- , seq: 85548, time: 2014-09-18 18:08:22.925656, event: header_read, op: pg_notify(2.410(11) epoch 11701) -9916> 2014-09-18 18:08:22.925758 7f4c9f251700 5 -- op tracker -- , seq: 85548, time: 2014-09-18 18:08:22.925657, event: throttled, op: pg_notify(2.410(11) epoch 11701) -9915> 2014-09-18 18:08:22.925778 7f4c9f251700 5 -- op tracker -- , seq: 85548, time: 2014-09-18 18:08:22.925703, event: all_read, op: pg_notify(2.410(11) epoch 11701) -9914> 2014-09-18 18:08:22.925788 7f4c9f251700 5 -- op tracker -- , seq: 85548, time: 2014-09-18 18:08:22.925744, event: dispatched, op: pg_notify(2.410(11) epoch 11701) -9913> 2014-09-18 18:08:22.925795 7f4c9f251700 5 -- op tracker -- , seq: 85548, time: 2014-09-18 18:08:22.925793, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11701) -9912> 2014-09-18 18:08:22.925802 7f4c9f251700 5 -- op tracker -- , seq: 85548, time: 2014-09-18 18:08:22.925801, event: started, op: pg_notify(2.410(11) epoch 11701) -9911> 2014-09-18 18:08:22.925820 7f4c9f251700 5 -- op tracker -- , seq: 85548, time: 2014-09-18 18:08:22.925818, event: done, op: pg_notify(2.410(11) epoch 11701) -9910> 2014-09-18 18:08:22.928128 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 896 ==== pg_notify(2.4c1(15) epoch 11701) v5 ==== 1507+0+0 (1692337956 0 0) 0x11bbe5a0 con 0x1395f8c0 -9909> 2014-09-18 18:08:22.928143 7f4c9f251700 5 -- op tracker -- , seq: 85549, time: 2014-09-18 18:08:22.928081, event: header_read, op: pg_notify(2.4c1(15) epoch 11701) -9908> 2014-09-18 18:08:22.928152 7f4c9f251700 5 -- op tracker -- , seq: 85549, time: 2014-09-18 18:08:22.928082, event: throttled, op: pg_notify(2.4c1(15) epoch 11701) -9907> 2014-09-18 18:08:22.928158 7f4c9f251700 5 -- op tracker -- , seq: 85549, time: 2014-09-18 18:08:22.928116, event: all_read, op: pg_notify(2.4c1(15) epoch 11701) -9906> 2014-09-18 18:08:22.928164 7f4c9f251700 5 -- op tracker -- , seq: 85549, time: 2014-09-18 18:08:22.928140, event: dispatched, op: pg_notify(2.4c1(15) epoch 11701) -9905> 2014-09-18 18:08:22.928170 7f4c9f251700 5 -- op tracker -- , seq: 85549, time: 2014-09-18 18:08:22.928169, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11701) -9904> 2014-09-18 18:08:22.928178 7f4c9f251700 5 -- op tracker -- , seq: 85549, time: 2014-09-18 18:08:22.928176, event: started, op: pg_notify(2.4c1(15) epoch 11701) -9903> 2014-09-18 18:08:22.928194 7f4c9f251700 5 -- op tracker -- , seq: 85549, time: 2014-09-18 18:08:22.928193, event: done, op: pg_notify(2.4c1(15) epoch 11701) -9902> 2014-09-18 18:08:22.928211 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.014114 3 0.000110 -9901> 2014-09-18 18:08:22.928231 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -9900> 2014-09-18 18:08:22.928265 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000034 0 0.000000 -9899> 2014-09-18 18:08:22.928278 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -9898> 2014-09-18 18:08:22.928289 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000011 0 0.000000 -9897> 2014-09-18 18:08:22.928300 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -9896> 2014-09-18 18:08:22.930631 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 897 ==== pg_query(2.34f epoch 11701) v3 ==== 144+0+0 (2522007422 0 0) 0xf098b20 con 0x1395f8c0 -9895> 2014-09-18 18:08:22.930666 7f4c9f251700 5 -- op tracker -- , seq: 85550, time: 2014-09-18 18:08:22.930525, event: header_read, op: pg_query(2.34f epoch 11701) -9894> 2014-09-18 18:08:22.930678 7f4c9f251700 5 -- op tracker -- , seq: 85550, time: 2014-09-18 18:08:22.930527, event: throttled, op: pg_query(2.34f epoch 11701) -9893> 2014-09-18 18:08:22.930685 7f4c9f251700 5 -- op tracker -- , seq: 85550, time: 2014-09-18 18:08:22.930562, event: all_read, op: pg_query(2.34f epoch 11701) -9892> 2014-09-18 18:08:22.930692 7f4c9f251700 5 -- op tracker -- , seq: 85550, time: 2014-09-18 18:08:22.930661, event: dispatched, op: pg_query(2.34f epoch 11701) -9891> 2014-09-18 18:08:22.930698 7f4c9f251700 5 -- op tracker -- , seq: 85550, time: 2014-09-18 18:08:22.930697, event: waiting_for_osdmap, op: pg_query(2.34f epoch 11701) -9890> 2014-09-18 18:08:22.930707 7f4c9f251700 5 -- op tracker -- , seq: 85550, time: 2014-09-18 18:08:22.930706, event: started, op: pg_query(2.34f epoch 11701) -9889> 2014-09-18 18:08:22.930724 7f4c9f251700 5 -- op tracker -- , seq: 85550, time: 2014-09-18 18:08:22.930723, event: done, op: pg_query(2.34f epoch 11701) -9888> 2014-09-18 18:08:22.930919 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_notify(2.34f(73) epoch 11701) v5 -- ?+0 0xf0acb20 con 0x1395f8c0 -9887> 2014-09-18 18:08:22.935010 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3124 ==== osd_map(11701..11701 src has 1..11701) v3 ==== 2801+0+0 (2938961775 0 0) 0x15e30d80 con 0x1395ec60 -9886> 2014-09-18 18:08:22.935030 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11701,11701], i have 11701, src has [1,11701] -9885> 2014-09-18 18:08:22.935041 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3125 ==== pg_notify(2.71(19) epoch 11701) v5 ==== 1795+0+0 (2768240631 0 0) 0xf4f1840 con 0x1395ec60 -9884> 2014-09-18 18:08:22.935052 7f4c9f251700 5 -- op tracker -- , seq: 85551, time: 2014-09-18 18:08:22.934981, event: header_read, op: pg_notify(2.71(19) epoch 11701) -9883> 2014-09-18 18:08:22.935063 7f4c9f251700 5 -- op tracker -- , seq: 85551, time: 2014-09-18 18:08:22.934982, event: throttled, op: pg_notify(2.71(19) epoch 11701) -9882> 2014-09-18 18:08:22.935067 7f4c9f251700 5 -- op tracker -- , seq: 85551, time: 2014-09-18 18:08:22.935013, event: all_read, op: pg_notify(2.71(19) epoch 11701) -9881> 2014-09-18 18:08:22.935073 7f4c9f251700 5 -- op tracker -- , seq: 85551, time: 2014-09-18 18:08:22.935049, event: dispatched, op: pg_notify(2.71(19) epoch 11701) -9880> 2014-09-18 18:08:22.935079 7f4c9f251700 5 -- op tracker -- , seq: 85551, time: 2014-09-18 18:08:22.935077, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11701) -9879> 2014-09-18 18:08:22.935087 7f4c9f251700 5 -- op tracker -- , seq: 85551, time: 2014-09-18 18:08:22.935086, event: started, op: pg_notify(2.71(19) epoch 11701) -9878> 2014-09-18 18:08:22.935108 7f4c9f251700 5 -- op tracker -- , seq: 85551, time: 2014-09-18 18:08:22.935106, event: done, op: pg_notify(2.71(19) epoch 11701) -9877> 2014-09-18 18:08:22.941932 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 898 ==== pg_notify(2.1d8(9),2.4c1(15),2.78f(152) epoch 11701) v5 ==== 11918+0+0 (3138170058 0 0) 0xf0acb20 con 0x1395f8c0 -9876> 2014-09-18 18:08:22.942055 7f4c9f251700 5 -- op tracker -- , seq: 85552, time: 2014-09-18 18:08:22.941788, event: header_read, op: pg_notify(2.1d8(9),2.4c1(15),2.78f(152) epoch 11701) -9875> 2014-09-18 18:08:22.942065 7f4c9f251700 5 -- op tracker -- , seq: 85552, time: 2014-09-18 18:08:22.941789, event: throttled, op: pg_notify(2.1d8(9),2.4c1(15),2.78f(152) epoch 11701) -9874> 2014-09-18 18:08:22.942072 7f4c9f251700 5 -- op tracker -- , seq: 85552, time: 2014-09-18 18:08:22.941918, event: all_read, op: pg_notify(2.1d8(9),2.4c1(15),2.78f(152) epoch 11701) -9873> 2014-09-18 18:08:22.942078 7f4c9f251700 5 -- op tracker -- , seq: 85552, time: 2014-09-18 18:08:22.942049, event: dispatched, op: pg_notify(2.1d8(9),2.4c1(15),2.78f(152) epoch 11701) -9872> 2014-09-18 18:08:22.942085 7f4c9f251700 5 -- op tracker -- , seq: 85552, time: 2014-09-18 18:08:22.942084, event: waiting_for_osdmap, op: pg_notify(2.1d8(9),2.4c1(15),2.78f(152) epoch 11701) -9871> 2014-09-18 18:08:22.942094 7f4c9f251700 5 -- op tracker -- , seq: 85552, time: 2014-09-18 18:08:22.942092, event: started, op: pg_notify(2.1d8(9),2.4c1(15),2.78f(152) epoch 11701) -9870> 2014-09-18 18:08:22.942125 7f4c9f251700 5 -- op tracker -- , seq: 85552, time: 2014-09-18 18:08:22.942123, event: done, op: pg_notify(2.1d8(9),2.4c1(15),2.78f(152) epoch 11701) -9869> 2014-09-18 18:08:22.942372 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 899 ==== pg_notify(2.410(11) epoch 11701) v5 ==== 1287+0+0 (781577358 0 0) 0xf0db2c0 con 0x1395f8c0 -9868> 2014-09-18 18:08:22.942391 7f4c9f251700 5 -- op tracker -- , seq: 85553, time: 2014-09-18 18:08:22.942318, event: header_read, op: pg_notify(2.410(11) epoch 11701) -9867> 2014-09-18 18:08:22.942404 7f4c9f251700 5 -- op tracker -- , seq: 85553, time: 2014-09-18 18:08:22.942319, event: throttled, op: pg_notify(2.410(11) epoch 11701) -9866> 2014-09-18 18:08:22.942413 7f4c9f251700 5 -- op tracker -- , seq: 85553, time: 2014-09-18 18:08:22.942359, event: all_read, op: pg_notify(2.410(11) epoch 11701) -9865> 2014-09-18 18:08:22.942424 7f4c9f251700 5 -- op tracker -- , seq: 85553, time: 2014-09-18 18:08:22.942386, event: dispatched, op: pg_notify(2.410(11) epoch 11701) -9864> 2014-09-18 18:08:22.942435 7f4c9f251700 5 -- op tracker -- , seq: 85553, time: 2014-09-18 18:08:22.942433, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11701) -9863> 2014-09-18 18:08:22.942447 7f4c9f251700 5 -- op tracker -- , seq: 85553, time: 2014-09-18 18:08:22.942445, event: started, op: pg_notify(2.410(11) epoch 11701) -9862> 2014-09-18 18:08:22.942463 7f4c9f251700 5 -- op tracker -- , seq: 85553, time: 2014-09-18 18:08:22.942462, event: done, op: pg_notify(2.410(11) epoch 11701) -9861> 2014-09-18 18:08:22.948191 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3126 ==== pg_notify(2.410(11) epoch 11701) v5 ==== 1325+0+0 (2900761850 0 0) 0xf12a3c0 con 0x1395ec60 -9860> 2014-09-18 18:08:22.948211 7f4c9f251700 5 -- op tracker -- , seq: 85554, time: 2014-09-18 18:08:22.948104, event: header_read, op: pg_notify(2.410(11) epoch 11701) -9859> 2014-09-18 18:08:22.948221 7f4c9f251700 5 -- op tracker -- , seq: 85554, time: 2014-09-18 18:08:22.948105, event: throttled, op: pg_notify(2.410(11) epoch 11701) -9858> 2014-09-18 18:08:22.948228 7f4c9f251700 5 -- op tracker -- , seq: 85554, time: 2014-09-18 18:08:22.948141, event: all_read, op: pg_notify(2.410(11) epoch 11701) -9857> 2014-09-18 18:08:22.948232 7f4c9f251700 5 -- op tracker -- , seq: 85554, time: 2014-09-18 18:08:22.948206, event: dispatched, op: pg_notify(2.410(11) epoch 11701) -9856> 2014-09-18 18:08:22.948239 7f4c9f251700 5 -- op tracker -- , seq: 85554, time: 2014-09-18 18:08:22.948237, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11701) -9855> 2014-09-18 18:08:22.948247 7f4c9f251700 5 -- op tracker -- , seq: 85554, time: 2014-09-18 18:08:22.948245, event: started, op: pg_notify(2.410(11) epoch 11701) -9854> 2014-09-18 18:08:22.948265 7f4c9f251700 5 -- op tracker -- , seq: 85554, time: 2014-09-18 18:08:22.948263, event: done, op: pg_notify(2.410(11) epoch 11701) -9853> 2014-09-18 18:08:22.951024 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3127 ==== pg_notify(2.74e(11) epoch 11701) v5 ==== 1303+0+0 (3359078290 0 0) 0x24fa2580 con 0x1395ec60 -9852> 2014-09-18 18:08:22.951044 7f4c9f251700 5 -- op tracker -- , seq: 85555, time: 2014-09-18 18:08:22.950937, event: header_read, op: pg_notify(2.74e(11) epoch 11701) -9851> 2014-09-18 18:08:22.951053 7f4c9f251700 5 -- op tracker -- , seq: 85555, time: 2014-09-18 18:08:22.950938, event: throttled, op: pg_notify(2.74e(11) epoch 11701) -9850> 2014-09-18 18:08:22.951057 7f4c9f251700 5 -- op tracker -- , seq: 85555, time: 2014-09-18 18:08:22.950985, event: all_read, op: pg_notify(2.74e(11) epoch 11701) -9849> 2014-09-18 18:08:22.951061 7f4c9f251700 5 -- op tracker -- , seq: 85555, time: 2014-09-18 18:08:22.951040, event: dispatched, op: pg_notify(2.74e(11) epoch 11701) -9848> 2014-09-18 18:08:22.951065 7f4c9f251700 5 -- op tracker -- , seq: 85555, time: 2014-09-18 18:08:22.951064, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11701) -9847> 2014-09-18 18:08:22.951071 7f4c9f251700 5 -- op tracker -- , seq: 85555, time: 2014-09-18 18:08:22.951070, event: started, op: pg_notify(2.74e(11) epoch 11701) -9846> 2014-09-18 18:08:22.951098 7f4c9f251700 5 -- op tracker -- , seq: 85555, time: 2014-09-18 18:08:22.951096, event: done, op: pg_notify(2.74e(11) epoch 11701) -9845> 2014-09-18 18:08:22.951111 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3128 ==== pg_notify(2.186(12) epoch 11701) v5 ==== 1386+0+0 (3035618684 0 0) 0x24fa1fe0 con 0x1395ec60 -9844> 2014-09-18 18:08:22.951120 7f4c9f251700 5 -- op tracker -- , seq: 85556, time: 2014-09-18 18:08:22.951009, event: header_read, op: pg_notify(2.186(12) epoch 11701) -9843> 2014-09-18 18:08:22.951127 7f4c9f251700 5 -- op tracker -- , seq: 85556, time: 2014-09-18 18:08:22.951010, event: throttled, op: pg_notify(2.186(12) epoch 11701) -9842> 2014-09-18 18:08:22.951133 7f4c9f251700 5 -- op tracker -- , seq: 85556, time: 2014-09-18 18:08:22.951035, event: all_read, op: pg_notify(2.186(12) epoch 11701) -9841> 2014-09-18 18:08:22.951139 7f4c9f251700 5 -- op tracker -- , seq: 85556, time: 2014-09-18 18:08:22.951118, event: dispatched, op: pg_notify(2.186(12) epoch 11701) -9840> 2014-09-18 18:08:22.951144 7f4c9f251700 5 -- op tracker -- , seq: 85556, time: 2014-09-18 18:08:22.951142, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11701) -9839> 2014-09-18 18:08:22.951151 7f4c9f251700 5 -- op tracker -- , seq: 85556, time: 2014-09-18 18:08:22.951149, event: started, op: pg_notify(2.186(12) epoch 11701) -9838> 2014-09-18 18:08:22.951166 7f4c9f251700 5 -- op tracker -- , seq: 85556, time: 2014-09-18 18:08:22.951165, event: done, op: pg_notify(2.186(12) epoch 11701) -9837> 2014-09-18 18:08:22.953487 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3129 ==== pg_notify(2.4b1(13) epoch 11701) v5 ==== 1441+0+0 (658576922 0 0) 0xf96bc20 con 0x1395ec60 -9836> 2014-09-18 18:08:22.953506 7f4c9f251700 5 -- op tracker -- , seq: 85557, time: 2014-09-18 18:08:22.953419, event: header_read, op: pg_notify(2.4b1(13) epoch 11701) -9835> 2014-09-18 18:08:22.953516 7f4c9f251700 5 -- op tracker -- , seq: 85557, time: 2014-09-18 18:08:22.953420, event: throttled, op: pg_notify(2.4b1(13) epoch 11701) -9834> 2014-09-18 18:08:22.953523 7f4c9f251700 5 -- op tracker -- , seq: 85557, time: 2014-09-18 18:08:22.953453, event: all_read, op: pg_notify(2.4b1(13) epoch 11701) -9833> 2014-09-18 18:08:22.953529 7f4c9f251700 5 -- op tracker -- , seq: 85557, time: 2014-09-18 18:08:22.953502, event: dispatched, op: pg_notify(2.4b1(13) epoch 11701) -9832> 2014-09-18 18:08:22.953535 7f4c9f251700 5 -- op tracker -- , seq: 85557, time: 2014-09-18 18:08:22.953533, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11701) -9831> 2014-09-18 18:08:22.953543 7f4c9f251700 5 -- op tracker -- , seq: 85557, time: 2014-09-18 18:08:22.953541, event: started, op: pg_notify(2.4b1(13) epoch 11701) -9830> 2014-09-18 18:08:22.953562 7f4c9f251700 5 -- op tracker -- , seq: 85557, time: 2014-09-18 18:08:22.953561, event: done, op: pg_notify(2.4b1(13) epoch 11701) -9829> 2014-09-18 18:08:22.954427 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3130 ==== pg_notify(2.799(9),2.4d0(10) epoch 11701) v5 ==== 2509+0+0 (4155250141 0 0) 0xf96a3c0 con 0x1395ec60 -9828> 2014-09-18 18:08:22.954442 7f4c9f251700 5 -- op tracker -- , seq: 85558, time: 2014-09-18 18:08:22.954354, event: header_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11701) -9827> 2014-09-18 18:08:22.954450 7f4c9f251700 5 -- op tracker -- , seq: 85558, time: 2014-09-18 18:08:22.954355, event: throttled, op: pg_notify(2.799(9),2.4d0(10) epoch 11701) -9826> 2014-09-18 18:08:22.954456 7f4c9f251700 5 -- op tracker -- , seq: 85558, time: 2014-09-18 18:08:22.954392, event: all_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11701) -9825> 2014-09-18 18:08:22.954462 7f4c9f251700 5 -- op tracker -- , seq: 85558, time: 2014-09-18 18:08:22.954438, event: dispatched, op: pg_notify(2.799(9),2.4d0(10) epoch 11701) -9824> 2014-09-18 18:08:22.954469 7f4c9f251700 5 -- op tracker -- , seq: 85558, time: 2014-09-18 18:08:22.954467, event: waiting_for_osdmap, op: pg_notify(2.799(9),2.4d0(10) epoch 11701) -9823> 2014-09-18 18:08:22.954476 7f4c9f251700 5 -- op tracker -- , seq: 85558, time: 2014-09-18 18:08:22.954475, event: started, op: pg_notify(2.799(9),2.4d0(10) epoch 11701) -9822> 2014-09-18 18:08:22.954500 7f4c9f251700 5 -- op tracker -- , seq: 85558, time: 2014-09-18 18:08:22.954499, event: done, op: pg_notify(2.799(9),2.4d0(10) epoch 11701) -9821> 2014-09-18 18:08:22.955141 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3131 ==== pg_notify(2.4c1(15),2.78f(50),2.4c6(15) epoch 11701) v5 ==== 6390+0+0 (3744610653 0 0) 0xf96b860 con 0x1395ec60 -9820> 2014-09-18 18:08:22.955168 7f4c9f251700 5 -- op tracker -- , seq: 85559, time: 2014-09-18 18:08:22.955041, event: header_read, op: pg_notify(2.4c1(15),2.78f(50),2.4c6(15) epoch 11701) -9819> 2014-09-18 18:08:22.955177 7f4c9f251700 5 -- op tracker -- , seq: 85559, time: 2014-09-18 18:08:22.955042, event: throttled, op: pg_notify(2.4c1(15),2.78f(50),2.4c6(15) epoch 11701) -9818> 2014-09-18 18:08:22.955184 7f4c9f251700 5 -- op tracker -- , seq: 85559, time: 2014-09-18 18:08:22.955127, event: all_read, op: pg_notify(2.4c1(15),2.78f(50),2.4c6(15) epoch 11701) -9817> 2014-09-18 18:08:22.955190 7f4c9f251700 5 -- op tracker -- , seq: 85559, time: 2014-09-18 18:08:22.955163, event: dispatched, op: pg_notify(2.4c1(15),2.78f(50),2.4c6(15) epoch 11701) -9816> 2014-09-18 18:08:22.955197 7f4c9f251700 5 -- op tracker -- , seq: 85559, time: 2014-09-18 18:08:22.955195, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(50),2.4c6(15) epoch 11701) -9815> 2014-09-18 18:08:22.955205 7f4c9f251700 5 -- op tracker -- , seq: 85559, time: 2014-09-18 18:08:22.955203, event: started, op: pg_notify(2.4c1(15),2.78f(50),2.4c6(15) epoch 11701) -9814> 2014-09-18 18:08:22.955316 7f4c9f251700 5 -- op tracker -- , seq: 85559, time: 2014-09-18 18:08:22.955314, event: done, op: pg_notify(2.4c1(15),2.78f(50),2.4c6(15) epoch 11701) -9813> 2014-09-18 18:08:23.088655 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 944 ==== osd_map(11701..11701 src has 1..11701) v3 ==== 2801+0+0 (2938961775 0 0) 0x1182d840 con 0x10a878c0 -9812> 2014-09-18 18:08:23.088673 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11701,11701], i have 11701, src has [1,11701] -9811> 2014-09-18 18:08:23.088708 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 945 ==== pg_notify(2.517(11) epoch 11701) v5 ==== 1335+0+0 (2004193652 0 0) 0x16250940 con 0x10a878c0 -9810> 2014-09-18 18:08:23.088719 7f4c9f251700 5 -- op tracker -- , seq: 85560, time: 2014-09-18 18:08:23.088654, event: header_read, op: pg_notify(2.517(11) epoch 11701) -9809> 2014-09-18 18:08:23.088728 7f4c9f251700 5 -- op tracker -- , seq: 85560, time: 2014-09-18 18:08:23.088654, event: throttled, op: pg_notify(2.517(11) epoch 11701) -9808> 2014-09-18 18:08:23.088735 7f4c9f251700 5 -- op tracker -- , seq: 85560, time: 2014-09-18 18:08:23.088698, event: all_read, op: pg_notify(2.517(11) epoch 11701) -9807> 2014-09-18 18:08:23.088739 7f4c9f251700 5 -- op tracker -- , seq: 85560, time: 2014-09-18 18:08:23.088716, event: dispatched, op: pg_notify(2.517(11) epoch 11701) -9806> 2014-09-18 18:08:23.088744 7f4c9f251700 5 -- op tracker -- , seq: 85560, time: 2014-09-18 18:08:23.088743, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11701) -9805> 2014-09-18 18:08:23.088752 7f4c9f251700 5 -- op tracker -- , seq: 85560, time: 2014-09-18 18:08:23.088751, event: started, op: pg_notify(2.517(11) epoch 11701) -9804> 2014-09-18 18:08:23.088770 7f4c9f251700 5 -- op tracker -- , seq: 85560, time: 2014-09-18 18:08:23.088769, event: done, op: pg_notify(2.517(11) epoch 11701) -9803> 2014-09-18 18:08:23.089701 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 946 ==== pg_notify(2.4c6(15) epoch 11701) v5 ==== 1511+0+0 (477311383 0 0) 0x1624e5a0 con 0x10a878c0 -9802> 2014-09-18 18:08:23.089714 7f4c9f251700 5 -- op tracker -- , seq: 85561, time: 2014-09-18 18:08:23.089643, event: header_read, op: pg_notify(2.4c6(15) epoch 11701) -9801> 2014-09-18 18:08:23.089719 7f4c9f251700 5 -- op tracker -- , seq: 85561, time: 2014-09-18 18:08:23.089644, event: throttled, op: pg_notify(2.4c6(15) epoch 11701) -9800> 2014-09-18 18:08:23.089724 7f4c9f251700 5 -- op tracker -- , seq: 85561, time: 2014-09-18 18:08:23.089689, event: all_read, op: pg_notify(2.4c6(15) epoch 11701) -9799> 2014-09-18 18:08:23.089727 7f4c9f251700 5 -- op tracker -- , seq: 85561, time: 2014-09-18 18:08:23.089711, event: dispatched, op: pg_notify(2.4c6(15) epoch 11701) -9798> 2014-09-18 18:08:23.089732 7f4c9f251700 5 -- op tracker -- , seq: 85561, time: 2014-09-18 18:08:23.089731, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11701) -9797> 2014-09-18 18:08:23.089739 7f4c9f251700 5 -- op tracker -- , seq: 85561, time: 2014-09-18 18:08:23.089738, event: started, op: pg_notify(2.4c6(15) epoch 11701) -9796> 2014-09-18 18:08:23.089756 7f4c9f251700 5 -- op tracker -- , seq: 85561, time: 2014-09-18 18:08:23.089754, event: done, op: pg_notify(2.4c6(15) epoch 11701) -9795> 2014-09-18 18:08:23.244855 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.39 10.242.42.172:0/2660 1586 ==== osd_ping(ping e11701 stamp 2014-09-18 23:37:12.349352) v2 ==== 47+0+0 (4034400396 0 0) 0x21e121c0 con 0x4750840 -9794> 2014-09-18 18:08:23.244892 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.39 10.242.42.172:0/2660 1586 ==== osd_ping(ping e11701 stamp 2014-09-18 23:37:12.349352) v2 ==== 47+0+0 (4034400396 0 0) 0x11b0e5a0 con 0xec39340 -9793> 2014-09-18 18:08:23.244919 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11701 stamp 2014-09-18 23:37:12.349352) v2 -- ?+0 0xf4f03a0 con 0x4750840 -9792> 2014-09-18 18:08:23.244956 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11701 stamp 2014-09-18 23:37:12.349352) v2 -- ?+0 0xf2b3a40 con 0xec39340 -9791> 2014-09-18 18:08:23.278064 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 666 ==== osd_map(11701..11701 src has 1..11701) v3 ==== 2801+0+0 (2938961775 0 0) 0x24d1a000 con 0x1f103e40 -9790> 2014-09-18 18:08:23.278093 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11701,11701], i have 11701, src has [1,11701] -9789> 2014-09-18 18:08:23.278142 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 667 ==== pg_notify(2.4f6(15) epoch 11701) v5 ==== 1543+0+0 (1883016810 0 0) 0x10413fe0 con 0x1f103e40 -9788> 2014-09-18 18:08:23.278162 7f4c9f251700 5 -- op tracker -- , seq: 85562, time: 2014-09-18 18:08:23.278081, event: header_read, op: pg_notify(2.4f6(15) epoch 11701) -9787> 2014-09-18 18:08:23.278172 7f4c9f251700 5 -- op tracker -- , seq: 85562, time: 2014-09-18 18:08:23.278081, event: throttled, op: pg_notify(2.4f6(15) epoch 11701) -9786> 2014-09-18 18:08:23.278180 7f4c9f251700 5 -- op tracker -- , seq: 85562, time: 2014-09-18 18:08:23.278130, event: all_read, op: pg_notify(2.4f6(15) epoch 11701) -9785> 2014-09-18 18:08:23.278188 7f4c9f251700 5 -- op tracker -- , seq: 85562, time: 2014-09-18 18:08:23.278157, event: dispatched, op: pg_notify(2.4f6(15) epoch 11701) -9784> 2014-09-18 18:08:23.278194 7f4c9f251700 5 -- op tracker -- , seq: 85562, time: 2014-09-18 18:08:23.278193, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11701) -9783> 2014-09-18 18:08:23.278203 7f4c9f251700 5 -- op tracker -- , seq: 85562, time: 2014-09-18 18:08:23.278202, event: started, op: pg_notify(2.4f6(15) epoch 11701) -9782> 2014-09-18 18:08:23.278226 7f4c9f251700 5 -- op tracker -- , seq: 85562, time: 2014-09-18 18:08:23.278225, event: done, op: pg_notify(2.4f6(15) epoch 11701) -9781> 2014-09-18 18:08:23.280262 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.56 10.242.42.172:0/36458 651 ==== osd_ping(ping e11701 stamp 2014-09-18 23:37:12.384993) v2 ==== 47+0+0 (1566461346 0 0) 0xf0e41c0 con 0x14392c60 -9780> 2014-09-18 18:08:23.280285 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/36458 -- osd_ping(ping_reply e11701 stamp 2014-09-18 23:37:12.384993) v2 -- ?+0 0x21e121c0 con 0x14392c60 -9779> 2014-09-18 18:08:23.280295 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.56 10.242.42.172:0/36458 651 ==== osd_ping(ping e11701 stamp 2014-09-18 23:37:12.384993) v2 ==== 47+0+0 (1566461346 0 0) 0x21d143a0 con 0x10a81700 -9778> 2014-09-18 18:08:23.280320 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/36458 -- osd_ping(ping_reply e11701 stamp 2014-09-18 23:37:12.384993) v2 -- ?+0 0x11b0e5a0 con 0x10a81700 -9777> 2014-09-18 18:08:23.280406 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 668 ==== pg_notify(2.799(10),2.4c6(15) epoch 11701) v5 ==== 2779+0+0 (2316874257 0 0) 0x104130e0 con 0x1f103e40 -9776> 2014-09-18 18:08:23.280431 7f4c9f251700 5 -- op tracker -- , seq: 85563, time: 2014-09-18 18:08:23.280332, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11701) -9775> 2014-09-18 18:08:23.280446 7f4c9f251700 5 -- op tracker -- , seq: 85563, time: 2014-09-18 18:08:23.280333, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11701) -9774> 2014-09-18 18:08:23.280457 7f4c9f251700 5 -- op tracker -- , seq: 85563, time: 2014-09-18 18:08:23.280393, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11701) -9773> 2014-09-18 18:08:23.280466 7f4c9f251700 5 -- op tracker -- , seq: 85563, time: 2014-09-18 18:08:23.280425, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11701) -9772> 2014-09-18 18:08:23.280477 7f4c9f251700 5 -- op tracker -- , seq: 85563, time: 2014-09-18 18:08:23.280474, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11701) -9771> 2014-09-18 18:08:23.280489 7f4c9f251700 5 -- op tracker -- , seq: 85563, time: 2014-09-18 18:08:23.280487, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11701) -9770> 2014-09-18 18:08:23.280519 7f4c9f251700 5 -- op tracker -- , seq: 85563, time: 2014-09-18 18:08:23.280517, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11701) -9769> 2014-09-18 18:08:23.300342 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 327 ==== osd_map(11701..11701 src has 1..11701) v3 ==== 2801+0+0 (2938961775 0 0) 0xe2ed200 con 0x12562260 -9768> 2014-09-18 18:08:23.300366 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11701,11701], i have 11701, src has [1,11701] -9767> 2014-09-18 18:08:23.300412 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 328 ==== pg_notify(2.719(17) epoch 11701) v5 ==== 1601+0+0 (2224572869 0 0) 0xf2b1a20 con 0x12562260 -9766> 2014-09-18 18:08:23.300428 7f4c9f251700 5 -- op tracker -- , seq: 85564, time: 2014-09-18 18:08:23.300357, event: header_read, op: pg_notify(2.719(17) epoch 11701) -9765> 2014-09-18 18:08:23.300438 7f4c9f251700 5 -- op tracker -- , seq: 85564, time: 2014-09-18 18:08:23.300358, event: throttled, op: pg_notify(2.719(17) epoch 11701) -9764> 2014-09-18 18:08:23.300443 7f4c9f251700 5 -- op tracker -- , seq: 85564, time: 2014-09-18 18:08:23.300400, event: all_read, op: pg_notify(2.719(17) epoch 11701) -9763> 2014-09-18 18:08:23.300449 7f4c9f251700 5 -- op tracker -- , seq: 85564, time: 2014-09-18 18:08:23.300424, event: dispatched, op: pg_notify(2.719(17) epoch 11701) -9762> 2014-09-18 18:08:23.300455 7f4c9f251700 5 -- op tracker -- , seq: 85564, time: 2014-09-18 18:08:23.300453, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11701) -9761> 2014-09-18 18:08:23.300461 7f4c9f251700 5 -- op tracker -- , seq: 85564, time: 2014-09-18 18:08:23.300459, event: started, op: pg_notify(2.719(17) epoch 11701) -9760> 2014-09-18 18:08:23.300486 7f4c9f251700 5 -- op tracker -- , seq: 85564, time: 2014-09-18 18:08:23.300484, event: done, op: pg_notify(2.719(17) epoch 11701) -9759> 2014-09-18 18:08:23.487985 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.10 10.242.42.168:0/10695 687 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:23.354959) v2 ==== 47+0+0 (880569501 0 0) 0xfccb660 con 0x140f9b20 -9758> 2014-09-18 18:08:23.487978 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.10 10.242.42.168:0/10695 687 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:23.354959) v2 ==== 47+0+0 (880569501 0 0) 0xf2b9840 con 0x109f9b20 -9757> 2014-09-18 18:08:23.488037 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10695 -- osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.354959) v2 -- ?+0 0xf0e41c0 con 0x140f9b20 -9756> 2014-09-18 18:08:23.488073 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10695 -- osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.354959) v2 -- ?+0 0x21d143a0 con 0x109f9b20 -9755> 2014-09-18 18:08:23.502453 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.23 10.242.42.172:0/11463 1546 ==== osd_ping(ping e11701 stamp 2014-09-18 23:37:12.606994) v2 ==== 47+0+0 (2761524360 0 0) 0xfccb2a0 con 0xe567080 -9754> 2014-09-18 18:08:23.502477 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/11463 -- osd_ping(ping_reply e11701 stamp 2014-09-18 23:37:12.606994) v2 -- ?+0 0xf2b9840 con 0xe567080 -9753> 2014-09-18 18:08:23.503135 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.23 10.242.42.172:0/11463 1546 ==== osd_ping(ping e11701 stamp 2014-09-18 23:37:12.606994) v2 ==== 47+0+0 (2761524360 0 0) 0xe1a23c0 con 0xec5c160 -9752> 2014-09-18 18:08:23.503155 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/11463 -- osd_ping(ping_reply e11701 stamp 2014-09-18 23:37:12.606994) v2 -- ?+0 0xfccb660 con 0xec5c160 -9751> 2014-09-18 18:08:23.519262 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3571 ==== pg_notify(2.40a(162) epoch 11701) v5 ==== 9736+0+0 (2728674815 0 0) 0x12c11860 con 0xcbecec0 -9750> 2014-09-18 18:08:23.519284 7f4c9f251700 5 -- op tracker -- , seq: 85565, time: 2014-09-18 18:08:23.519030, event: header_read, op: pg_notify(2.40a(162) epoch 11701) -9749> 2014-09-18 18:08:23.519301 7f4c9f251700 5 -- op tracker -- , seq: 85565, time: 2014-09-18 18:08:23.519031, event: throttled, op: pg_notify(2.40a(162) epoch 11701) -9748> 2014-09-18 18:08:23.519318 7f4c9f251700 5 -- op tracker -- , seq: 85565, time: 2014-09-18 18:08:23.519245, event: all_read, op: pg_notify(2.40a(162) epoch 11701) -9747> 2014-09-18 18:08:23.519329 7f4c9f251700 5 -- op tracker -- , seq: 85565, time: 2014-09-18 18:08:23.519277, event: dispatched, op: pg_notify(2.40a(162) epoch 11701) -9746> 2014-09-18 18:08:23.519341 7f4c9f251700 5 -- op tracker -- , seq: 85565, time: 2014-09-18 18:08:23.519338, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11701) -9745> 2014-09-18 18:08:23.519354 7f4c9f251700 5 -- op tracker -- , seq: 85565, time: 2014-09-18 18:08:23.519351, event: started, op: pg_notify(2.40a(162) epoch 11701) -9744> 2014-09-18 18:08:23.519392 7f4c9f251700 5 -- op tracker -- , seq: 85565, time: 2014-09-18 18:08:23.519390, event: done, op: pg_notify(2.40a(162) epoch 11701) -9743> 2014-09-18 18:08:23.521397 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3572 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11701) v5 ==== 6174+0+0 (3295282659 0 0) 0x12c10f00 con 0xcbecec0 -9742> 2014-09-18 18:08:23.521416 7f4c9f251700 5 -- op tracker -- , seq: 85566, time: 2014-09-18 18:08:23.521269, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11701) -9741> 2014-09-18 18:08:23.521467 7f4c9f251700 5 -- op tracker -- , seq: 85566, time: 2014-09-18 18:08:23.521271, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11701) -9740> 2014-09-18 18:08:23.521477 7f4c9f251700 5 -- op tracker -- , seq: 85566, time: 2014-09-18 18:08:23.521381, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11701) -9739> 2014-09-18 18:08:23.521486 7f4c9f251700 5 -- op tracker -- , seq: 85566, time: 2014-09-18 18:08:23.521411, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11701) -9738> 2014-09-18 18:08:23.521493 7f4c9f251700 5 -- op tracker -- , seq: 85566, time: 2014-09-18 18:08:23.521491, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11701) -9737> 2014-09-18 18:08:23.521503 7f4c9f251700 5 -- op tracker -- , seq: 85566, time: 2014-09-18 18:08:23.521499, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11701) -9736> 2014-09-18 18:08:23.521566 7f4c9f251700 5 -- op tracker -- , seq: 85566, time: 2014-09-18 18:08:23.521564, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11701) -9735> 2014-09-18 18:08:23.522465 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3573 ==== pg_notify(2.1bb(9) epoch 11701) v5 ==== 1189+0+0 (3666553171 0 0) 0x12c12b20 con 0xcbecec0 -9734> 2014-09-18 18:08:23.522485 7f4c9f251700 5 -- op tracker -- , seq: 85567, time: 2014-09-18 18:08:23.522393, event: header_read, op: pg_notify(2.1bb(9) epoch 11701) -9733> 2014-09-18 18:08:23.522495 7f4c9f251700 5 -- op tracker -- , seq: 85567, time: 2014-09-18 18:08:23.522394, event: throttled, op: pg_notify(2.1bb(9) epoch 11701) -9732> 2014-09-18 18:08:23.522502 7f4c9f251700 5 -- op tracker -- , seq: 85567, time: 2014-09-18 18:08:23.522431, event: all_read, op: pg_notify(2.1bb(9) epoch 11701) -9731> 2014-09-18 18:08:23.522508 7f4c9f251700 5 -- op tracker -- , seq: 85567, time: 2014-09-18 18:08:23.522480, event: dispatched, op: pg_notify(2.1bb(9) epoch 11701) -9730> 2014-09-18 18:08:23.522514 7f4c9f251700 5 -- op tracker -- , seq: 85567, time: 2014-09-18 18:08:23.522513, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11701) -9729> 2014-09-18 18:08:23.522523 7f4c9f251700 5 -- op tracker -- , seq: 85567, time: 2014-09-18 18:08:23.522521, event: started, op: pg_notify(2.1bb(9) epoch 11701) -9728> 2014-09-18 18:08:23.522556 7f4c9f251700 5 -- op tracker -- , seq: 85567, time: 2014-09-18 18:08:23.522554, event: done, op: pg_notify(2.1bb(9) epoch 11701) -9727> 2014-09-18 18:08:23.525463 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3574 ==== pg_notify(2.78f(53) epoch 11701) v5 ==== 3561+0+0 (2996137846 0 0) 0xfa29480 con 0xcbecec0 -9726> 2014-09-18 18:08:23.525483 7f4c9f251700 5 -- op tracker -- , seq: 85568, time: 2014-09-18 18:08:23.525327, event: header_read, op: pg_notify(2.78f(53) epoch 11701) -9725> 2014-09-18 18:08:23.525494 7f4c9f251700 5 -- op tracker -- , seq: 85568, time: 2014-09-18 18:08:23.525328, event: throttled, op: pg_notify(2.78f(53) epoch 11701) -9724> 2014-09-18 18:08:23.525498 7f4c9f251700 5 -- op tracker -- , seq: 85568, time: 2014-09-18 18:08:23.525421, event: all_read, op: pg_notify(2.78f(53) epoch 11701) -9723> 2014-09-18 18:08:23.525510 7f4c9f251700 5 -- op tracker -- , seq: 85568, time: 2014-09-18 18:08:23.525479, event: dispatched, op: pg_notify(2.78f(53) epoch 11701) -9722> 2014-09-18 18:08:23.525517 7f4c9f251700 5 -- op tracker -- , seq: 85568, time: 2014-09-18 18:08:23.525515, event: waiting_for_osdmap, op: pg_notify(2.78f(53) epoch 11701) -9721> 2014-09-18 18:08:23.525525 7f4c9f251700 5 -- op tracker -- , seq: 85568, time: 2014-09-18 18:08:23.525523, event: started, op: pg_notify(2.78f(53) epoch 11701) -9720> 2014-09-18 18:08:23.525543 7f4c9f251700 5 -- op tracker -- , seq: 85568, time: 2014-09-18 18:08:23.525541, event: done, op: pg_notify(2.78f(53) epoch 11701) -9719> 2014-09-18 18:08:23.525561 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.612916 5 0.000206 -9718> 2014-09-18 18:08:23.525769 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -9717> 2014-09-18 18:08:23.525815 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000046 0 0.000000 -9716> 2014-09-18 18:08:23.525846 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -9715> 2014-09-18 18:08:23.525958 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000112 0 0.000000 -9714> 2014-09-18 18:08:23.525994 7f4c97241700 5 osd.42 pg_epoch: 11701 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -9713> 2014-09-18 18:08:23.526132 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3575 ==== pg_notify(2.78f(53) epoch 11701) v5 ==== 3561+0+0 (2996137846 0 0) 0xfa261e0 con 0xcbecec0 -9712> 2014-09-18 18:08:23.526145 7f4c9f251700 5 -- op tracker -- , seq: 85569, time: 2014-09-18 18:08:23.526059, event: header_read, op: pg_notify(2.78f(53) epoch 11701) -9711> 2014-09-18 18:08:23.526153 7f4c9f251700 5 -- op tracker -- , seq: 85569, time: 2014-09-18 18:08:23.526060, event: throttled, op: pg_notify(2.78f(53) epoch 11701) -9710> 2014-09-18 18:08:23.526159 7f4c9f251700 5 -- op tracker -- , seq: 85569, time: 2014-09-18 18:08:23.526121, event: all_read, op: pg_notify(2.78f(53) epoch 11701) -9709> 2014-09-18 18:08:23.526165 7f4c9f251700 5 -- op tracker -- , seq: 85569, time: 2014-09-18 18:08:23.526142, event: dispatched, op: pg_notify(2.78f(53) epoch 11701) -9708> 2014-09-18 18:08:23.526171 7f4c9f251700 5 -- op tracker -- , seq: 85569, time: 2014-09-18 18:08:23.526169, event: waiting_for_osdmap, op: pg_notify(2.78f(53) epoch 11701) -9707> 2014-09-18 18:08:23.526178 7f4c9f251700 5 -- op tracker -- , seq: 85569, time: 2014-09-18 18:08:23.526177, event: started, op: pg_notify(2.78f(53) epoch 11701) -9706> 2014-09-18 18:08:23.526195 7f4c9f251700 5 -- op tracker -- , seq: 85569, time: 2014-09-18 18:08:23.526193, event: done, op: pg_notify(2.78f(53) epoch 11701) -9705> 2014-09-18 18:08:23.545848 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.9 10.242.42.166:0/24959 108 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:23.547996) v2 ==== 47+0+0 (2030643358 0 0) 0xf9752a0 con 0x10a5c520 -9704> 2014-09-18 18:08:23.545873 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.547996) v2 -- ?+0 0xe1a23c0 con 0x10a5c520 -9703> 2014-09-18 18:08:23.545918 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.9 10.242.42.166:0/24959 108 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:23.547996) v2 ==== 47+0+0 (2030643358 0 0) 0x10506780 con 0x110bb9c0 -9702> 2014-09-18 18:08:23.545940 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.547996) v2 -- ?+0 0xfccb2a0 con 0x110bb9c0 -9701> 2014-09-18 18:08:23.634139 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.14 10.242.42.168:0/16237 138 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:23.501013) v2 ==== 47+0+0 (1017752422 0 0) 0xf7a65a0 con 0xcc49a20 -9700> 2014-09-18 18:08:23.634135 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.14 10.242.42.168:0/16237 138 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:23.501013) v2 ==== 47+0+0 (1017752422 0 0) 0xe0bb4a0 con 0x101cc3c0 -9699> 2014-09-18 18:08:23.634171 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.501013) v2 -- ?+0 0xf9752a0 con 0x101cc3c0 -9698> 2014-09-18 18:08:23.634201 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.501013) v2 -- ?+0 0x10506780 con 0xcc49a20 -9697> 2014-09-18 18:08:23.733338 7f4caae41700 5 osd.42 11701 tick -9696> 2014-09-18 18:08:23.841057 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1818 ==== osd_map(11701..11701 src has 1..11701) v3 ==== 2801+0+0 (2938961775 0 0) 0x15e318c0 con 0x10f60520 -9695> 2014-09-18 18:08:23.841106 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11701,11701], i have 11701, src has [1,11701] -9694> 2014-09-18 18:08:23.841115 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1819 ==== pg_notify(2.4f6(15) epoch 11701) v5 ==== 1581+0+0 (1256715868 0 0) 0x1310f660 con 0x10f60520 -9693> 2014-09-18 18:08:23.841133 7f4c9f251700 5 -- op tracker -- , seq: 85570, time: 2014-09-18 18:08:23.841010, event: header_read, op: pg_notify(2.4f6(15) epoch 11701) -9692> 2014-09-18 18:08:23.841154 7f4c9f251700 5 -- op tracker -- , seq: 85570, time: 2014-09-18 18:08:23.841010, event: throttled, op: pg_notify(2.4f6(15) epoch 11701) -9691> 2014-09-18 18:08:23.841162 7f4c9f251700 5 -- op tracker -- , seq: 85570, time: 2014-09-18 18:08:23.841066, event: all_read, op: pg_notify(2.4f6(15) epoch 11701) -9690> 2014-09-18 18:08:23.841169 7f4c9f251700 5 -- op tracker -- , seq: 85570, time: 2014-09-18 18:08:23.841128, event: dispatched, op: pg_notify(2.4f6(15) epoch 11701) -9689> 2014-09-18 18:08:23.841176 7f4c9f251700 5 -- op tracker -- , seq: 85570, time: 2014-09-18 18:08:23.841175, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11701) -9688> 2014-09-18 18:08:23.841188 7f4c9f251700 5 -- op tracker -- , seq: 85570, time: 2014-09-18 18:08:23.841187, event: started, op: pg_notify(2.4f6(15) epoch 11701) -9687> 2014-09-18 18:08:23.841217 7f4c9f251700 5 -- op tracker -- , seq: 85570, time: 2014-09-18 18:08:23.841215, event: done, op: pg_notify(2.4f6(15) epoch 11701) -9686> 2014-09-18 18:08:23.844532 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1820 ==== pg_notify(2.517(11) epoch 11701) v5 ==== 1335+0+0 (1026664535 0 0) 0x1310dfe0 con 0x10f60520 -9685> 2014-09-18 18:08:23.844552 7f4c9f251700 5 -- op tracker -- , seq: 85571, time: 2014-09-18 18:08:23.844442, event: header_read, op: pg_notify(2.517(11) epoch 11701) -9684> 2014-09-18 18:08:23.844562 7f4c9f251700 5 -- op tracker -- , seq: 85571, time: 2014-09-18 18:08:23.844443, event: throttled, op: pg_notify(2.517(11) epoch 11701) -9683> 2014-09-18 18:08:23.844570 7f4c9f251700 5 -- op tracker -- , seq: 85571, time: 2014-09-18 18:08:23.844487, event: all_read, op: pg_notify(2.517(11) epoch 11701) -9682> 2014-09-18 18:08:23.844577 7f4c9f251700 5 -- op tracker -- , seq: 85571, time: 2014-09-18 18:08:23.844547, event: dispatched, op: pg_notify(2.517(11) epoch 11701) -9681> 2014-09-18 18:08:23.844584 7f4c9f251700 5 -- op tracker -- , seq: 85571, time: 2014-09-18 18:08:23.844582, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11701) -9680> 2014-09-18 18:08:23.844592 7f4c9f251700 5 -- op tracker -- , seq: 85571, time: 2014-09-18 18:08:23.844590, event: started, op: pg_notify(2.517(11) epoch 11701) -9679> 2014-09-18 18:08:23.844611 7f4c9f251700 5 -- op tracker -- , seq: 85571, time: 2014-09-18 18:08:23.844610, event: done, op: pg_notify(2.517(11) epoch 11701) -9678> 2014-09-18 18:08:23.845453 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1821 ==== pg_notify(2.1d8(9) epoch 11701) v5 ==== 1261+0+0 (1225975860 0 0) 0x1310d860 con 0x10f60520 -9677> 2014-09-18 18:08:23.845473 7f4c9f251700 5 -- op tracker -- , seq: 85572, time: 2014-09-18 18:08:23.845381, event: header_read, op: pg_notify(2.1d8(9) epoch 11701) -9676> 2014-09-18 18:08:23.845483 7f4c9f251700 5 -- op tracker -- , seq: 85572, time: 2014-09-18 18:08:23.845382, event: throttled, op: pg_notify(2.1d8(9) epoch 11701) -9675> 2014-09-18 18:08:23.845490 7f4c9f251700 5 -- op tracker -- , seq: 85572, time: 2014-09-18 18:08:23.845415, event: all_read, op: pg_notify(2.1d8(9) epoch 11701) -9674> 2014-09-18 18:08:23.845496 7f4c9f251700 5 -- op tracker -- , seq: 85572, time: 2014-09-18 18:08:23.845468, event: dispatched, op: pg_notify(2.1d8(9) epoch 11701) -9673> 2014-09-18 18:08:23.845502 7f4c9f251700 5 -- op tracker -- , seq: 85572, time: 2014-09-18 18:08:23.845501, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11701) -9672> 2014-09-18 18:08:23.845510 7f4c9f251700 5 -- op tracker -- , seq: 85572, time: 2014-09-18 18:08:23.845509, event: started, op: pg_notify(2.1d8(9) epoch 11701) -9671> 2014-09-18 18:08:23.845529 7f4c9f251700 5 -- op tracker -- , seq: 85572, time: 2014-09-18 18:08:23.845527, event: done, op: pg_notify(2.1d8(9) epoch 11701) -9670> 2014-09-18 18:08:23.911307 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 296 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0xeeea640 con 0x140f95a0 -9669> 2014-09-18 18:08:23.911432 7f4c9f251700 3 osd.42 11701 handle_osd_map epochs [11702,11702], i have 11701, src has [1,11702] -9668> 2014-09-18 18:08:23.913361 7f4c8fa32700 5 osd.42 11701 heartbeat: osd_stat(230 GB used, 136 GB avail, 366 GB total, peers [0,2,3,4,6,9,11,20,22,23,24,27,30,32,35,39,43,47,48,53,55,56,57,58,63]/[] op hist []) -9667> 2014-09-18 18:08:23.913380 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6802/15878 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf812780 con 0x4751e40 -9666> 2014-09-18 18:08:23.913407 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6803/15878 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24fd0f00 con 0x149b0ec0 -9665> 2014-09-18 18:08:23.913423 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6818/19139 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24fd3c00 con 0x12b6a940 -9664> 2014-09-18 18:08:23.913439 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6820/19139 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24fd14a0 con 0x1d518aa0 -9663> 2014-09-18 18:08:23.913455 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6824/20623 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24fd2d00 con 0x1d662b00 -9662> 2014-09-18 18:08:23.913470 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6825/20623 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24fd2760 con 0x1d6629a0 -9661> 2014-09-18 18:08:23.913485 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6826/22324 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24fd12c0 con 0x1277db20 -9660> 2014-09-18 18:08:23.913499 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6828/22324 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xedfa960 con 0x1277d180 -9659> 2014-09-18 18:08:23.913515 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6810/22481 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xedfdde0 con 0x1096fa20 -9658> 2014-09-18 18:08:23.913530 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6813/22481 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xedfa000 con 0x10970520 -9657> 2014-09-18 18:08:23.913545 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6827/24959 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xedfc760 con 0x1db5c420 -9656> 2014-09-18 18:08:23.913560 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6840/24959 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf2b30e0 con 0x1db5c160 -9655> 2014-09-18 18:08:23.913576 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6822/17298 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae92a0 con 0x1217b760 -9654> 2014-09-18 18:08:23.913591 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6823/17298 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae90c0 con 0x1217c7e0 -9653> 2014-09-18 18:08:23.913607 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6823/33871 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae70e0 con 0xe58a9a0 -9652> 2014-09-18 18:08:23.913624 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6825/33871 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae7a40 con 0xe58a840 -9651> 2014-09-18 18:08:23.913641 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6820/37668 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae9de0 con 0x10a407e0 -9650> 2014-09-18 18:08:23.913660 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6832/37668 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae9c00 con 0x10a40260 -9649> 2014-09-18 18:08:23.913674 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6826/11463 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae6960 con 0xe417b20 -9648> 2014-09-18 18:08:23.913689 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6827/11463 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae8ee0 con 0xe58ab00 -9647> 2014-09-18 18:08:23.913705 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6803/5612 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae7c20 con 0xeff6f20 -9646> 2014-09-18 18:08:23.913720 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6810/5612 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xfae9480 con 0xeff6c60 -9645> 2014-09-18 18:08:23.913734 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6802/21548 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf872000 con 0x1da53a20 -9644> 2014-09-18 18:08:23.913749 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6803/21548 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x21f045a0 con 0x1da53760 -9643> 2014-09-18 18:08:23.913764 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6830/1204 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf728580 con 0x1db5cdc0 -9642> 2014-09-18 18:08:23.913778 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6831/1204 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf7281c0 con 0x1db5cc60 -9641> 2014-09-18 18:08:23.913777 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6818/19139 179 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0x1247cb40 con 0x12b6a940 -9640> 2014-09-18 18:08:23.913795 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6825/1099 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf29b2c0 con 0x1db5da20 -9639> 2014-09-18 18:08:23.913839 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6826/1099 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf0a0ee0 con 0x1db5db80 -9638> 2014-09-18 18:08:23.913866 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6813/17421 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24dead00 con 0x125239c0 -9637> 2014-09-18 18:08:23.913883 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6814/17421 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24de94a0 con 0x11d45860 -9636> 2014-09-18 18:08:23.913904 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6817/2660 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24de9a40 con 0x238dc420 -9635> 2014-09-18 18:08:23.913919 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6818/2660 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x24dea580 con 0x238dcb00 -9634> 2014-09-18 18:08:23.913933 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6843/4511 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba0960 con 0x149b1de0 -9633> 2014-09-18 18:08:23.913945 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6845/4511 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba2b20 con 0x149b07e0 -9632> 2014-09-18 18:08:23.913961 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6832/25184 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba2ee0 con 0x1217d180 -9631> 2014-09-18 18:08:23.913975 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6833/25184 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba0b40 con 0x1217caa0 -9630> 2014-09-18 18:08:23.914014 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6837/27583 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba0f00 con 0x10db26e0 -9629> 2014-09-18 18:08:23.914032 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6838/27583 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba3de0 con 0x10db4ec0 -9628> 2014-09-18 18:08:23.914073 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6826/2318 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba12c0 con 0x1db5c6e0 -9627> 2014-09-18 18:08:23.914091 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6832/2318 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba0d20 con 0x1db5e260 -9626> 2014-09-18 18:08:23.914113 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6830/19404 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba3480 con 0xe58d180 -9625> 2014-09-18 18:08:23.914129 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6831/19404 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba2760 con 0xe58d020 -9624> 2014-09-18 18:08:23.914212 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6840/36458 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba3660 con 0x1364b700 -9623> 2014-09-18 18:08:23.914243 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6842/36458 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0x12ba01e0 con 0x1364b860 -9622> 2014-09-18 18:08:23.914266 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6802/988 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xe76da40 con 0x1db5d600 -9621> 2014-09-18 18:08:23.914281 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6803/988 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xe76d2c0 con 0x1db5dfa0 -9620> 2014-09-18 18:08:23.914304 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6839/46756 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xe76e3a0 con 0x1d7c7a20 -9619> 2014-09-18 18:08:23.914319 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6842/46756 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf09cee0 con 0x1da543c0 -9618> 2014-09-18 18:08:23.914333 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6805/2487 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf09a5a0 con 0x1d996aa0 -9617> 2014-09-18 18:08:23.914349 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6809/2487 -- osd_ping(ping e11701 stamp 2014-09-18 18:08:23.913378) v2 -- ?+0 0xf09bfe0 con 0x1d996c00 -9616> 2014-09-18 18:08:23.914385 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6803/15878 188 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0x2b93b0e0 con 0x149b0ec0 -9615> 2014-09-18 18:08:23.914417 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6818/2660 183 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0x1393a3c0 con 0x238dcb00 -9614> 2014-09-18 18:08:23.914461 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6820/19139 179 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0x2b987e00 con 0x1d518aa0 -9613> 2014-09-18 18:08:23.914481 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6824/20623 168 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0x24dea3a0 con 0x1d662b00 -9612> 2014-09-18 18:08:23.914499 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6825/20623 168 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf5783a0 con 0x1d6629a0 -9611> 2014-09-18 18:08:23.914518 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6826/22324 162 ==== osd_ping(ping_reply e11699 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (2026088095 0 0) 0xf874940 con 0x1277db20 -9610> 2014-09-18 18:08:23.915066 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6819/22324 -- osd_map(11700..11701 src has 1..11701) v3 -- ?+0 0x10e23200 con 0x1f102840 -9609> 2014-09-18 18:08:23.915095 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6828/22324 162 ==== osd_ping(ping_reply e11699 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (2026088095 0 0) 0xf96c1c0 con 0x1277d180 -9608> 2014-09-18 18:08:23.915124 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6843/4511 160 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xfae74a0 con 0x149b1de0 -9607> 2014-09-18 18:08:23.915147 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6839/46756 158 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0x12112960 con 0x1d7c7a20 -9606> 2014-09-18 18:08:23.915169 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6842/46756 158 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf2b0ee0 con 0x1da543c0 -9605> 2014-09-18 18:08:23.915186 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6802/21548 157 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf275c00 con 0x1da53a20 -9604> 2014-09-18 18:08:23.915260 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6803/21548 157 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0x1393dc00 con 0x1da53760 -9603> 2014-09-18 18:08:23.915269 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11701/11701) [24,27,42] r=2 lpr=11701 pi=2630-11700/64 crt=6750'24785 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.007100 4 0.000245 -9602> 2014-09-18 18:08:23.915301 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6822/17298 135 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xe76c780 con 0x1217b760 -9601> 2014-09-18 18:08:23.915317 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11701/11701) [24,27,42] r=2 lpr=11701 pi=2630-11700/64 crt=6750'24785 lcod 0'0 inactive NOTIFY] exit Started 1.007209 0 0.000000 -9600> 2014-09-18 18:08:23.915336 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11701/11701) [24,27,42] r=2 lpr=11701 pi=2630-11700/64 crt=6750'24785 lcod 0'0 inactive NOTIFY] enter Reset -9599> 2014-09-18 18:08:23.915340 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6823/17298 135 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf0dc3a0 con 0x1217c7e0 -9598> 2014-09-18 18:08:23.915377 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6832/25184 135 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf09c580 con 0x1217d180 -9597> 2014-09-18 18:08:23.915408 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6831/25184 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x10e23f80 con 0x12562260 -9596> 2014-09-18 18:08:23.915425 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6833/25184 135 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf866b40 con 0x1217caa0 -9595> 2014-09-18 18:08:23.915457 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6802/988 123 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xe0c9660 con 0x1db5d600 -9594> 2014-09-18 18:08:23.915476 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6803/988 123 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xe9fef00 con 0x1db5dfa0 -9593> 2014-09-18 18:08:23.915499 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 24 ==== osd_map(11699..11702 src has 1..11702) v3 ==== 5463+0+0 (2524449185 0 0) 0x1182b200 con 0x111771e0 -9592> 2014-09-18 18:08:23.915543 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6825/1099 123 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf93b0e0 con 0x1db5da20 -9591> 2014-09-18 18:08:23.915574 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x1182da80 con 0x1395ef20 -9590> 2014-09-18 18:08:23.915591 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6826/1099 123 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0x24de8780 con 0x1db5db80 -9589> 2014-09-18 18:08:23.915646 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] exit Reset 0.000310 1 0.000124 -9588> 2014-09-18 18:08:23.915672 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] enter Started -9587> 2014-09-18 18:08:23.915682 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] enter Start -9586> 2014-09-18 18:08:23.915697 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6830/1204 123 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0x104123c0 con 0x1db5cdc0 -9585> 2014-09-18 18:08:23.915708 7f4ca0a54700 3 osd.42 11702 handle_osd_map epochs [11699,11702], i have 11702, src has [1,11702] -9584> 2014-09-18 18:08:23.915702 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 297 ==== pg_info(1 pgs e11702:2.756(15)) v4 ==== 1531+0+0 (4110897726 0 0) 0xe1abfe0 con 0x140f95a0 -9583> 2014-09-18 18:08:23.915714 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6831/1204 123 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf872f00 con 0x1db5cc60 -9582> 2014-09-18 18:08:23.915726 7f4c9f251700 5 -- op tracker -- , seq: 85573, time: 2014-09-18 18:08:23.911327, event: header_read, op: pg_info(1 pgs e11702:2.756(15)) -9581> 2014-09-18 18:08:23.915728 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6827/24959 121 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf875480 con 0x1db5c420 -9580> 2014-09-18 18:08:23.915740 7f4c9f251700 5 -- op tracker -- , seq: 85573, time: 2014-09-18 18:08:23.911328, event: throttled, op: pg_info(1 pgs e11702:2.756(15)) -9579> 2014-09-18 18:08:23.915746 7f4c9f251700 5 -- op tracker -- , seq: 85573, time: 2014-09-18 18:08:23.911376, event: all_read, op: pg_info(1 pgs e11702:2.756(15)) -9578> 2014-09-18 18:08:23.915749 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x2db44fc0 con 0x1395ec60 -9577> 2014-09-18 18:08:23.915752 7f4c9f251700 5 -- op tracker -- , seq: 85573, time: 2014-09-18 18:08:23.915721, event: dispatched, op: pg_info(1 pgs e11702:2.756(15)) -9576> 2014-09-18 18:08:23.915760 7f4c9f251700 5 -- op tracker -- , seq: 85573, time: 2014-09-18 18:08:23.915758, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.756(15)) -9575> 2014-09-18 18:08:23.915770 7f4c9f251700 5 -- op tracker -- , seq: 85573, time: 2014-09-18 18:08:23.915769, event: started, op: pg_info(1 pgs e11702:2.756(15)) -9574> 2014-09-18 18:08:23.915767 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6840/24959 121 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xea010c0 con 0x1db5c160 -9573> 2014-09-18 18:08:23.915690 7f4c96a40700 1 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -9572> 2014-09-18 18:08:23.915809 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6826/2318 121 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0x12112000 con 0x1db5c6e0 -9571> 2014-09-18 18:08:23.915812 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] exit Start 0.000130 0 0.000000 -9570> 2014-09-18 18:08:23.915838 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.170:6816/2318 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x2db46880 con 0xe9a27e0 -9569> 2014-09-18 18:08:23.915837 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] enter Started/Stray -9568> 2014-09-18 18:08:23.915859 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6832/2318 121 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0x2b923a40 con 0x1db5e260 -9567> 2014-09-18 18:08:23.915882 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6837/27583 101 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xfae7fe0 con 0x10db26e0 -9566> 2014-09-18 18:08:23.915902 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6838/27583 101 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0x19f36960 con 0x10db4ec0 -9565> 2014-09-18 18:08:23.915919 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6826/11463 1574 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf160760 con 0xe417b20 -9564> 2014-09-18 18:08:23.915976 7f4c9f251700 5 -- op tracker -- , seq: 85573, time: 2014-09-18 18:08:23.915974, event: done, op: pg_info(1 pgs e11702:2.756(15)) -9563> 2014-09-18 18:08:23.915939 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6827/11463 1574 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xe1a50c0 con 0xe58ab00 -9562> 2014-09-18 18:08:23.916239 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6823/33871 1574 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xe7952c0 con 0xe58a9a0 -9561> 2014-09-18 18:08:23.916266 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6825/33871 1574 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xe958ee0 con 0xe58a840 -9560> 2014-09-18 18:08:23.916286 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6830/19404 1574 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xe1a5840 con 0xe58d180 -9559> 2014-09-18 18:08:23.916310 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6829/19404 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x2db45b00 con 0xe2d9a20 -9558> 2014-09-18 18:08:23.916330 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6831/19404 1574 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf1e9840 con 0xe58d020 -9557> 2014-09-18 18:08:23.916162 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1024 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0xe939840 con 0x14358260 -9556> 2014-09-18 18:08:23.916364 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9555> 2014-09-18 18:08:23.916375 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.43 10.242.42.178:6841/4511 2 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x14409b00 con 0x101e38c0 -9554> 2014-09-18 18:08:23.916385 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9553> 2014-09-18 18:08:23.916389 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 43 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x10d18640 con 0x142f8260 -9552> 2014-09-18 18:08:23.916397 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9551> 2014-09-18 18:08:23.916401 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 900 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x2fb9f440 con 0x1395f8c0 -9550> 2014-09-18 18:08:23.916409 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9549> 2014-09-18 18:08:23.916414 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 27 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x24dbd680 con 0x1395f340 -9548> 2014-09-18 18:08:23.916422 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9547> 2014-09-18 18:08:23.916426 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 114 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x2fb9e000 con 0x1395f080 -9546> 2014-09-18 18:08:23.916435 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9545> 2014-09-18 18:08:23.916439 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 103 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x1e282fc0 con 0x1395edc0 -9544> 2014-09-18 18:08:23.916446 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9543> 2014-09-18 18:08:23.916450 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.48 10.242.42.164:6836/27583 11 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x11bb93c0 con 0x10aebe40 -9542> 2014-09-18 18:08:23.916458 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9541> 2014-09-18 18:08:23.916464 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2330 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0xeee9d40 con 0xdd8c940 -9540> 2014-09-18 18:08:23.916472 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9539> 2014-09-18 18:08:23.916477 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3266 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x24d1bb00 con 0x4900580 -9538> 2014-09-18 18:08:23.916485 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9537> 2014-09-18 18:08:23.916493 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2582 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x19df1f80 con 0x4900b00 -9536> 2014-09-18 18:08:23.916502 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9535> 2014-09-18 18:08:23.916509 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1709 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x24dbc6c0 con 0x4900000 -9534> 2014-09-18 18:08:23.916518 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9533> 2014-09-18 18:08:23.916508 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6840/36458 654 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf12d480 con 0x1364b700 -9532> 2014-09-18 18:08:23.916523 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1025 ==== pg_notify(2.719(17) epoch 11702) v5 ==== 1601+0+0 (4234779878 0 0) 0x2b93fe00 con 0x14358260 -9531> 2014-09-18 18:08:23.916548 7f4c9f251700 5 -- op tracker -- , seq: 85574, time: 2014-09-18 18:08:23.916236, event: header_read, op: pg_notify(2.719(17) epoch 11702) -9530> 2014-09-18 18:08:23.916564 7f4c9f251700 5 -- op tracker -- , seq: 85574, time: 2014-09-18 18:08:23.916237, event: throttled, op: pg_notify(2.719(17) epoch 11702) -9529> 2014-09-18 18:08:23.916546 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6842/36458 654 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf4e6d20 con 0x1364b860 -9528> 2014-09-18 18:08:23.916573 7f4c9f251700 5 -- op tracker -- , seq: 85574, time: 2014-09-18 18:08:23.916296, event: all_read, op: pg_notify(2.719(17) epoch 11702) -9527> 2014-09-18 18:08:23.916578 7f4c9f251700 5 -- op tracker -- , seq: 85574, time: 2014-09-18 18:08:23.916541, event: dispatched, op: pg_notify(2.719(17) epoch 11702) -9526> 2014-09-18 18:08:23.916584 7f4c9f251700 5 -- op tracker -- , seq: 85574, time: 2014-09-18 18:08:23.916583, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11702) -9525> 2014-09-18 18:08:23.916582 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6810/22481 454 ==== osd_ping(ping_reply e11700 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (375124941 0 0) 0x128cb4a0 con 0x1096fa20 -9524> 2014-09-18 18:08:23.916593 7f4c9f251700 5 -- op tracker -- , seq: 85574, time: 2014-09-18 18:08:23.916591, event: started, op: pg_notify(2.719(17) epoch 11702) -9523> 2014-09-18 18:08:23.916610 7f4c9f251700 5 -- op tracker -- , seq: 85574, time: 2014-09-18 18:08:23.916609, event: done, op: pg_notify(2.719(17) epoch 11702) -9522> 2014-09-18 18:08:23.916625 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 901 ==== pg_notify(2.1d8(9) epoch 11702) v5 ==== 1261+0+0 (784760658 0 0) 0xf93a5a0 con 0x1395f8c0 -9521> 2014-09-18 18:08:23.916637 7f4c9f251700 5 -- op tracker -- , seq: 85575, time: 2014-09-18 18:08:23.913445, event: header_read, op: pg_notify(2.1d8(9) epoch 11702) -9520> 2014-09-18 18:08:23.916642 7f4c9f251700 5 -- op tracker -- , seq: 85575, time: 2014-09-18 18:08:23.913446, event: throttled, op: pg_notify(2.1d8(9) epoch 11702) -9519> 2014-09-18 18:08:23.916646 7f4c9f251700 5 -- op tracker -- , seq: 85575, time: 2014-09-18 18:08:23.913483, event: all_read, op: pg_notify(2.1d8(9) epoch 11702) -9518> 2014-09-18 18:08:23.916652 7f4c9f251700 5 -- op tracker -- , seq: 85575, time: 2014-09-18 18:08:23.916635, event: dispatched, op: pg_notify(2.1d8(9) epoch 11702) -9517> 2014-09-18 18:08:23.916659 7f4c9f251700 5 -- op tracker -- , seq: 85575, time: 2014-09-18 18:08:23.916657, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11702) -9516> 2014-09-18 18:08:23.916664 7f4c9f251700 5 -- op tracker -- , seq: 85575, time: 2014-09-18 18:08:23.916663, event: started, op: pg_notify(2.1d8(9) epoch 11702) -9515> 2014-09-18 18:08:23.916696 7f4c9f251700 5 -- op tracker -- , seq: 85575, time: 2014-09-18 18:08:23.916695, event: done, op: pg_notify(2.1d8(9) epoch 11702) -9514> 2014-09-18 18:08:23.916708 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 28 ==== pg_info(1 pgs e11702:2.29c(20)) v4 ==== 1786+0+0 (2162697931 0 0) 0xe1abc20 con 0x1395f340 -9513> 2014-09-18 18:08:23.916718 7f4c9f251700 5 -- op tracker -- , seq: 85576, time: 2014-09-18 18:08:23.915811, event: header_read, op: pg_info(1 pgs e11702:2.29c(20)) -9512> 2014-09-18 18:08:23.916725 7f4c9f251700 5 -- op tracker -- , seq: 85576, time: 2014-09-18 18:08:23.915812, event: throttled, op: pg_info(1 pgs e11702:2.29c(20)) -9511> 2014-09-18 18:08:23.916731 7f4c9f251700 5 -- op tracker -- , seq: 85576, time: 2014-09-18 18:08:23.915856, event: all_read, op: pg_info(1 pgs e11702:2.29c(20)) -9510> 2014-09-18 18:08:23.916737 7f4c9f251700 5 -- op tracker -- , seq: 85576, time: 2014-09-18 18:08:23.916716, event: dispatched, op: pg_info(1 pgs e11702:2.29c(20)) -9509> 2014-09-18 18:08:23.916743 7f4c9f251700 5 -- op tracker -- , seq: 85576, time: 2014-09-18 18:08:23.916742, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.29c(20)) -9508> 2014-09-18 18:08:23.916751 7f4c9f251700 5 -- op tracker -- , seq: 85576, time: 2014-09-18 18:08:23.916749, event: started, op: pg_info(1 pgs e11702:2.29c(20)) -9507> 2014-09-18 18:08:23.916764 7f4c9f251700 5 -- op tracker -- , seq: 85576, time: 2014-09-18 18:08:23.916763, event: done, op: pg_info(1 pgs e11702:2.29c(20)) -9506> 2014-09-18 18:08:23.916775 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3267 ==== pg_notify(2.4f6(184) epoch 11702) v5 ==== 11296+0+0 (511599619 0 0) 0x24fd3840 con 0x4900580 -9505> 2014-09-18 18:08:23.916786 7f4c9f251700 5 -- op tracker -- , seq: 85577, time: 2014-09-18 18:08:23.915646, event: header_read, op: pg_notify(2.4f6(184) epoch 11702) -9504> 2014-09-18 18:08:23.916793 7f4c9f251700 5 -- op tracker -- , seq: 85577, time: 2014-09-18 18:08:23.915647, event: throttled, op: pg_notify(2.4f6(184) epoch 11702) -9503> 2014-09-18 18:08:23.916798 7f4c9f251700 5 -- op tracker -- , seq: 85577, time: 2014-09-18 18:08:23.915902, event: all_read, op: pg_notify(2.4f6(184) epoch 11702) -9502> 2014-09-18 18:08:23.916803 7f4c9f251700 5 -- op tracker -- , seq: 85577, time: 2014-09-18 18:08:23.916783, event: dispatched, op: pg_notify(2.4f6(184) epoch 11702) -9501> 2014-09-18 18:08:23.916808 7f4c9f251700 5 -- op tracker -- , seq: 85577, time: 2014-09-18 18:08:23.916807, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11702) -9500> 2014-09-18 18:08:23.916815 7f4c9f251700 5 -- op tracker -- , seq: 85577, time: 2014-09-18 18:08:23.916814, event: started, op: pg_notify(2.4f6(184) epoch 11702) -9499> 2014-09-18 18:08:23.916811 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x2db473c0 con 0x1088aec0 -9498> 2014-09-18 18:08:23.916833 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6813/22481 454 ==== osd_ping(ping_reply e11700 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (375124941 0 0) 0x21f050e0 con 0x10970520 -9497> 2014-09-18 18:08:23.916881 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6820/37668 448 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf974b20 con 0x10a407e0 -9496> 2014-09-18 18:08:23.916885 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.3b5(65) epoch 11702) v5 -- ?+0 0xe1aaf00 con 0x1395ec60 -9495> 2014-09-18 18:08:23.916901 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6815/37668 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0xf85b440 con 0x1088b440 -9494> 2014-09-18 18:08:23.916914 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6832/37668 448 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf242000 con 0x10a40260 -9493> 2014-09-18 18:08:23.916951 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6803/5612 340 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf96d0c0 con 0xeff6f20 -9492> 2014-09-18 18:08:23.916972 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6810/5612 340 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0x24fa32a0 con 0xeff6c60 -9491> 2014-09-18 18:08:23.916986 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6805/2487 329 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0x1368fde0 con 0x1d996aa0 -9490> 2014-09-18 18:08:23.917000 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6803/2487 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0xf85a000 con 0xe8db760 -9489> 2014-09-18 18:08:23.917013 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6809/2487 329 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xec68d20 con 0x1d996c00 -9488> 2014-09-18 18:08:23.917042 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6802/15878 188 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xfae6b40 con 0x4751e40 -9487> 2014-09-18 18:08:23.917060 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0xf85c880 con 0x10a878c0 -9486> 2014-09-18 18:08:23.917071 7f4c9f251700 5 -- op tracker -- , seq: 85577, time: 2014-09-18 18:08:23.917065, event: done, op: pg_notify(2.4f6(184) epoch 11702) -9485> 2014-09-18 18:08:23.917071 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6813/17421 185 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xeb79e00 con 0x125239c0 -9484> 2014-09-18 18:08:23.917089 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0xf85bb00 con 0xcbecec0 -9483> 2014-09-18 18:08:23.917100 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2583 ==== pg_notify(2.71c(12) epoch 11702) v5 ==== 1424+0+0 (3547524162 0 0) 0xf6e1de0 con 0x4900b00 -9482> 2014-09-18 18:08:23.917106 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6814/17421 185 ==== osd_ping(ping_reply e11701 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (1862046528 0 0) 0xf1ec1c0 con 0x11d45860 -9481> 2014-09-18 18:08:23.917116 7f4c9f251700 5 -- op tracker -- , seq: 85578, time: 2014-09-18 18:08:23.914729, event: header_read, op: pg_notify(2.71c(12) epoch 11702) -9480> 2014-09-18 18:08:23.917125 7f4c9f251700 5 -- op tracker -- , seq: 85578, time: 2014-09-18 18:08:23.914730, event: throttled, op: pg_notify(2.71c(12) epoch 11702) -9479> 2014-09-18 18:08:23.917124 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6817/2660 183 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0xf29af00 con 0x238dc420 -9478> 2014-09-18 18:08:23.917133 7f4c9f251700 5 -- op tracker -- , seq: 85578, time: 2014-09-18 18:08:23.914763, event: all_read, op: pg_notify(2.71c(12) epoch 11702) -9477> 2014-09-18 18:08:23.917136 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6845/4511 160 ==== osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.913378) v2 ==== 47+0+0 (3876926167 0 0) 0x12c11680 con 0x149b07e0 -9476> 2014-09-18 18:08:23.917141 7f4c9f251700 5 -- op tracker -- , seq: 85578, time: 2014-09-18 18:08:23.917111, event: dispatched, op: pg_notify(2.71c(12) epoch 11702) -9475> 2014-09-18 18:08:23.917148 7f4c9f251700 5 -- op tracker -- , seq: 85578, time: 2014-09-18 18:08:23.917147, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11702) -9474> 2014-09-18 18:08:23.917156 7f4c9f251700 5 -- op tracker -- , seq: 85578, time: 2014-09-18 18:08:23.917153, event: started, op: pg_notify(2.71c(12) epoch 11702) -9473> 2014-09-18 18:08:23.917173 7f4c9f251700 5 -- op tracker -- , seq: 85578, time: 2014-09-18 18:08:23.917171, event: done, op: pg_notify(2.71c(12) epoch 11702) -9472> 2014-09-18 18:08:23.917183 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1710 ==== pg_notify(2.e0(79) epoch 11702) v5 ==== 5119+0+0 (2141318656 0 0) 0xf1ed840 con 0x4900000 -9471> 2014-09-18 18:08:23.917195 7f4c9f251700 5 -- op tracker -- , seq: 85579, time: 2014-09-18 18:08:23.916374, event: header_read, op: pg_notify(2.e0(79) epoch 11702) -9470> 2014-09-18 18:08:23.917203 7f4c9f251700 5 -- op tracker -- , seq: 85579, time: 2014-09-18 18:08:23.916375, event: throttled, op: pg_notify(2.e0(79) epoch 11702) -9469> 2014-09-18 18:08:23.917210 7f4c9f251700 5 -- op tracker -- , seq: 85579, time: 2014-09-18 18:08:23.916482, event: all_read, op: pg_notify(2.e0(79) epoch 11702) -9468> 2014-09-18 18:08:23.917217 7f4c9f251700 5 -- op tracker -- , seq: 85579, time: 2014-09-18 18:08:23.917192, event: dispatched, op: pg_notify(2.e0(79) epoch 11702) -9467> 2014-09-18 18:08:23.917225 7f4c9f251700 5 -- op tracker -- , seq: 85579, time: 2014-09-18 18:08:23.917222, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11702) -9466> 2014-09-18 18:08:23.917234 7f4c9f251700 5 -- op tracker -- , seq: 85579, time: 2014-09-18 18:08:23.917231, event: started, op: pg_notify(2.e0(79) epoch 11702) -9465> 2014-09-18 18:08:23.917253 7f4c9f251700 5 -- op tracker -- , seq: 85579, time: 2014-09-18 18:08:23.917251, event: done, op: pg_notify(2.e0(79) epoch 11702) -9464> 2014-09-18 18:08:23.917271 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 29 ==== pg_notify(2.1d8(9) epoch 11702) v5 ==== 1261+0+0 (784760658 0 0) 0x100efa40 con 0x1395f340 -9463> 2014-09-18 18:08:23.917283 7f4c9f251700 5 -- op tracker -- , seq: 85580, time: 2014-09-18 18:08:23.916042, event: header_read, op: pg_notify(2.1d8(9) epoch 11702) -9462> 2014-09-18 18:08:23.917291 7f4c9f251700 5 -- op tracker -- , seq: 85580, time: 2014-09-18 18:08:23.916043, event: throttled, op: pg_notify(2.1d8(9) epoch 11702) -9461> 2014-09-18 18:08:23.917299 7f4c9f251700 5 -- op tracker -- , seq: 85580, time: 2014-09-18 18:08:23.916082, event: all_read, op: pg_notify(2.1d8(9) epoch 11702) -9460> 2014-09-18 18:08:23.917304 7f4c9f251700 5 -- op tracker -- , seq: 85580, time: 2014-09-18 18:08:23.917279, event: dispatched, op: pg_notify(2.1d8(9) epoch 11702) -9459> 2014-09-18 18:08:23.917309 7f4c9f251700 5 -- op tracker -- , seq: 85580, time: 2014-09-18 18:08:23.917307, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11702) -9458> 2014-09-18 18:08:23.917314 7f4c9f251700 5 -- op tracker -- , seq: 85580, time: 2014-09-18 18:08:23.917313, event: started, op: pg_notify(2.1d8(9) epoch 11702) -9457> 2014-09-18 18:08:23.917326 7f4c9f251700 5 -- op tracker -- , seq: 85580, time: 2014-09-18 18:08:23.917325, event: done, op: pg_notify(2.1d8(9) epoch 11702) -9456> 2014-09-18 18:08:23.917334 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2331 ==== pg_notify(2.2c7(139) epoch 11702) v5 ==== 8651+0+0 (3946967282 0 0) 0xe1ad480 con 0xdd8c940 -9455> 2014-09-18 18:08:23.917342 7f4c9f251700 5 -- op tracker -- , seq: 85581, time: 2014-09-18 18:08:23.916942, event: header_read, op: pg_notify(2.2c7(139) epoch 11702) -9454> 2014-09-18 18:08:23.917346 7f4c9f251700 5 -- op tracker -- , seq: 85581, time: 2014-09-18 18:08:23.916943, event: throttled, op: pg_notify(2.2c7(139) epoch 11702) -9453> 2014-09-18 18:08:23.917350 7f4c9f251700 5 -- op tracker -- , seq: 85581, time: 2014-09-18 18:08:23.917152, event: all_read, op: pg_notify(2.2c7(139) epoch 11702) -9452> 2014-09-18 18:08:23.917354 7f4c9f251700 5 -- op tracker -- , seq: 85581, time: 2014-09-18 18:08:23.917339, event: dispatched, op: pg_notify(2.2c7(139) epoch 11702) -9451> 2014-09-18 18:08:23.917358 7f4c9f251700 5 -- op tracker -- , seq: 85581, time: 2014-09-18 18:08:23.917357, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11702) -9450> 2014-09-18 18:08:23.917363 7f4c9f251700 5 -- op tracker -- , seq: 85581, time: 2014-09-18 18:08:23.917362, event: started, op: pg_notify(2.2c7(139) epoch 11702) -9449> 2014-09-18 18:08:23.917374 7f4c9f251700 5 -- op tracker -- , seq: 85581, time: 2014-09-18 18:08:23.917373, event: done, op: pg_notify(2.2c7(139) epoch 11702) -9448> 2014-09-18 18:08:23.917418 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3268 ==== pg_notify(2.17e(17) epoch 11702) v5 ==== 1593+0+0 (2095721058 0 0) 0x2b93ffe0 con 0x4900580 -9447> 2014-09-18 18:08:23.917420 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x2fb9ff80 con 0x140f95a0 -9446> 2014-09-18 18:08:23.917426 7f4c9f251700 5 -- op tracker -- , seq: 85582, time: 2014-09-18 18:08:23.915943, event: header_read, op: pg_notify(2.17e(17) epoch 11702) -9445> 2014-09-18 18:08:23.917431 7f4c9f251700 5 -- op tracker -- , seq: 85582, time: 2014-09-18 18:08:23.915944, event: throttled, op: pg_notify(2.17e(17) epoch 11702) -9444> 2014-09-18 18:08:23.917435 7f4c9f251700 5 -- op tracker -- , seq: 85582, time: 2014-09-18 18:08:23.916005, event: all_read, op: pg_notify(2.17e(17) epoch 11702) -9443> 2014-09-18 18:08:23.917439 7f4c9f251700 5 -- op tracker -- , seq: 85582, time: 2014-09-18 18:08:23.917424, event: dispatched, op: pg_notify(2.17e(17) epoch 11702) -9442> 2014-09-18 18:08:23.917443 7f4c9f251700 5 -- op tracker -- , seq: 85582, time: 2014-09-18 18:08:23.917442, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11702) -9441> 2014-09-18 18:08:23.917448 7f4c9f251700 5 -- op tracker -- , seq: 85582, time: 2014-09-18 18:08:23.917447, event: started, op: pg_notify(2.17e(17) epoch 11702) -9440> 2014-09-18 18:08:23.917460 7f4c9f251700 5 -- op tracker -- , seq: 85582, time: 2014-09-18 18:08:23.917458, event: done, op: pg_notify(2.17e(17) epoch 11702) -9439> 2014-09-18 18:08:23.917436 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11702) v5 -- ?+0 0x100f1a20 con 0x140f95a0 -9438> 2014-09-18 18:08:23.917485 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11702) v5 -- ?+0 0x12c11a40 con 0x1395edc0 -9437> 2014-09-18 18:08:23.917614 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6809/18319 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x1182ab40 con 0x1395eb00 -9436> 2014-09-18 18:08:23.917815 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 902 ==== pg_notify(2.78f(153) epoch 11702) v5 ==== 9229+0+0 (1305360145 0 0) 0xf8741c0 con 0x1395f8c0 -9435> 2014-09-18 18:08:23.917839 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6809/18319 -- pg_notify(2.6a6(17) epoch 11702) v5 -- ?+0 0xe1acd00 con 0x1395eb00 -9434> 2014-09-18 18:08:23.917842 7f4c9f251700 5 -- op tracker -- , seq: 85583, time: 2014-09-18 18:08:23.917653, event: header_read, op: pg_notify(2.78f(153) epoch 11702) -9433> 2014-09-18 18:08:23.917851 7f4c9f251700 5 -- op tracker -- , seq: 85583, time: 2014-09-18 18:08:23.917654, event: throttled, op: pg_notify(2.78f(153) epoch 11702) -9432> 2014-09-18 18:08:23.917858 7f4c9f251700 5 -- op tracker -- , seq: 85583, time: 2014-09-18 18:08:23.917797, event: all_read, op: pg_notify(2.78f(153) epoch 11702) -9431> 2014-09-18 18:08:23.917863 7f4c9f251700 5 -- op tracker -- , seq: 85583, time: 2014-09-18 18:08:23.917836, event: dispatched, op: pg_notify(2.78f(153) epoch 11702) -9430> 2014-09-18 18:08:23.917867 7f4c9f251700 5 -- op tracker -- , seq: 85583, time: 2014-09-18 18:08:23.917866, event: waiting_for_osdmap, op: pg_notify(2.78f(153) epoch 11702) -9429> 2014-09-18 18:08:23.917873 7f4c9f251700 5 -- op tracker -- , seq: 85583, time: 2014-09-18 18:08:23.917872, event: started, op: pg_notify(2.78f(153) epoch 11702) -9428> 2014-09-18 18:08:23.917889 7f4c9f251700 5 -- op tracker -- , seq: 85583, time: 2014-09-18 18:08:23.917887, event: done, op: pg_notify(2.78f(153) epoch 11702) -9427> 2014-09-18 18:08:23.917892 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0x1182cd00 con 0x1395f1e0 -9426> 2014-09-18 18:08:23.917905 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11702) v5 -- ?+0 0xe1aab40 con 0x1395f1e0 -9425> 2014-09-18 18:08:23.917913 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3269 ==== pg_notify(2.1bb(9) epoch 11702) v5 ==== 1189+0+0 (34004194 0 0) 0x2b941480 con 0x4900580 -9424> 2014-09-18 18:08:23.917927 7f4c9f251700 5 -- op tracker -- , seq: 85584, time: 2014-09-18 18:08:23.917780, event: header_read, op: pg_notify(2.1bb(9) epoch 11702) -9423> 2014-09-18 18:08:23.917937 7f4c9f251700 5 -- op tracker -- , seq: 85584, time: 2014-09-18 18:08:23.917781, event: throttled, op: pg_notify(2.1bb(9) epoch 11702) -9422> 2014-09-18 18:08:23.917950 7f4c9f251700 5 -- op tracker -- , seq: 85584, time: 2014-09-18 18:08:23.917833, event: all_read, op: pg_notify(2.1bb(9) epoch 11702) -9421> 2014-09-18 18:08:23.917956 7f4c9f251700 5 -- op tracker -- , seq: 85584, time: 2014-09-18 18:08:23.917923, event: dispatched, op: pg_notify(2.1bb(9) epoch 11702) -9420> 2014-09-18 18:08:23.917963 7f4c9f251700 5 -- op tracker -- , seq: 85584, time: 2014-09-18 18:08:23.917961, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11702) -9419> 2014-09-18 18:08:23.917972 7f4c9f251700 5 -- op tracker -- , seq: 85584, time: 2014-09-18 18:08:23.917970, event: started, op: pg_notify(2.1bb(9) epoch 11702) -9418> 2014-09-18 18:08:23.917992 7f4c9f251700 5 -- op tracker -- , seq: 85584, time: 2014-09-18 18:08:23.917989, event: done, op: pg_notify(2.1bb(9) epoch 11702) -9417> 2014-09-18 18:08:23.918202 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6804/17298 -- pg_notify(2.29c(79) epoch 11702) v5 -- ?+0 0xf488940 con 0x1395f340 -9416> 2014-09-18 18:08:23.918421 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1822 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x1182c400 con 0x10f60520 -9415> 2014-09-18 18:08:23.918442 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9414> 2014-09-18 18:08:23.918500 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1823 ==== pg_notify(2.4f6(15) epoch 11702) v5 ==== 1581+0+0 (808019215 0 0) 0x1310c1e0 con 0x10f60520 -9413> 2014-09-18 18:08:23.918518 7f4c9f251700 5 -- op tracker -- , seq: 85585, time: 2014-09-18 18:08:23.918417, event: header_read, op: pg_notify(2.4f6(15) epoch 11702) -9412> 2014-09-18 18:08:23.918526 7f4c9f251700 5 -- op tracker -- , seq: 85585, time: 2014-09-18 18:08:23.918417, event: throttled, op: pg_notify(2.4f6(15) epoch 11702) -9411> 2014-09-18 18:08:23.918532 7f4c9f251700 5 -- op tracker -- , seq: 85585, time: 2014-09-18 18:08:23.918484, event: all_read, op: pg_notify(2.4f6(15) epoch 11702) -9410> 2014-09-18 18:08:23.918549 7f4c9f251700 5 -- op tracker -- , seq: 85585, time: 2014-09-18 18:08:23.918513, event: dispatched, op: pg_notify(2.4f6(15) epoch 11702) -9409> 2014-09-18 18:08:23.918557 7f4c9f251700 5 -- op tracker -- , seq: 85585, time: 2014-09-18 18:08:23.918555, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11702) -9408> 2014-09-18 18:08:23.918565 7f4c9f251700 5 -- op tracker -- , seq: 85585, time: 2014-09-18 18:08:23.918563, event: started, op: pg_notify(2.4f6(15) epoch 11702) -9407> 2014-09-18 18:08:23.918584 7f4c9f251700 5 -- op tracker -- , seq: 85585, time: 2014-09-18 18:08:23.918581, event: done, op: pg_notify(2.4f6(15) epoch 11702) -9406> 2014-09-18 18:08:23.919049 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13),2.756(26) epoch 11702) v5 -- ?+0 0xf05ba40 con 0x140f95a0 -9405> 2014-09-18 18:08:23.919128 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6e3(27) epoch 11702) v5 -- ?+0 0xf05b4a0 con 0x1395edc0 -9404> 2014-09-18 18:08:23.919176 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1250 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0xf85a000 con 0xe8db760 -9403> 2014-09-18 18:08:23.919191 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9402> 2014-09-18 18:08:23.919198 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1251 ==== pg_notify(2.17e(17) epoch 11702) v5 ==== 1593+0+0 (2095721058 0 0) 0xf109480 con 0xe8db760 -9401> 2014-09-18 18:08:23.919209 7f4c9f251700 5 -- op tracker -- , seq: 85586, time: 2014-09-18 18:08:23.919098, event: header_read, op: pg_notify(2.17e(17) epoch 11702) -9400> 2014-09-18 18:08:23.919215 7f4c9f251700 5 -- op tracker -- , seq: 85586, time: 2014-09-18 18:08:23.919098, event: throttled, op: pg_notify(2.17e(17) epoch 11702) -9399> 2014-09-18 18:08:23.919220 7f4c9f251700 5 -- op tracker -- , seq: 85586, time: 2014-09-18 18:08:23.919140, event: all_read, op: pg_notify(2.17e(17) epoch 11702) -9398> 2014-09-18 18:08:23.919227 7f4c9f251700 5 -- op tracker -- , seq: 85586, time: 2014-09-18 18:08:23.919206, event: dispatched, op: pg_notify(2.17e(17) epoch 11702) -9397> 2014-09-18 18:08:23.919234 7f4c9f251700 5 -- op tracker -- , seq: 85586, time: 2014-09-18 18:08:23.919232, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11702) -9396> 2014-09-18 18:08:23.919244 7f4c9f251700 5 -- op tracker -- , seq: 85586, time: 2014-09-18 18:08:23.919241, event: started, op: pg_notify(2.17e(17) epoch 11702) -9395> 2014-09-18 18:08:23.919263 7f4c9f251700 5 -- op tracker -- , seq: 85586, time: 2014-09-18 18:08:23.919261, event: done, op: pg_notify(2.17e(17) epoch 11702) -9394> 2014-09-18 18:08:23.919057 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.393063 3 0.000464 -9393> 2014-09-18 18:08:23.919363 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.006732 0 0.000000 -9392> 2014-09-18 18:08:23.919430 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1180 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0xf85b440 con 0x1088b440 -9391> 2014-09-18 18:08:23.919443 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9390> 2014-09-18 18:08:23.919376 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Started/Primary 1.006759 0 0.000000 -9389> 2014-09-18 18:08:23.919467 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Started 1.006927 0 0.000000 -9388> 2014-09-18 18:08:23.919479 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11701/11701/11258) [27,42]/[42,35] r=0 lpr=11701 pi=5697-11700/152 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Reset -9387> 2014-09-18 18:08:23.919518 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000039 1 0.000461 -9386> 2014-09-18 18:08:23.919531 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started -9385> 2014-09-18 18:08:23.919539 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Start -9384> 2014-09-18 18:08:23.919545 7f4c96a40700 1 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -9383> 2014-09-18 18:08:23.919553 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000014 0 0.000000 -9382> 2014-09-18 18:08:23.919565 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -9381> 2014-09-18 18:08:23.919576 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -9380> 2014-09-18 18:08:23.919591 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -9379> 2014-09-18 18:08:23.919767 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1181 ==== pg_notify(2.2d8(173) epoch 11702) v5 ==== 10757+0+0 (2438307065 0 0) 0x100f0ee0 con 0x1088b440 -9378> 2014-09-18 18:08:23.919787 7f4c9f251700 5 -- op tracker -- , seq: 85587, time: 2014-09-18 18:08:23.919533, event: header_read, op: pg_notify(2.2d8(173) epoch 11702) -9377> 2014-09-18 18:08:23.919799 7f4c9f251700 5 -- op tracker -- , seq: 85587, time: 2014-09-18 18:08:23.919534, event: throttled, op: pg_notify(2.2d8(173) epoch 11702) -9376> 2014-09-18 18:08:23.919805 7f4c9f251700 5 -- op tracker -- , seq: 85587, time: 2014-09-18 18:08:23.919752, event: all_read, op: pg_notify(2.2d8(173) epoch 11702) -9375> 2014-09-18 18:08:23.919811 7f4c9f251700 5 -- op tracker -- , seq: 85587, time: 2014-09-18 18:08:23.919782, event: dispatched, op: pg_notify(2.2d8(173) epoch 11702) -9374> 2014-09-18 18:08:23.919818 7f4c9f251700 5 -- op tracker -- , seq: 85587, time: 2014-09-18 18:08:23.919816, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11702) -9373> 2014-09-18 18:08:23.919826 7f4c9f251700 5 -- op tracker -- , seq: 85587, time: 2014-09-18 18:08:23.919823, event: started, op: pg_notify(2.2d8(173) epoch 11702) -9372> 2014-09-18 18:08:23.919842 7f4c9f251700 5 -- op tracker -- , seq: 85587, time: 2014-09-18 18:08:23.919840, event: done, op: pg_notify(2.2d8(173) epoch 11702) -9371> 2014-09-18 18:08:23.919920 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3576 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0xf85bb00 con 0xcbecec0 -9370> 2014-09-18 18:08:23.919938 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9369> 2014-09-18 18:08:23.919944 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3577 ==== pg_notify(2.4f6(12) epoch 11702) v5 ==== 1382+0+0 (3512853777 0 0) 0xfa29a20 con 0xcbecec0 -9368> 2014-09-18 18:08:23.919955 7f4c9f251700 5 -- op tracker -- , seq: 85588, time: 2014-09-18 18:08:23.919795, event: header_read, op: pg_notify(2.4f6(12) epoch 11702) -9367> 2014-09-18 18:08:23.919968 7f4c9f251700 5 -- op tracker -- , seq: 85588, time: 2014-09-18 18:08:23.919796, event: throttled, op: pg_notify(2.4f6(12) epoch 11702) -9366> 2014-09-18 18:08:23.919973 7f4c9f251700 5 -- op tracker -- , seq: 85588, time: 2014-09-18 18:08:23.919840, event: all_read, op: pg_notify(2.4f6(12) epoch 11702) -9365> 2014-09-18 18:08:23.919978 7f4c9f251700 5 -- op tracker -- , seq: 85588, time: 2014-09-18 18:08:23.919951, event: dispatched, op: pg_notify(2.4f6(12) epoch 11702) -9364> 2014-09-18 18:08:23.919985 7f4c9f251700 5 -- op tracker -- , seq: 85588, time: 2014-09-18 18:08:23.919982, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11702) -9363> 2014-09-18 18:08:23.919992 7f4c9f251700 5 -- op tracker -- , seq: 85588, time: 2014-09-18 18:08:23.919990, event: started, op: pg_notify(2.4f6(12) epoch 11702) -9362> 2014-09-18 18:08:23.919996 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -9361> 2014-09-18 18:08:23.920003 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_alive(want up_thru 11702 have 11702) v1 -- ?+0 0x14182d20 con 0x111771e0 -9360> 2014-09-18 18:08:23.920010 7f4c9f251700 5 -- op tracker -- , seq: 85588, time: 2014-09-18 18:08:23.920008, event: done, op: pg_notify(2.4f6(12) epoch 11702) -9359> 2014-09-18 18:08:23.920090 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_notify(2.34f(73) epoch 11702) v5 -- ?+0 0x14185c00 con 0x1395f8c0 -9358> 2014-09-18 18:08:23.920156 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(10) epoch 11702) v5 -- ?+0 0xe9fef00 con 0x1395ef20 -9357> 2014-09-18 18:08:23.920261 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6884/38815 -- osd_map(11699..11702 src has 1..11702) v3 -- ?+0 0x1e282480 con 0x12b3a680 -9356> 2014-09-18 18:08:23.920278 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6884/38815 -- pg_query(2.78f epoch 11702) v3 -- ?+0 0x2b93be00 con 0x12b3a680 -9355> 2014-09-18 18:08:23.920336 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 1996 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x2db45b00 con 0xe2d9a20 -9354> 2014-09-18 18:08:23.920357 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9353> 2014-09-18 18:08:23.920339 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.78f epoch 11702) v3 -- ?+0 0x2b93a3c0 con 0x1395f8c0 -9352> 2014-09-18 18:08:23.920411 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_query(2.78f epoch 11702) v3 -- ?+0 0xf05cb20 con 0xcbecec0 -9351> 2014-09-18 18:08:23.920425 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 1997 ==== pg_notify(2.1d8(15),2.26d(9) epoch 11702) v5 ==== 2848+0+0 (4098418114 0 0) 0x2b93bfe0 con 0xe2d9a20 -9350> 2014-09-18 18:08:23.920442 7f4c9f251700 5 -- op tracker -- , seq: 85589, time: 2014-09-18 18:08:23.920334, event: header_read, op: pg_notify(2.1d8(15),2.26d(9) epoch 11702) -9349> 2014-09-18 18:08:23.920450 7f4c9f251700 5 -- op tracker -- , seq: 85589, time: 2014-09-18 18:08:23.920335, event: throttled, op: pg_notify(2.1d8(15),2.26d(9) epoch 11702) -9348> 2014-09-18 18:08:23.920456 7f4c9f251700 5 -- op tracker -- , seq: 85589, time: 2014-09-18 18:08:23.920411, event: all_read, op: pg_notify(2.1d8(15),2.26d(9) epoch 11702) -9347> 2014-09-18 18:08:23.920463 7f4c9f251700 5 -- op tracker -- , seq: 85589, time: 2014-09-18 18:08:23.920438, event: dispatched, op: pg_notify(2.1d8(15),2.26d(9) epoch 11702) -9346> 2014-09-18 18:08:23.920471 7f4c9f251700 5 -- op tracker -- , seq: 85589, time: 2014-09-18 18:08:23.920468, event: waiting_for_osdmap, op: pg_notify(2.1d8(15),2.26d(9) epoch 11702) -9345> 2014-09-18 18:08:23.920481 7f4c9f251700 5 -- op tracker -- , seq: 85589, time: 2014-09-18 18:08:23.920478, event: started, op: pg_notify(2.1d8(15),2.26d(9) epoch 11702) -9344> 2014-09-18 18:08:23.920527 7f4c9f251700 5 -- op tracker -- , seq: 85589, time: 2014-09-18 18:08:23.920523, event: done, op: pg_notify(2.1d8(15),2.26d(9) epoch 11702) -9343> 2014-09-18 18:08:23.920874 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1182 ==== pg_notify(2.1d8(9) epoch 11702) v5 ==== 1261+0+0 (784760658 0 0) 0x100f0580 con 0x1088b440 -9342> 2014-09-18 18:08:23.920894 7f4c9f251700 5 -- op tracker -- , seq: 85590, time: 2014-09-18 18:08:23.920822, event: header_read, op: pg_notify(2.1d8(9) epoch 11702) -9341> 2014-09-18 18:08:23.920905 7f4c9f251700 5 -- op tracker -- , seq: 85590, time: 2014-09-18 18:08:23.920823, event: throttled, op: pg_notify(2.1d8(9) epoch 11702) -9340> 2014-09-18 18:08:23.920914 7f4c9f251700 5 -- op tracker -- , seq: 85590, time: 2014-09-18 18:08:23.920859, event: all_read, op: pg_notify(2.1d8(9) epoch 11702) -9339> 2014-09-18 18:08:23.920922 7f4c9f251700 5 -- op tracker -- , seq: 85590, time: 2014-09-18 18:08:23.920889, event: dispatched, op: pg_notify(2.1d8(9) epoch 11702) -9338> 2014-09-18 18:08:23.920931 7f4c9f251700 5 -- op tracker -- , seq: 85590, time: 2014-09-18 18:08:23.920929, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11702) -9337> 2014-09-18 18:08:23.920943 7f4c9f251700 5 -- op tracker -- , seq: 85590, time: 2014-09-18 18:08:23.920940, event: started, op: pg_notify(2.1d8(9) epoch 11702) -9336> 2014-09-18 18:08:23.920971 7f4c9f251700 5 -- op tracker -- , seq: 85590, time: 2014-09-18 18:08:23.920969, event: done, op: pg_notify(2.1d8(9) epoch 11702) -9335> 2014-09-18 18:08:23.921416 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.37 10.242.42.168:6809/18319 8 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x1182ab40 con 0x1395eb00 -9334> 2014-09-18 18:08:23.921434 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9333> 2014-09-18 18:08:23.921453 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.37 10.242.42.168:6809/18319 9 ==== pg_info(1 pgs e11702:2.6a6(9)) v4 ==== 1205+0+0 (2975511915 0 0) 0xe1acd00 con 0x1395eb00 -9332> 2014-09-18 18:08:23.921464 7f4c9f251700 5 -- op tracker -- , seq: 85591, time: 2014-09-18 18:08:23.921414, event: header_read, op: pg_info(1 pgs e11702:2.6a6(9)) -9331> 2014-09-18 18:08:23.921473 7f4c9f251700 5 -- op tracker -- , seq: 85591, time: 2014-09-18 18:08:23.921414, event: throttled, op: pg_info(1 pgs e11702:2.6a6(9)) -9330> 2014-09-18 18:08:23.921479 7f4c9f251700 5 -- op tracker -- , seq: 85591, time: 2014-09-18 18:08:23.921444, event: all_read, op: pg_info(1 pgs e11702:2.6a6(9)) -9329> 2014-09-18 18:08:23.921485 7f4c9f251700 5 -- op tracker -- , seq: 85591, time: 2014-09-18 18:08:23.921461, event: dispatched, op: pg_info(1 pgs e11702:2.6a6(9)) -9328> 2014-09-18 18:08:23.921491 7f4c9f251700 5 -- op tracker -- , seq: 85591, time: 2014-09-18 18:08:23.921490, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.6a6(9)) -9327> 2014-09-18 18:08:23.921499 7f4c9f251700 5 -- op tracker -- , seq: 85591, time: 2014-09-18 18:08:23.921497, event: started, op: pg_info(1 pgs e11702:2.6a6(9)) -9326> 2014-09-18 18:08:23.921604 7f4c9f251700 5 -- op tracker -- , seq: 85591, time: 2014-09-18 18:08:23.921601, event: done, op: pg_info(1 pgs e11702:2.6a6(9)) -9325> 2014-09-18 18:08:23.922029 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1824 ==== pg_notify(2.517(11) epoch 11702) v5 ==== 1335+0+0 (1041981719 0 0) 0x1310e1c0 con 0x10f60520 -9324> 2014-09-18 18:08:23.922046 7f4c9f251700 5 -- op tracker -- , seq: 85592, time: 2014-09-18 18:08:23.921963, event: header_read, op: pg_notify(2.517(11) epoch 11702) -9323> 2014-09-18 18:08:23.922054 7f4c9f251700 5 -- op tracker -- , seq: 85592, time: 2014-09-18 18:08:23.921964, event: throttled, op: pg_notify(2.517(11) epoch 11702) -9322> 2014-09-18 18:08:23.922063 7f4c9f251700 5 -- op tracker -- , seq: 85592, time: 2014-09-18 18:08:23.922014, event: all_read, op: pg_notify(2.517(11) epoch 11702) -9321> 2014-09-18 18:08:23.922072 7f4c9f251700 5 -- op tracker -- , seq: 85592, time: 2014-09-18 18:08:23.922042, event: dispatched, op: pg_notify(2.517(11) epoch 11702) -9320> 2014-09-18 18:08:23.922079 7f4c9f251700 5 -- op tracker -- , seq: 85592, time: 2014-09-18 18:08:23.922077, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11702) -9319> 2014-09-18 18:08:23.922086 7f4c9f251700 5 -- op tracker -- , seq: 85592, time: 2014-09-18 18:08:23.922085, event: started, op: pg_notify(2.517(11) epoch 11702) -9318> 2014-09-18 18:08:23.922101 7f4c9f251700 5 -- op tracker -- , seq: 85592, time: 2014-09-18 18:08:23.922099, event: done, op: pg_notify(2.517(11) epoch 11702) -9317> 2014-09-18 18:08:23.922112 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1825 ==== pg_notify(2.1d8(9) epoch 11702) v5 ==== 1261+0+0 (784760658 0 0) 0x1310cf00 con 0x10f60520 -9316> 2014-09-18 18:08:23.922122 7f4c9f251700 5 -- op tracker -- , seq: 85593, time: 2014-09-18 18:08:23.922029, event: header_read, op: pg_notify(2.1d8(9) epoch 11702) -9315> 2014-09-18 18:08:23.922128 7f4c9f251700 5 -- op tracker -- , seq: 85593, time: 2014-09-18 18:08:23.922029, event: throttled, op: pg_notify(2.1d8(9) epoch 11702) -9314> 2014-09-18 18:08:23.922134 7f4c9f251700 5 -- op tracker -- , seq: 85593, time: 2014-09-18 18:08:23.922053, event: all_read, op: pg_notify(2.1d8(9) epoch 11702) -9313> 2014-09-18 18:08:23.922140 7f4c9f251700 5 -- op tracker -- , seq: 85593, time: 2014-09-18 18:08:23.922119, event: dispatched, op: pg_notify(2.1d8(9) epoch 11702) -9312> 2014-09-18 18:08:23.922147 7f4c9f251700 5 -- op tracker -- , seq: 85593, time: 2014-09-18 18:08:23.922145, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11702) -9311> 2014-09-18 18:08:23.922154 7f4c9f251700 5 -- op tracker -- , seq: 85593, time: 2014-09-18 18:08:23.922152, event: started, op: pg_notify(2.1d8(9) epoch 11702) -9310> 2014-09-18 18:08:23.922167 7f4c9f251700 5 -- op tracker -- , seq: 85593, time: 2014-09-18 18:08:23.922165, event: done, op: pg_notify(2.1d8(9) epoch 11702) -9309> 2014-09-18 18:08:23.922818 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2332 ==== pg_notify(2.457(21) epoch 11702) v5 ==== 1875+0+0 (4046176083 0 0) 0xf106000 con 0xdd8c940 -9308> 2014-09-18 18:08:23.922838 7f4c9f251700 5 -- op tracker -- , seq: 85594, time: 2014-09-18 18:08:23.922719, event: header_read, op: pg_notify(2.457(21) epoch 11702) -9307> 2014-09-18 18:08:23.922850 7f4c9f251700 5 -- op tracker -- , seq: 85594, time: 2014-09-18 18:08:23.922720, event: throttled, op: pg_notify(2.457(21) epoch 11702) -9306> 2014-09-18 18:08:23.922861 7f4c9f251700 5 -- op tracker -- , seq: 85594, time: 2014-09-18 18:08:23.922769, event: all_read, op: pg_notify(2.457(21) epoch 11702) -9305> 2014-09-18 18:08:23.922787 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 1.003832 7 0.000307 -9304> 2014-09-18 18:08:23.922871 7f4c9f251700 5 -- op tracker -- , seq: 85594, time: 2014-09-18 18:08:23.922832, event: dispatched, op: pg_notify(2.457(21) epoch 11702) -9303> 2014-09-18 18:08:23.922890 7f4c9f251700 5 -- op tracker -- , seq: 85594, time: 2014-09-18 18:08:23.922887, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11702) -9302> 2014-09-18 18:08:23.922902 7f4c9f251700 5 -- op tracker -- , seq: 85594, time: 2014-09-18 18:08:23.922899, event: started, op: pg_notify(2.457(21) epoch 11702) -9301> 2014-09-18 18:08:23.922923 7f4c9f251700 5 -- op tracker -- , seq: 85594, time: 2014-09-18 18:08:23.922921, event: done, op: pg_notify(2.457(21) epoch 11702) -9300> 2014-09-18 18:08:23.922883 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.009170 0 0.000000 -9299> 2014-09-18 18:08:23.922969 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11485 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 remapped] enter Started/Primary/Active -9298> 2014-09-18 18:08:23.923294 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11702 n=492 ec=188 les/c 11485/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 remapped] enter Started/Primary/Active/Activating -9297> 2014-09-18 18:08:23.923337 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11702..11702 src has 1..11702) v3 -- ?+0 0xe937440 con 0x23a74f20 -9296> 2014-09-18 18:08:23.923361 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11702) v5 -- ?+0 0xf05dde0 con 0x23a74f20 -9295> 2014-09-18 18:08:23.923440 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.995140 5 0.000122 -9294> 2014-09-18 18:08:23.923477 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.009390 0 0.000000 -9293> 2014-09-18 18:08:23.923495 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11696 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -9292> 2014-09-18 18:08:23.923444 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11696 n=512 ec=188 les/c 11696/10658 11701/11701/10657) [9,13,30]/[13,42] r=1 lpr=11701 pi=9952-11700/26 crt=6789'26114 remapped NOTIFY] exit Started/Stray 1.011923 7 0.000537 -9291> 2014-09-18 18:08:23.923553 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11696 n=512 ec=188 les/c 11696/10658 11701/11701/10657) [9,13,30]/[13,42] r=1 lpr=11701 pi=9952-11700/26 crt=6789'26114 remapped NOTIFY] enter Started/ReplicaActive -9290> 2014-09-18 18:08:23.923627 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11696 n=512 ec=188 les/c 11696/10658 11701/11701/10657) [9,13,30]/[13,42] r=1 lpr=11701 pi=9952-11700/26 crt=6789'26114 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -9289> 2014-09-18 18:08:23.923764 7f4c96a40700 0 log_channel(default) log [INF] : 2.4c1 restarting backfill on osd.48 from (0'0,0'0] MAX to 6761'26972 -9288> 2014-09-18 18:08:23.923936 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.29c( v 6805'31672 (4155'28673,6805'31672] local-les=11484 n=495 ec=188 les/c 11484/10658 11701/11701/11456) [11,27,0]/[11,42] r=1 lpr=11701 pi=6308-11700/79 crt=6805'31672 remapped NOTIFY] exit Started/Stray 1.013655 7 0.000895 -9287> 2014-09-18 18:08:23.923969 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.29c( v 6805'31672 (4155'28673,6805'31672] local-les=11484 n=495 ec=188 les/c 11484/10658 11701/11701/11456) [11,27,0]/[11,42] r=1 lpr=11701 pi=6308-11700/79 crt=6805'31672 remapped NOTIFY] enter Started/ReplicaActive -9286> 2014-09-18 18:08:23.923986 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.29c( v 6805'31672 (4155'28673,6805'31672] local-les=11484 n=495 ec=188 les/c 11484/10658 11701/11701/11456) [11,27,0]/[11,42] r=1 lpr=11701 pi=6308-11700/79 crt=6805'31672 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -9285> 2014-09-18 18:08:23.924151 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_info(1 pgs e11702:2.756) v4 -- ?+0 0xe797a20 con 0x140f95a0 -9284> 2014-09-18 18:08:23.924396 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6836/27583 -- pg_log(2.4c1 epoch 11702 log log((4138'23485,6761'26972], crt=6761'26972) query_epoch 11702) v4 -- ?+0 0x21faae00 con 0x10aebe40 -9283> 2014-09-18 18:08:23.924412 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.6a6( v 6775'31318 (4142'27917,6775'31318] local-les=11520 n=530 ec=188 les/c 11520/10658 11701/11701/11519) [37,9,27]/[37,42] r=1 lpr=11701 pi=9949-11700/17 crt=6775'31318 remapped NOTIFY] exit Started/Stray 1.015125 7 0.000695 -9282> 2014-09-18 18:08:23.924470 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.6a6( v 6775'31318 (4142'27917,6775'31318] local-les=11520 n=530 ec=188 les/c 11520/10658 11701/11701/11519) [37,9,27]/[37,42] r=1 lpr=11701 pi=9949-11700/17 crt=6775'31318 remapped NOTIFY] enter Started/ReplicaActive -9281> 2014-09-18 18:08:23.924482 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.6a6( v 6775'31318 (4142'27917,6775'31318] local-les=11520 n=530 ec=188 les/c 11520/10658 11701/11701/11519) [37,9,27]/[37,42] r=1 lpr=11701 pi=9949-11700/17 crt=6775'31318 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -9280> 2014-09-18 18:08:23.924563 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6804/17298 -- pg_info(1 pgs e11702:2.29c) v4 -- ?+0 0xf0b90c0 con 0x1395f340 -9279> 2014-09-18 18:08:23.924577 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11702 n=535 ec=188 les/c 11696/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -9278> 2014-09-18 18:08:23.925485 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 903 ==== pg_notify(2.410(11) epoch 11702) v5 ==== 1287+0+0 (2226169055 0 0) 0x2b93a3c0 con 0x1395f8c0 -9277> 2014-09-18 18:08:23.925504 7f4c9f251700 5 -- op tracker -- , seq: 85595, time: 2014-09-18 18:08:23.925429, event: header_read, op: pg_notify(2.410(11) epoch 11702) -9276> 2014-09-18 18:08:23.925514 7f4c9f251700 5 -- op tracker -- , seq: 85595, time: 2014-09-18 18:08:23.925430, event: throttled, op: pg_notify(2.410(11) epoch 11702) -9275> 2014-09-18 18:08:23.925520 7f4c9f251700 5 -- op tracker -- , seq: 85595, time: 2014-09-18 18:08:23.925471, event: all_read, op: pg_notify(2.410(11) epoch 11702) -9274> 2014-09-18 18:08:23.925526 7f4c9f251700 5 -- op tracker -- , seq: 85595, time: 2014-09-18 18:08:23.925500, event: dispatched, op: pg_notify(2.410(11) epoch 11702) -9273> 2014-09-18 18:08:23.925533 7f4c9f251700 5 -- op tracker -- , seq: 85595, time: 2014-09-18 18:08:23.925532, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11702) -9272> 2014-09-18 18:08:23.925541 7f4c9f251700 5 -- op tracker -- , seq: 85595, time: 2014-09-18 18:08:23.925539, event: started, op: pg_notify(2.410(11) epoch 11702) -9271> 2014-09-18 18:08:23.925560 7f4c9f251700 5 -- op tracker -- , seq: 85595, time: 2014-09-18 18:08:23.925558, event: done, op: pg_notify(2.410(11) epoch 11702) -9270> 2014-09-18 18:08:23.926952 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3578 ==== pg_notify(2.3b6(10) epoch 11702) v5 ==== 1260+0+0 (3075599441 0 0) 0xf05cb20 con 0xcbecec0 -9269> 2014-09-18 18:08:23.926982 7f4c9f251700 5 -- op tracker -- , seq: 85596, time: 2014-09-18 18:08:23.926886, event: header_read, op: pg_notify(2.3b6(10) epoch 11702) -9268> 2014-09-18 18:08:23.926993 7f4c9f251700 5 -- op tracker -- , seq: 85596, time: 2014-09-18 18:08:23.926887, event: throttled, op: pg_notify(2.3b6(10) epoch 11702) -9267> 2014-09-18 18:08:23.926999 7f4c9f251700 5 -- op tracker -- , seq: 85596, time: 2014-09-18 18:08:23.926928, event: all_read, op: pg_notify(2.3b6(10) epoch 11702) -9266> 2014-09-18 18:08:23.927006 7f4c9f251700 5 -- op tracker -- , seq: 85596, time: 2014-09-18 18:08:23.926978, event: dispatched, op: pg_notify(2.3b6(10) epoch 11702) -9265> 2014-09-18 18:08:23.927012 7f4c9f251700 5 -- op tracker -- , seq: 85596, time: 2014-09-18 18:08:23.927011, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11702) -9264> 2014-09-18 18:08:23.927020 7f4c9f251700 5 -- op tracker -- , seq: 85596, time: 2014-09-18 18:08:23.927019, event: started, op: pg_notify(2.3b6(10) epoch 11702) -9263> 2014-09-18 18:08:23.927036 7f4c9f251700 5 -- op tracker -- , seq: 85596, time: 2014-09-18 18:08:23.927035, event: done, op: pg_notify(2.3b6(10) epoch 11702) -9262> 2014-09-18 18:08:23.927472 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6809/18319 -- pg_info(1 pgs e11702:2.6a6) v4 -- ?+0 0xf0b6960 con 0x1395eb00 -9261> 2014-09-18 18:08:23.927700 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6815/37668 -- pg_info(1 pgs e11702:2.1d8(9)) v4 -- ?+0 0xf05cd00 con 0x1088b440 -9260> 2014-09-18 18:08:23.927794 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(2 pgs e11702:2.1d8(9),2.4c1(15)) v4 -- ?+0 0xf05c580 con 0x1395f8c0 -9259> 2014-09-18 18:08:23.927866 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_info(1 pgs e11702:2.1d8(9)) v4 -- ?+0 0x21e105a0 con 0x1395ef20 -9258> 2014-09-18 18:08:23.927940 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6829/19404 -- pg_info(1 pgs e11702:2.1d8(9)) v4 -- ?+0 0xe1ad2a0 con 0xe2d9a20 -9257> 2014-09-18 18:08:23.928547 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 104 ==== pg_info(1 pgs e11702:2.6e3(27)) v4 ==== 2187+0+0 (3892671691 0 0) 0xf05b4a0 con 0x1395edc0 -9256> 2014-09-18 18:08:23.928563 7f4c9f251700 5 -- op tracker -- , seq: 85597, time: 2014-09-18 18:08:23.928493, event: header_read, op: pg_info(1 pgs e11702:2.6e3(27)) -9255> 2014-09-18 18:08:23.928572 7f4c9f251700 5 -- op tracker -- , seq: 85597, time: 2014-09-18 18:08:23.928494, event: throttled, op: pg_info(1 pgs e11702:2.6e3(27)) -9254> 2014-09-18 18:08:23.928583 7f4c9f251700 5 -- op tracker -- , seq: 85597, time: 2014-09-18 18:08:23.928534, event: all_read, op: pg_info(1 pgs e11702:2.6e3(27)) -9253> 2014-09-18 18:08:23.928590 7f4c9f251700 5 -- op tracker -- , seq: 85597, time: 2014-09-18 18:08:23.928559, event: dispatched, op: pg_info(1 pgs e11702:2.6e3(27)) -9252> 2014-09-18 18:08:23.928602 7f4c9f251700 5 -- op tracker -- , seq: 85597, time: 2014-09-18 18:08:23.928595, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.6e3(27)) -9251> 2014-09-18 18:08:23.928611 7f4c9f251700 5 -- op tracker -- , seq: 85597, time: 2014-09-18 18:08:23.928610, event: started, op: pg_info(1 pgs e11702:2.6e3(27)) -9250> 2014-09-18 18:08:23.928631 7f4c9f251700 5 -- op tracker -- , seq: 85597, time: 2014-09-18 18:08:23.928627, event: done, op: pg_info(1 pgs e11702:2.6e3(27)) -9249> 2014-09-18 18:08:23.928636 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.6e3( v 6825'31592 (4125'28015,6825'31592] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11520/10731 11701/11701/11483) [20,42,30]/[30,53] r=-1 lpr=11701 pi=10730-11700/27 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started/Stray 1.017699 7 0.000311 -9248> 2014-09-18 18:08:23.928655 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.6e3( v 6825'31592 (4125'28015,6825'31592] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11520/10731 11701/11701/11483) [20,42,30]/[30,53] r=-1 lpr=11701 pi=10730-11700/27 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -9247> 2014-09-18 18:08:23.928666 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.6e3( v 6825'31592 (4125'28015,6825'31592] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11520/10731 11701/11701/11483) [20,42,30]/[30,53] r=-1 lpr=11701 pi=10730-11700/27 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -9246> 2014-09-18 18:08:23.929066 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_info(1 pgs e11702:2.6e3) v4 -- ?+0 0xf93d660 con 0x1395edc0 -9245> 2014-09-18 18:08:23.930181 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 1 ==== pg_notify(2.120(15) epoch 11702) v5 ==== 1535+0+0 (2720403280 0 0) 0x2b93be00 con 0x12b3a680 -9244> 2014-09-18 18:08:23.930196 7f4c9f251700 5 -- op tracker -- , seq: 85598, time: 2014-09-18 18:08:23.930129, event: header_read, op: pg_notify(2.120(15) epoch 11702) -9243> 2014-09-18 18:08:23.930202 7f4c9f251700 5 -- op tracker -- , seq: 85598, time: 2014-09-18 18:08:23.930130, event: throttled, op: pg_notify(2.120(15) epoch 11702) -9242> 2014-09-18 18:08:23.930206 7f4c9f251700 5 -- op tracker -- , seq: 85598, time: 2014-09-18 18:08:23.930167, event: all_read, op: pg_notify(2.120(15) epoch 11702) -9241> 2014-09-18 18:08:23.930210 7f4c9f251700 5 -- op tracker -- , seq: 85598, time: 2014-09-18 18:08:23.930192, event: dispatched, op: pg_notify(2.120(15) epoch 11702) -9240> 2014-09-18 18:08:23.930214 7f4c9f251700 5 -- op tracker -- , seq: 85598, time: 2014-09-18 18:08:23.930213, event: waiting_for_osdmap, op: pg_notify(2.120(15) epoch 11702) -9239> 2014-09-18 18:08:23.930220 7f4c9f251700 5 -- op tracker -- , seq: 85598, time: 2014-09-18 18:08:23.930218, event: started, op: pg_notify(2.120(15) epoch 11702) -9238> 2014-09-18 18:08:23.930236 7f4c9f251700 5 -- op tracker -- , seq: 85598, time: 2014-09-18 18:08:23.930235, event: done, op: pg_notify(2.120(15) epoch 11702) -9237> 2014-09-18 18:08:23.930926 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 772 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x2db46880 con 0xe9a27e0 -9236> 2014-09-18 18:08:23.930948 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9235> 2014-09-18 18:08:23.931243 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 773 ==== pg_notify(2.2d8(173) epoch 11702) v5 ==== 10757+0+0 (2438307065 0 0) 0xe0c8b20 con 0xe9a27e0 -9234> 2014-09-18 18:08:23.931259 7f4c9f251700 5 -- op tracker -- , seq: 85599, time: 2014-09-18 18:08:23.931055, event: header_read, op: pg_notify(2.2d8(173) epoch 11702) -9233> 2014-09-18 18:08:23.931270 7f4c9f251700 5 -- op tracker -- , seq: 85599, time: 2014-09-18 18:08:23.931056, event: throttled, op: pg_notify(2.2d8(173) epoch 11702) -9232> 2014-09-18 18:08:23.931280 7f4c9f251700 5 -- op tracker -- , seq: 85599, time: 2014-09-18 18:08:23.931230, event: all_read, op: pg_notify(2.2d8(173) epoch 11702) -9231> 2014-09-18 18:08:23.931290 7f4c9f251700 5 -- op tracker -- , seq: 85599, time: 2014-09-18 18:08:23.931255, event: dispatched, op: pg_notify(2.2d8(173) epoch 11702) -9230> 2014-09-18 18:08:23.931299 7f4c9f251700 5 -- op tracker -- , seq: 85599, time: 2014-09-18 18:08:23.931297, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11702) -9229> 2014-09-18 18:08:23.931310 7f4c9f251700 5 -- op tracker -- , seq: 85599, time: 2014-09-18 18:08:23.931308, event: started, op: pg_notify(2.2d8(173) epoch 11702) -9228> 2014-09-18 18:08:23.931334 7f4c9f251700 5 -- op tracker -- , seq: 85599, time: 2014-09-18 18:08:23.931331, event: done, op: pg_notify(2.2d8(173) epoch 11702) -9227> 2014-09-18 18:08:23.931745 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 904 ==== pg_notify(2.4c1(15) epoch 11702) v5 ==== 1507+0+0 (3757276282 0 0) 0xf05c580 con 0x1395f8c0 -9226> 2014-09-18 18:08:23.931761 7f4c9f251700 5 -- op tracker -- , seq: 85600, time: 2014-09-18 18:08:23.931689, event: header_read, op: pg_notify(2.4c1(15) epoch 11702) -9225> 2014-09-18 18:08:23.931771 7f4c9f251700 5 -- op tracker -- , seq: 85600, time: 2014-09-18 18:08:23.931689, event: throttled, op: pg_notify(2.4c1(15) epoch 11702) -9224> 2014-09-18 18:08:23.931778 7f4c9f251700 5 -- op tracker -- , seq: 85600, time: 2014-09-18 18:08:23.931732, event: all_read, op: pg_notify(2.4c1(15) epoch 11702) -9223> 2014-09-18 18:08:23.931783 7f4c9f251700 5 -- op tracker -- , seq: 85600, time: 2014-09-18 18:08:23.931758, event: dispatched, op: pg_notify(2.4c1(15) epoch 11702) -9222> 2014-09-18 18:08:23.931790 7f4c9f251700 5 -- op tracker -- , seq: 85600, time: 2014-09-18 18:08:23.931788, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11702) -9221> 2014-09-18 18:08:23.931798 7f4c9f251700 5 -- op tracker -- , seq: 85600, time: 2014-09-18 18:08:23.931796, event: started, op: pg_notify(2.4c1(15) epoch 11702) -9220> 2014-09-18 18:08:23.931815 7f4c9f251700 5 -- op tracker -- , seq: 85600, time: 2014-09-18 18:08:23.931813, event: done, op: pg_notify(2.4c1(15) epoch 11702) -9219> 2014-09-18 18:08:23.933349 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 2 ==== pg_notify(2.40a(162) epoch 11702) v5 ==== 9736+0+0 (2801268882 0 0) 0x128cd480 con 0x12b3a680 -9218> 2014-09-18 18:08:23.933363 7f4c9f251700 5 -- op tracker -- , seq: 85601, time: 2014-09-18 18:08:23.933226, event: header_read, op: pg_notify(2.40a(162) epoch 11702) -9217> 2014-09-18 18:08:23.933371 7f4c9f251700 5 -- op tracker -- , seq: 85601, time: 2014-09-18 18:08:23.933227, event: throttled, op: pg_notify(2.40a(162) epoch 11702) -9216> 2014-09-18 18:08:23.933377 7f4c9f251700 5 -- op tracker -- , seq: 85601, time: 2014-09-18 18:08:23.933336, event: all_read, op: pg_notify(2.40a(162) epoch 11702) -9215> 2014-09-18 18:08:23.933383 7f4c9f251700 5 -- op tracker -- , seq: 85601, time: 2014-09-18 18:08:23.933360, event: dispatched, op: pg_notify(2.40a(162) epoch 11702) -9214> 2014-09-18 18:08:23.933390 7f4c9f251700 5 -- op tracker -- , seq: 85601, time: 2014-09-18 18:08:23.933388, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11702) -9213> 2014-09-18 18:08:23.933397 7f4c9f251700 5 -- op tracker -- , seq: 85601, time: 2014-09-18 18:08:23.933396, event: started, op: pg_notify(2.40a(162) epoch 11702) -9212> 2014-09-18 18:08:23.933414 7f4c9f251700 5 -- op tracker -- , seq: 85601, time: 2014-09-18 18:08:23.933413, event: done, op: pg_notify(2.40a(162) epoch 11702) -9211> 2014-09-18 18:08:23.933876 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 905 ==== pg_info(1 pgs e11702:2.34f(22)) v4 ==== 1936+0+0 (1049526909 0 0) 0x14185c00 con 0x1395f8c0 -9210> 2014-09-18 18:08:23.933890 7f4c9f251700 5 -- op tracker -- , seq: 85602, time: 2014-09-18 18:08:23.933793, event: header_read, op: pg_info(1 pgs e11702:2.34f(22)) -9209> 2014-09-18 18:08:23.933898 7f4c9f251700 5 -- op tracker -- , seq: 85602, time: 2014-09-18 18:08:23.933794, event: throttled, op: pg_info(1 pgs e11702:2.34f(22)) -9208> 2014-09-18 18:08:23.933904 7f4c9f251700 5 -- op tracker -- , seq: 85602, time: 2014-09-18 18:08:23.933846, event: all_read, op: pg_info(1 pgs e11702:2.34f(22)) -9207> 2014-09-18 18:08:23.933911 7f4c9f251700 5 -- op tracker -- , seq: 85602, time: 2014-09-18 18:08:23.933886, event: dispatched, op: pg_info(1 pgs e11702:2.34f(22)) -9206> 2014-09-18 18:08:23.933918 7f4c9f251700 5 -- op tracker -- , seq: 85602, time: 2014-09-18 18:08:23.933916, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.34f(22)) -9205> 2014-09-18 18:08:23.933926 7f4c9f251700 5 -- op tracker -- , seq: 85602, time: 2014-09-18 18:08:23.933924, event: started, op: pg_info(1 pgs e11702:2.34f(22)) -9204> 2014-09-18 18:08:23.933943 7f4c9f251700 5 -- op tracker -- , seq: 85602, time: 2014-09-18 18:08:23.933941, event: done, op: pg_info(1 pgs e11702:2.34f(22)) -9203> 2014-09-18 18:08:23.933951 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.34f( v 6840'31042 (4157'28041,6840'31042] local-les=11430 n=520 ec=188 les/c 11430/10731 11701/11701/11406) [3,60,30]/[27,42] r=1 lpr=11701 pi=1300-11700/73 crt=6840'31035 lcod 0'0 remapped NOTIFY] exit Started/Stray 1.021786 7 0.000849 -9202> 2014-09-18 18:08:23.933968 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.34f( v 6840'31042 (4157'28041,6840'31042] local-les=11430 n=520 ec=188 les/c 11430/10731 11701/11701/11406) [3,60,30]/[27,42] r=1 lpr=11701 pi=1300-11700/73 crt=6840'31035 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -9201> 2014-09-18 18:08:23.933979 7f4c97241700 5 osd.42 pg_epoch: 11702 pg[2.34f( v 6840'31042 (4157'28041,6840'31042] local-les=11430 n=520 ec=188 les/c 11430/10731 11701/11701/11406) [3,60,30]/[27,42] r=1 lpr=11701 pi=1300-11700/73 crt=6840'31035 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -9200> 2014-09-18 18:08:23.934468 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11702:2.34f) v4 -- ?+0 0x128cd0c0 con 0x1395f8c0 -9199> 2014-09-18 18:08:23.935100 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3579 ==== pg_notify(2.517(11) epoch 11702) v5 ==== 1343+0+0 (3384679394 0 0) 0xfa28940 con 0xcbecec0 -9198> 2014-09-18 18:08:23.935120 7f4c9f251700 5 -- op tracker -- , seq: 85603, time: 2014-09-18 18:08:23.935042, event: header_read, op: pg_notify(2.517(11) epoch 11702) -9197> 2014-09-18 18:08:23.935129 7f4c9f251700 5 -- op tracker -- , seq: 85603, time: 2014-09-18 18:08:23.935043, event: throttled, op: pg_notify(2.517(11) epoch 11702) -9196> 2014-09-18 18:08:23.935135 7f4c9f251700 5 -- op tracker -- , seq: 85603, time: 2014-09-18 18:08:23.935075, event: all_read, op: pg_notify(2.517(11) epoch 11702) -9195> 2014-09-18 18:08:23.935141 7f4c9f251700 5 -- op tracker -- , seq: 85603, time: 2014-09-18 18:08:23.935115, event: dispatched, op: pg_notify(2.517(11) epoch 11702) -9194> 2014-09-18 18:08:23.935148 7f4c9f251700 5 -- op tracker -- , seq: 85603, time: 2014-09-18 18:08:23.935146, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11702) -9193> 2014-09-18 18:08:23.935155 7f4c9f251700 5 -- op tracker -- , seq: 85603, time: 2014-09-18 18:08:23.935153, event: started, op: pg_notify(2.517(11) epoch 11702) -9192> 2014-09-18 18:08:23.935190 7f4c9f251700 5 -- op tracker -- , seq: 85603, time: 2014-09-18 18:08:23.935189, event: done, op: pg_notify(2.517(11) epoch 11702) -9191> 2014-09-18 18:08:23.937108 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3580 ==== pg_notify(2.2c7(139) epoch 11702) v5 ==== 8651+0+0 (720768793 0 0) 0xfa27860 con 0xcbecec0 -9190> 2014-09-18 18:08:23.937128 7f4c9f251700 5 -- op tracker -- , seq: 85604, time: 2014-09-18 18:08:23.936946, event: header_read, op: pg_notify(2.2c7(139) epoch 11702) -9189> 2014-09-18 18:08:23.937146 7f4c9f251700 5 -- op tracker -- , seq: 85604, time: 2014-09-18 18:08:23.936947, event: throttled, op: pg_notify(2.2c7(139) epoch 11702) -9188> 2014-09-18 18:08:23.937152 7f4c9f251700 5 -- op tracker -- , seq: 85604, time: 2014-09-18 18:08:23.937095, event: all_read, op: pg_notify(2.2c7(139) epoch 11702) -9187> 2014-09-18 18:08:23.937158 7f4c9f251700 5 -- op tracker -- , seq: 85604, time: 2014-09-18 18:08:23.937123, event: dispatched, op: pg_notify(2.2c7(139) epoch 11702) -9186> 2014-09-18 18:08:23.937165 7f4c9f251700 5 -- op tracker -- , seq: 85604, time: 2014-09-18 18:08:23.937163, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11702) -9185> 2014-09-18 18:08:23.937172 7f4c9f251700 5 -- op tracker -- , seq: 85604, time: 2014-09-18 18:08:23.937171, event: started, op: pg_notify(2.2c7(139) epoch 11702) -9184> 2014-09-18 18:08:23.937190 7f4c9f251700 5 -- op tracker -- , seq: 85604, time: 2014-09-18 18:08:23.937188, event: done, op: pg_notify(2.2c7(139) epoch 11702) -9183> 2014-09-18 18:08:23.939645 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3581 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11702) v5 ==== 6174+0+0 (980528174 0 0) 0xfa281c0 con 0xcbecec0 -9182> 2014-09-18 18:08:23.939665 7f4c9f251700 5 -- op tracker -- , seq: 85605, time: 2014-09-18 18:08:23.939547, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11702) -9181> 2014-09-18 18:08:23.939696 7f4c9f251700 5 -- op tracker -- , seq: 85605, time: 2014-09-18 18:08:23.939548, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11702) -9180> 2014-09-18 18:08:23.939703 7f4c9f251700 5 -- op tracker -- , seq: 85605, time: 2014-09-18 18:08:23.939633, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11702) -9179> 2014-09-18 18:08:23.939709 7f4c9f251700 5 -- op tracker -- , seq: 85605, time: 2014-09-18 18:08:23.939661, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11702) -9178> 2014-09-18 18:08:23.939717 7f4c9f251700 5 -- op tracker -- , seq: 85605, time: 2014-09-18 18:08:23.939715, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11702) -9177> 2014-09-18 18:08:23.939725 7f4c9f251700 5 -- op tracker -- , seq: 85605, time: 2014-09-18 18:08:23.939723, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11702) -9176> 2014-09-18 18:08:23.939771 7f4c9f251700 5 -- op tracker -- , seq: 85605, time: 2014-09-18 18:08:23.939769, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11702) -9175> 2014-09-18 18:08:23.940045 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3582 ==== pg_notify(2.40a(162) epoch 11702) v5 ==== 9736+0+0 (1486072244 0 0) 0xf274b20 con 0xcbecec0 -9174> 2014-09-18 18:08:23.940061 7f4c9f251700 5 -- op tracker -- , seq: 85606, time: 2014-09-18 18:08:23.939884, event: header_read, op: pg_notify(2.40a(162) epoch 11702) -9173> 2014-09-18 18:08:23.940070 7f4c9f251700 5 -- op tracker -- , seq: 85606, time: 2014-09-18 18:08:23.939885, event: throttled, op: pg_notify(2.40a(162) epoch 11702) -9172> 2014-09-18 18:08:23.940076 7f4c9f251700 5 -- op tracker -- , seq: 85606, time: 2014-09-18 18:08:23.940030, event: all_read, op: pg_notify(2.40a(162) epoch 11702) -9171> 2014-09-18 18:08:23.940082 7f4c9f251700 5 -- op tracker -- , seq: 85606, time: 2014-09-18 18:08:23.940057, event: dispatched, op: pg_notify(2.40a(162) epoch 11702) -9170> 2014-09-18 18:08:23.940172 7f4c9f251700 5 -- op tracker -- , seq: 85606, time: 2014-09-18 18:08:23.940170, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11702) -9169> 2014-09-18 18:08:23.940181 7f4c9f251700 5 -- op tracker -- , seq: 85606, time: 2014-09-18 18:08:23.940179, event: started, op: pg_notify(2.40a(162) epoch 11702) -9168> 2014-09-18 18:08:23.940198 7f4c9f251700 5 -- op tracker -- , seq: 85606, time: 2014-09-18 18:08:23.940197, event: done, op: pg_notify(2.40a(162) epoch 11702) -9167> 2014-09-18 18:08:23.941335 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3583 ==== pg_notify(2.1bb(9) epoch 11702) v5 ==== 1189+0+0 (2928635633 0 0) 0xf275660 con 0xcbecec0 -9166> 2014-09-18 18:08:23.941359 7f4c9f251700 5 -- op tracker -- , seq: 85607, time: 2014-09-18 18:08:23.941292, event: header_read, op: pg_notify(2.1bb(9) epoch 11702) -9165> 2014-09-18 18:08:23.941374 7f4c9f251700 5 -- op tracker -- , seq: 85607, time: 2014-09-18 18:08:23.941292, event: throttled, op: pg_notify(2.1bb(9) epoch 11702) -9164> 2014-09-18 18:08:23.941384 7f4c9f251700 5 -- op tracker -- , seq: 85607, time: 2014-09-18 18:08:23.941320, event: all_read, op: pg_notify(2.1bb(9) epoch 11702) -9163> 2014-09-18 18:08:23.941393 7f4c9f251700 5 -- op tracker -- , seq: 85607, time: 2014-09-18 18:08:23.941353, event: dispatched, op: pg_notify(2.1bb(9) epoch 11702) -9162> 2014-09-18 18:08:23.941401 7f4c9f251700 5 -- op tracker -- , seq: 85607, time: 2014-09-18 18:08:23.941399, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11702) -9161> 2014-09-18 18:08:23.941409 7f4c9f251700 5 -- op tracker -- , seq: 85607, time: 2014-09-18 18:08:23.941407, event: started, op: pg_notify(2.1bb(9) epoch 11702) -9160> 2014-09-18 18:08:23.941427 7f4c9f251700 5 -- op tracker -- , seq: 85607, time: 2014-09-18 18:08:23.941426, event: done, op: pg_notify(2.1bb(9) epoch 11702) -9159> 2014-09-18 18:08:23.944798 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1183 ==== pg_info(1 pgs e11702:2.1d8) v4 ==== 742+0+0 (1646841367 0 0) 0xf05cd00 con 0x1088b440 -9158> 2014-09-18 18:08:23.944818 7f4c9f251700 5 -- op tracker -- , seq: 85608, time: 2014-09-18 18:08:23.944673, event: header_read, op: pg_info(1 pgs e11702:2.1d8) -9157> 2014-09-18 18:08:23.944828 7f4c9f251700 5 -- op tracker -- , seq: 85608, time: 2014-09-18 18:08:23.944674, event: throttled, op: pg_info(1 pgs e11702:2.1d8) -9156> 2014-09-18 18:08:23.944834 7f4c9f251700 5 -- op tracker -- , seq: 85608, time: 2014-09-18 18:08:23.944720, event: all_read, op: pg_info(1 pgs e11702:2.1d8) -9155> 2014-09-18 18:08:23.944840 7f4c9f251700 5 -- op tracker -- , seq: 85608, time: 2014-09-18 18:08:23.944813, event: dispatched, op: pg_info(1 pgs e11702:2.1d8) -9154> 2014-09-18 18:08:23.944846 7f4c9f251700 5 -- op tracker -- , seq: 85608, time: 2014-09-18 18:08:23.944845, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.1d8) -9153> 2014-09-18 18:08:23.944854 7f4c9f251700 5 -- op tracker -- , seq: 85608, time: 2014-09-18 18:08:23.944853, event: started, op: pg_info(1 pgs e11702:2.1d8) -9152> 2014-09-18 18:08:23.944881 7f4c9f251700 5 -- op tracker -- , seq: 85608, time: 2014-09-18 18:08:23.944879, event: done, op: pg_info(1 pgs e11702:2.1d8) -9151> 2014-09-18 18:08:23.946797 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 906 ==== pg_notify(2.78f(153) epoch 11702) v5 ==== 9229+0+0 (1305360145 0 0) 0x128cd0c0 con 0x1395f8c0 -9150> 2014-09-18 18:08:23.946811 7f4c9f251700 5 -- op tracker -- , seq: 85609, time: 2014-09-18 18:08:23.946664, event: header_read, op: pg_notify(2.78f(153) epoch 11702) -9149> 2014-09-18 18:08:23.946819 7f4c9f251700 5 -- op tracker -- , seq: 85609, time: 2014-09-18 18:08:23.946664, event: throttled, op: pg_notify(2.78f(153) epoch 11702) -9148> 2014-09-18 18:08:23.946825 7f4c9f251700 5 -- op tracker -- , seq: 85609, time: 2014-09-18 18:08:23.946785, event: all_read, op: pg_notify(2.78f(153) epoch 11702) -9147> 2014-09-18 18:08:23.946831 7f4c9f251700 5 -- op tracker -- , seq: 85609, time: 2014-09-18 18:08:23.946808, event: dispatched, op: pg_notify(2.78f(153) epoch 11702) -9146> 2014-09-18 18:08:23.946847 7f4c9f251700 5 -- op tracker -- , seq: 85609, time: 2014-09-18 18:08:23.946845, event: waiting_for_osdmap, op: pg_notify(2.78f(153) epoch 11702) -9145> 2014-09-18 18:08:23.946854 7f4c9f251700 5 -- op tracker -- , seq: 85609, time: 2014-09-18 18:08:23.946853, event: started, op: pg_notify(2.78f(153) epoch 11702) -9144> 2014-09-18 18:08:23.946871 7f4c9f251700 5 -- op tracker -- , seq: 85609, time: 2014-09-18 18:08:23.946870, event: done, op: pg_notify(2.78f(153) epoch 11702) -9143> 2014-09-18 18:08:23.948019 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 1998 ==== pg_info(1 pgs e11702:2.1d8) v4 ==== 742+0+0 (637055677 0 0) 0xe1ad2a0 con 0xe2d9a20 -9142> 2014-09-18 18:08:23.948033 7f4c9f251700 5 -- op tracker -- , seq: 85610, time: 2014-09-18 18:08:23.947971, event: header_read, op: pg_info(1 pgs e11702:2.1d8) -9141> 2014-09-18 18:08:23.948041 7f4c9f251700 5 -- op tracker -- , seq: 85610, time: 2014-09-18 18:08:23.947972, event: throttled, op: pg_info(1 pgs e11702:2.1d8) -9140> 2014-09-18 18:08:23.948047 7f4c9f251700 5 -- op tracker -- , seq: 85610, time: 2014-09-18 18:08:23.948006, event: all_read, op: pg_info(1 pgs e11702:2.1d8) -9139> 2014-09-18 18:08:23.948053 7f4c9f251700 5 -- op tracker -- , seq: 85610, time: 2014-09-18 18:08:23.948030, event: dispatched, op: pg_info(1 pgs e11702:2.1d8) -9138> 2014-09-18 18:08:23.948059 7f4c9f251700 5 -- op tracker -- , seq: 85610, time: 2014-09-18 18:08:23.948058, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.1d8) -9137> 2014-09-18 18:08:23.948065 7f4c9f251700 5 -- op tracker -- , seq: 85610, time: 2014-09-18 18:08:23.948064, event: started, op: pg_info(1 pgs e11702:2.1d8) -9136> 2014-09-18 18:08:23.948079 7f4c9f251700 5 -- op tracker -- , seq: 85610, time: 2014-09-18 18:08:23.948077, event: done, op: pg_info(1 pgs e11702:2.1d8) -9135> 2014-09-18 18:08:23.952119 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 907 ==== pg_info(1 pgs e11702:2.1d8) v4 ==== 742+0+0 (1646841367 0 0) 0xf186780 con 0x1395f8c0 -9134> 2014-09-18 18:08:23.952132 7f4c9f251700 5 -- op tracker -- , seq: 85611, time: 2014-09-18 18:08:23.952003, event: header_read, op: pg_info(1 pgs e11702:2.1d8) -9133> 2014-09-18 18:08:23.952140 7f4c9f251700 5 -- op tracker -- , seq: 85611, time: 2014-09-18 18:08:23.952004, event: throttled, op: pg_info(1 pgs e11702:2.1d8) -9132> 2014-09-18 18:08:23.952146 7f4c9f251700 5 -- op tracker -- , seq: 85611, time: 2014-09-18 18:08:23.952025, event: all_read, op: pg_info(1 pgs e11702:2.1d8) -9131> 2014-09-18 18:08:23.952152 7f4c9f251700 5 -- op tracker -- , seq: 85611, time: 2014-09-18 18:08:23.952129, event: dispatched, op: pg_info(1 pgs e11702:2.1d8) -9130> 2014-09-18 18:08:23.952158 7f4c9f251700 5 -- op tracker -- , seq: 85611, time: 2014-09-18 18:08:23.952157, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.1d8) -9129> 2014-09-18 18:08:23.952166 7f4c9f251700 5 -- op tracker -- , seq: 85611, time: 2014-09-18 18:08:23.952164, event: started, op: pg_info(1 pgs e11702:2.1d8) -9128> 2014-09-18 18:08:23.952181 7f4c9f251700 5 -- op tracker -- , seq: 85611, time: 2014-09-18 18:08:23.952180, event: done, op: pg_info(1 pgs e11702:2.1d8) -9127> 2014-09-18 18:08:23.952552 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 908 ==== pg_info(1 pgs e11702:2.4c1) v4 ==== 734+0+0 (249665075 0 0) 0xf5772c0 con 0x1395f8c0 -9126> 2014-09-18 18:08:23.952564 7f4c9f251700 5 -- op tracker -- , seq: 85612, time: 2014-09-18 18:08:23.952495, event: header_read, op: pg_info(1 pgs e11702:2.4c1) -9125> 2014-09-18 18:08:23.952572 7f4c9f251700 5 -- op tracker -- , seq: 85612, time: 2014-09-18 18:08:23.952496, event: throttled, op: pg_info(1 pgs e11702:2.4c1) -9124> 2014-09-18 18:08:23.952577 7f4c9f251700 5 -- op tracker -- , seq: 85612, time: 2014-09-18 18:08:23.952518, event: all_read, op: pg_info(1 pgs e11702:2.4c1) -9123> 2014-09-18 18:08:23.952583 7f4c9f251700 5 -- op tracker -- , seq: 85612, time: 2014-09-18 18:08:23.952561, event: dispatched, op: pg_info(1 pgs e11702:2.4c1) -9122> 2014-09-18 18:08:23.952589 7f4c9f251700 5 -- op tracker -- , seq: 85612, time: 2014-09-18 18:08:23.952588, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.4c1) -9121> 2014-09-18 18:08:23.952596 7f4c9f251700 5 -- op tracker -- , seq: 85612, time: 2014-09-18 18:08:23.952594, event: started, op: pg_info(1 pgs e11702:2.4c1) -9120> 2014-09-18 18:08:23.952611 7f4c9f251700 5 -- op tracker -- , seq: 85612, time: 2014-09-18 18:08:23.952609, event: done, op: pg_info(1 pgs e11702:2.4c1) -9119> 2014-09-18 18:08:23.968263 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 3 ==== pg_notify(2.1a2(20) epoch 11702) v5 ==== 1798+0+0 (18077563 0 0) 0xe7954a0 con 0x12b3a680 -9118> 2014-09-18 18:08:23.968281 7f4c9f251700 5 -- op tracker -- , seq: 85613, time: 2014-09-18 18:08:23.968198, event: header_read, op: pg_notify(2.1a2(20) epoch 11702) -9117> 2014-09-18 18:08:23.968299 7f4c9f251700 5 -- op tracker -- , seq: 85613, time: 2014-09-18 18:08:23.968200, event: throttled, op: pg_notify(2.1a2(20) epoch 11702) -9116> 2014-09-18 18:08:23.968306 7f4c9f251700 5 -- op tracker -- , seq: 85613, time: 2014-09-18 18:08:23.968249, event: all_read, op: pg_notify(2.1a2(20) epoch 11702) -9115> 2014-09-18 18:08:23.968313 7f4c9f251700 5 -- op tracker -- , seq: 85613, time: 2014-09-18 18:08:23.968277, event: dispatched, op: pg_notify(2.1a2(20) epoch 11702) -9114> 2014-09-18 18:08:23.968319 7f4c9f251700 5 -- op tracker -- , seq: 85613, time: 2014-09-18 18:08:23.968318, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11702) -9113> 2014-09-18 18:08:23.968326 7f4c9f251700 5 -- op tracker -- , seq: 85613, time: 2014-09-18 18:08:23.968324, event: started, op: pg_notify(2.1a2(20) epoch 11702) -9112> 2014-09-18 18:08:23.968345 7f4c9f251700 5 -- op tracker -- , seq: 85613, time: 2014-09-18 18:08:23.968343, event: done, op: pg_notify(2.1a2(20) epoch 11702) -9111> 2014-09-18 18:08:23.972721 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3132 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x2db44fc0 con 0x1395ec60 -9110> 2014-09-18 18:08:23.972741 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9109> 2014-09-18 18:08:23.972802 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3133 ==== pg_notify(2.71(19) epoch 11702) v5 ==== 1795+0+0 (3046958991 0 0) 0xe1aaf00 con 0x1395ec60 -9108> 2014-09-18 18:08:23.972836 7f4c9f251700 5 -- op tracker -- , seq: 85614, time: 2014-09-18 18:08:23.972735, event: header_read, op: pg_notify(2.71(19) epoch 11702) -9107> 2014-09-18 18:08:23.972845 7f4c9f251700 5 -- op tracker -- , seq: 85614, time: 2014-09-18 18:08:23.972737, event: throttled, op: pg_notify(2.71(19) epoch 11702) -9106> 2014-09-18 18:08:23.972850 7f4c9f251700 5 -- op tracker -- , seq: 85614, time: 2014-09-18 18:08:23.972784, event: all_read, op: pg_notify(2.71(19) epoch 11702) -9105> 2014-09-18 18:08:23.972854 7f4c9f251700 5 -- op tracker -- , seq: 85614, time: 2014-09-18 18:08:23.972818, event: dispatched, op: pg_notify(2.71(19) epoch 11702) -9104> 2014-09-18 18:08:23.972860 7f4c9f251700 5 -- op tracker -- , seq: 85614, time: 2014-09-18 18:08:23.972859, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11702) -9103> 2014-09-18 18:08:23.972874 7f4c9f251700 5 -- op tracker -- , seq: 85614, time: 2014-09-18 18:08:23.972873, event: started, op: pg_notify(2.71(19) epoch 11702) -9102> 2014-09-18 18:08:23.972896 7f4c9f251700 5 -- op tracker -- , seq: 85614, time: 2014-09-18 18:08:23.972894, event: done, op: pg_notify(2.71(19) epoch 11702) -9101> 2014-09-18 18:08:23.975058 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3134 ==== pg_query(2.3b5 epoch 11702) v3 ==== 144+0+0 (1708007743 0 0) 0xf974760 con 0x1395ec60 -9100> 2014-09-18 18:08:23.975071 7f4c9f251700 5 -- op tracker -- , seq: 85615, time: 2014-09-18 18:08:23.974937, event: header_read, op: pg_query(2.3b5 epoch 11702) -9099> 2014-09-18 18:08:23.975077 7f4c9f251700 5 -- op tracker -- , seq: 85615, time: 2014-09-18 18:08:23.974938, event: throttled, op: pg_query(2.3b5 epoch 11702) -9098> 2014-09-18 18:08:23.975081 7f4c9f251700 5 -- op tracker -- , seq: 85615, time: 2014-09-18 18:08:23.974962, event: all_read, op: pg_query(2.3b5 epoch 11702) -9097> 2014-09-18 18:08:23.975085 7f4c9f251700 5 -- op tracker -- , seq: 85615, time: 2014-09-18 18:08:23.975068, event: dispatched, op: pg_query(2.3b5 epoch 11702) -9096> 2014-09-18 18:08:23.975089 7f4c9f251700 5 -- op tracker -- , seq: 85615, time: 2014-09-18 18:08:23.975088, event: waiting_for_osdmap, op: pg_query(2.3b5 epoch 11702) -9095> 2014-09-18 18:08:23.975094 7f4c9f251700 5 -- op tracker -- , seq: 85615, time: 2014-09-18 18:08:23.975093, event: started, op: pg_query(2.3b5 epoch 11702) -9094> 2014-09-18 18:08:23.975152 7f4c9f251700 5 -- op tracker -- , seq: 85615, time: 2014-09-18 18:08:23.975151, event: done, op: pg_query(2.3b5 epoch 11702) -9093> 2014-09-18 18:08:23.975340 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.3b5(65) epoch 11702) v5 -- ?+0 0xe796d00 con 0x1395ec60 -9092> 2014-09-18 18:08:23.976160 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 105 ==== pg_info(1 pgs e11702:2.6e3) v4 ==== 738+0+0 (1092519413 0 0) 0xf93d660 con 0x1395edc0 -9091> 2014-09-18 18:08:23.976176 7f4c9f251700 5 -- op tracker -- , seq: 85616, time: 2014-09-18 18:08:23.976118, event: header_read, op: pg_info(1 pgs e11702:2.6e3) -9090> 2014-09-18 18:08:23.976187 7f4c9f251700 5 -- op tracker -- , seq: 85616, time: 2014-09-18 18:08:23.976119, event: throttled, op: pg_info(1 pgs e11702:2.6e3) -9089> 2014-09-18 18:08:23.976193 7f4c9f251700 5 -- op tracker -- , seq: 85616, time: 2014-09-18 18:08:23.976145, event: all_read, op: pg_info(1 pgs e11702:2.6e3) -9088> 2014-09-18 18:08:23.976199 7f4c9f251700 5 -- op tracker -- , seq: 85616, time: 2014-09-18 18:08:23.976172, event: dispatched, op: pg_info(1 pgs e11702:2.6e3) -9087> 2014-09-18 18:08:23.976205 7f4c9f251700 5 -- op tracker -- , seq: 85616, time: 2014-09-18 18:08:23.976204, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.6e3) -9086> 2014-09-18 18:08:23.976213 7f4c9f251700 5 -- op tracker -- , seq: 85616, time: 2014-09-18 18:08:23.976211, event: started, op: pg_info(1 pgs e11702:2.6e3) -9085> 2014-09-18 18:08:23.976230 7f4c9f251700 5 -- op tracker -- , seq: 85616, time: 2014-09-18 18:08:23.976229, event: done, op: pg_info(1 pgs e11702:2.6e3) -9084> 2014-09-18 18:08:23.985731 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3135 ==== pg_notify(2.410(11) epoch 11702) v5 ==== 1325+0+0 (3456934353 0 0) 0xe796d00 con 0x1395ec60 -9083> 2014-09-18 18:08:23.985751 7f4c9f251700 5 -- op tracker -- , seq: 85617, time: 2014-09-18 18:08:23.985684, event: header_read, op: pg_notify(2.410(11) epoch 11702) -9082> 2014-09-18 18:08:23.985762 7f4c9f251700 5 -- op tracker -- , seq: 85617, time: 2014-09-18 18:08:23.985685, event: throttled, op: pg_notify(2.410(11) epoch 11702) -9081> 2014-09-18 18:08:23.985778 7f4c9f251700 5 -- op tracker -- , seq: 85617, time: 2014-09-18 18:08:23.985715, event: all_read, op: pg_notify(2.410(11) epoch 11702) -9080> 2014-09-18 18:08:23.985784 7f4c9f251700 5 -- op tracker -- , seq: 85617, time: 2014-09-18 18:08:23.985746, event: dispatched, op: pg_notify(2.410(11) epoch 11702) -9079> 2014-09-18 18:08:23.985791 7f4c9f251700 5 -- op tracker -- , seq: 85617, time: 2014-09-18 18:08:23.985789, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11702) -9078> 2014-09-18 18:08:23.985799 7f4c9f251700 5 -- op tracker -- , seq: 85617, time: 2014-09-18 18:08:23.985797, event: started, op: pg_notify(2.410(11) epoch 11702) -9077> 2014-09-18 18:08:23.985819 7f4c9f251700 5 -- op tracker -- , seq: 85617, time: 2014-09-18 18:08:23.985817, event: done, op: pg_notify(2.410(11) epoch 11702) -9076> 2014-09-18 18:08:23.985938 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3136 ==== pg_notify(2.186(12) epoch 11702) v5 ==== 1386+0+0 (1581473570 0 0) 0xf9734a0 con 0x1395ec60 -9075> 2014-09-18 18:08:23.985951 7f4c9f251700 5 -- op tracker -- , seq: 85618, time: 2014-09-18 18:08:23.985898, event: header_read, op: pg_notify(2.186(12) epoch 11702) -9074> 2014-09-18 18:08:23.985962 7f4c9f251700 5 -- op tracker -- , seq: 85618, time: 2014-09-18 18:08:23.985899, event: throttled, op: pg_notify(2.186(12) epoch 11702) -9073> 2014-09-18 18:08:23.985972 7f4c9f251700 5 -- op tracker -- , seq: 85618, time: 2014-09-18 18:08:23.985927, event: all_read, op: pg_notify(2.186(12) epoch 11702) -9072> 2014-09-18 18:08:23.985980 7f4c9f251700 5 -- op tracker -- , seq: 85618, time: 2014-09-18 18:08:23.985948, event: dispatched, op: pg_notify(2.186(12) epoch 11702) -9071> 2014-09-18 18:08:23.985988 7f4c9f251700 5 -- op tracker -- , seq: 85618, time: 2014-09-18 18:08:23.985986, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11702) -9070> 2014-09-18 18:08:23.985995 7f4c9f251700 5 -- op tracker -- , seq: 85618, time: 2014-09-18 18:08:23.985994, event: started, op: pg_notify(2.186(12) epoch 11702) -9069> 2014-09-18 18:08:23.986012 7f4c9f251700 5 -- op tracker -- , seq: 85618, time: 2014-09-18 18:08:23.986011, event: done, op: pg_notify(2.186(12) epoch 11702) -9068> 2014-09-18 18:08:23.986908 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3137 ==== pg_notify(2.74e(11) epoch 11702) v5 ==== 1303+0+0 (1992192668 0 0) 0xe2870e0 con 0x1395ec60 -9067> 2014-09-18 18:08:23.986927 7f4c9f251700 5 -- op tracker -- , seq: 85619, time: 2014-09-18 18:08:23.986823, event: header_read, op: pg_notify(2.74e(11) epoch 11702) -9066> 2014-09-18 18:08:23.986938 7f4c9f251700 5 -- op tracker -- , seq: 85619, time: 2014-09-18 18:08:23.986824, event: throttled, op: pg_notify(2.74e(11) epoch 11702) -9065> 2014-09-18 18:08:23.986944 7f4c9f251700 5 -- op tracker -- , seq: 85619, time: 2014-09-18 18:08:23.986864, event: all_read, op: pg_notify(2.74e(11) epoch 11702) -9064> 2014-09-18 18:08:23.986949 7f4c9f251700 5 -- op tracker -- , seq: 85619, time: 2014-09-18 18:08:23.986923, event: dispatched, op: pg_notify(2.74e(11) epoch 11702) -9063> 2014-09-18 18:08:23.986956 7f4c9f251700 5 -- op tracker -- , seq: 85619, time: 2014-09-18 18:08:23.986955, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11702) -9062> 2014-09-18 18:08:23.986964 7f4c9f251700 5 -- op tracker -- , seq: 85619, time: 2014-09-18 18:08:23.986962, event: started, op: pg_notify(2.74e(11) epoch 11702) -9061> 2014-09-18 18:08:23.986982 7f4c9f251700 5 -- op tracker -- , seq: 85619, time: 2014-09-18 18:08:23.986981, event: done, op: pg_notify(2.74e(11) epoch 11702) -9060> 2014-09-18 18:08:23.988472 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3138 ==== pg_notify(2.4b1(13) epoch 11702) v5 ==== 1441+0+0 (3733989693 0 0) 0xe146960 con 0x1395ec60 -9059> 2014-09-18 18:08:23.988492 7f4c9f251700 5 -- op tracker -- , seq: 85620, time: 2014-09-18 18:08:23.988390, event: header_read, op: pg_notify(2.4b1(13) epoch 11702) -9058> 2014-09-18 18:08:23.988503 7f4c9f251700 5 -- op tracker -- , seq: 85620, time: 2014-09-18 18:08:23.988391, event: throttled, op: pg_notify(2.4b1(13) epoch 11702) -9057> 2014-09-18 18:08:23.988509 7f4c9f251700 5 -- op tracker -- , seq: 85620, time: 2014-09-18 18:08:23.988432, event: all_read, op: pg_notify(2.4b1(13) epoch 11702) -9056> 2014-09-18 18:08:23.988515 7f4c9f251700 5 -- op tracker -- , seq: 85620, time: 2014-09-18 18:08:23.988488, event: dispatched, op: pg_notify(2.4b1(13) epoch 11702) -9055> 2014-09-18 18:08:23.988521 7f4c9f251700 5 -- op tracker -- , seq: 85620, time: 2014-09-18 18:08:23.988520, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11702) -9054> 2014-09-18 18:08:23.988529 7f4c9f251700 5 -- op tracker -- , seq: 85620, time: 2014-09-18 18:08:23.988528, event: started, op: pg_notify(2.4b1(13) epoch 11702) -9053> 2014-09-18 18:08:23.988549 7f4c9f251700 5 -- op tracker -- , seq: 85620, time: 2014-09-18 18:08:23.988547, event: done, op: pg_notify(2.4b1(13) epoch 11702) -9052> 2014-09-18 18:08:23.988630 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 513 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x1182da80 con 0x1395ef20 -9051> 2014-09-18 18:08:23.988643 7f4c9f251700 3 osd.42 11702 handle_osd_map epochs [11702,11702], i have 11702, src has [1,11702] -9050> 2014-09-18 18:08:23.988680 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 514 ==== pg_notify(2.1d8(20) epoch 11702) v5 ==== 1878+0+0 (2927411260 0 0) 0x21e105a0 con 0x1395ef20 -9049> 2014-09-18 18:08:23.988690 7f4c9f251700 5 -- op tracker -- , seq: 85621, time: 2014-09-18 18:08:23.988631, event: header_read, op: pg_notify(2.1d8(20) epoch 11702) -9048> 2014-09-18 18:08:23.988697 7f4c9f251700 5 -- op tracker -- , seq: 85621, time: 2014-09-18 18:08:23.988631, event: throttled, op: pg_notify(2.1d8(20) epoch 11702) -9047> 2014-09-18 18:08:23.988703 7f4c9f251700 5 -- op tracker -- , seq: 85621, time: 2014-09-18 18:08:23.988670, event: all_read, op: pg_notify(2.1d8(20) epoch 11702) -9046> 2014-09-18 18:08:23.988709 7f4c9f251700 5 -- op tracker -- , seq: 85621, time: 2014-09-18 18:08:23.988687, event: dispatched, op: pg_notify(2.1d8(20) epoch 11702) -9045> 2014-09-18 18:08:23.988715 7f4c9f251700 5 -- op tracker -- , seq: 85621, time: 2014-09-18 18:08:23.988714, event: waiting_for_osdmap, op: pg_notify(2.1d8(20) epoch 11702) -9044> 2014-09-18 18:08:23.988723 7f4c9f251700 5 -- op tracker -- , seq: 85621, time: 2014-09-18 18:08:23.988721, event: started, op: pg_notify(2.1d8(20) epoch 11702) -9043> 2014-09-18 18:08:23.988739 7f4c9f251700 5 -- op tracker -- , seq: 85621, time: 2014-09-18 18:08:23.988737, event: done, op: pg_notify(2.1d8(20) epoch 11702) -9042> 2014-09-18 18:08:23.991675 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3139 ==== pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11702) v5 ==== 8942+0+0 (3652728068 0 0) 0xe147c20 con 0x1395ec60 -9041> 2014-09-18 18:08:23.991838 7f4c9f251700 5 -- op tracker -- , seq: 85622, time: 2014-09-18 18:08:23.991512, event: header_read, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11702) -9040> 2014-09-18 18:08:23.991851 7f4c9f251700 5 -- op tracker -- , seq: 85622, time: 2014-09-18 18:08:23.991514, event: throttled, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11702) -9039> 2014-09-18 18:08:23.991858 7f4c9f251700 5 -- op tracker -- , seq: 85622, time: 2014-09-18 18:08:23.991646, event: all_read, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11702) -9038> 2014-09-18 18:08:23.991865 7f4c9f251700 5 -- op tracker -- , seq: 85622, time: 2014-09-18 18:08:23.991830, event: dispatched, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11702) -9037> 2014-09-18 18:08:23.991870 7f4c9f251700 5 -- op tracker -- , seq: 85622, time: 2014-09-18 18:08:23.991868, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11702) -9036> 2014-09-18 18:08:23.991877 7f4c9f251700 5 -- op tracker -- , seq: 85622, time: 2014-09-18 18:08:23.991875, event: started, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11702) -9035> 2014-09-18 18:08:23.991928 7f4c9f251700 5 -- op tracker -- , seq: 85622, time: 2014-09-18 18:08:23.991926, event: done, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11702) -9034> 2014-09-18 18:08:23.991989 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 4 ==== pg_notify(2.78f(153),2.4c6(15) epoch 11702) v5 ==== 10728+0+0 (921621635 0 0) 0x128cd480 con 0x12b3a680 -9033> 2014-09-18 18:08:23.992008 7f4c9f251700 5 -- op tracker -- , seq: 85623, time: 2014-09-18 18:08:23.991803, event: header_read, op: pg_notify(2.78f(153),2.4c6(15) epoch 11702) -9032> 2014-09-18 18:08:23.992019 7f4c9f251700 5 -- op tracker -- , seq: 85623, time: 2014-09-18 18:08:23.991804, event: throttled, op: pg_notify(2.78f(153),2.4c6(15) epoch 11702) -9031> 2014-09-18 18:08:23.992032 7f4c9f251700 5 -- op tracker -- , seq: 85623, time: 2014-09-18 18:08:23.991975, event: all_read, op: pg_notify(2.78f(153),2.4c6(15) epoch 11702) -9030> 2014-09-18 18:08:23.992038 7f4c9f251700 5 -- op tracker -- , seq: 85623, time: 2014-09-18 18:08:23.992003, event: dispatched, op: pg_notify(2.78f(153),2.4c6(15) epoch 11702) -9029> 2014-09-18 18:08:23.992045 7f4c9f251700 5 -- op tracker -- , seq: 85623, time: 2014-09-18 18:08:23.992043, event: waiting_for_osdmap, op: pg_notify(2.78f(153),2.4c6(15) epoch 11702) -9028> 2014-09-18 18:08:23.992053 7f4c9f251700 5 -- op tracker -- , seq: 85623, time: 2014-09-18 18:08:23.992051, event: started, op: pg_notify(2.78f(153),2.4c6(15) epoch 11702) -9027> 2014-09-18 18:08:23.992087 7f4c9f251700 5 -- op tracker -- , seq: 85623, time: 2014-09-18 18:08:23.992085, event: done, op: pg_notify(2.78f(153),2.4c6(15) epoch 11702) -9026> 2014-09-18 18:08:23.993279 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 5 ==== pg_notify(2.78f(153) epoch 11702) v5 ==== 9229+0+0 (1305360145 0 0) 0xeb79860 con 0x12b3a680 -9025> 2014-09-18 18:08:23.993296 7f4c9f251700 5 -- op tracker -- , seq: 85624, time: 2014-09-18 18:08:23.993115, event: header_read, op: pg_notify(2.78f(153) epoch 11702) -9024> 2014-09-18 18:08:23.993307 7f4c9f251700 5 -- op tracker -- , seq: 85624, time: 2014-09-18 18:08:23.993117, event: throttled, op: pg_notify(2.78f(153) epoch 11702) -9023> 2014-09-18 18:08:23.993314 7f4c9f251700 5 -- op tracker -- , seq: 85624, time: 2014-09-18 18:08:23.993253, event: all_read, op: pg_notify(2.78f(153) epoch 11702) -9022> 2014-09-18 18:08:23.993320 7f4c9f251700 5 -- op tracker -- , seq: 85624, time: 2014-09-18 18:08:23.993293, event: dispatched, op: pg_notify(2.78f(153) epoch 11702) -9021> 2014-09-18 18:08:23.993327 7f4c9f251700 5 -- op tracker -- , seq: 85624, time: 2014-09-18 18:08:23.993325, event: waiting_for_osdmap, op: pg_notify(2.78f(153) epoch 11702) -9020> 2014-09-18 18:08:23.993334 7f4c9f251700 5 -- op tracker -- , seq: 85624, time: 2014-09-18 18:08:23.993333, event: started, op: pg_notify(2.78f(153) epoch 11702) -9019> 2014-09-18 18:08:23.993352 7f4c9f251700 5 -- op tracker -- , seq: 85624, time: 2014-09-18 18:08:23.993351, event: done, op: pg_notify(2.78f(153) epoch 11702) -9018> 2014-09-18 18:08:23.994053 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.11 10.242.42.168:0/17298 123 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:23.861216) v2 ==== 47+0+0 (2181458475 0 0) 0xe76eb20 con 0xcc49e40 -9017> 2014-09-18 18:08:23.994075 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.861216) v2 -- ?+0 0xf7a65a0 con 0xcc49e40 -9016> 2014-09-18 18:08:23.994081 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.11 10.242.42.168:0/17298 123 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:23.861216) v2 ==== 47+0+0 (2181458475 0 0) 0xf09cd00 con 0x101cd180 -9015> 2014-09-18 18:08:23.994112 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:23.861216) v2 -- ?+0 0xe0bb4a0 con 0x101cd180 -9014> 2014-09-18 18:08:23.995486 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 515 ==== pg_notify(2.4d0(10) epoch 11702) v5 ==== 1292+0+0 (1686962077 0 0) 0xe9fef00 con 0x1395ef20 -9013> 2014-09-18 18:08:23.995503 7f4c9f251700 5 -- op tracker -- , seq: 85625, time: 2014-09-18 18:08:23.995436, event: header_read, op: pg_notify(2.4d0(10) epoch 11702) -9012> 2014-09-18 18:08:23.995514 7f4c9f251700 5 -- op tracker -- , seq: 85625, time: 2014-09-18 18:08:23.995437, event: throttled, op: pg_notify(2.4d0(10) epoch 11702) -9011> 2014-09-18 18:08:23.995521 7f4c9f251700 5 -- op tracker -- , seq: 85625, time: 2014-09-18 18:08:23.995468, event: all_read, op: pg_notify(2.4d0(10) epoch 11702) -9010> 2014-09-18 18:08:23.995527 7f4c9f251700 5 -- op tracker -- , seq: 85625, time: 2014-09-18 18:08:23.995499, event: dispatched, op: pg_notify(2.4d0(10) epoch 11702) -9009> 2014-09-18 18:08:23.995533 7f4c9f251700 5 -- op tracker -- , seq: 85625, time: 2014-09-18 18:08:23.995531, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11702) -9008> 2014-09-18 18:08:23.995541 7f4c9f251700 5 -- op tracker -- , seq: 85625, time: 2014-09-18 18:08:23.995539, event: started, op: pg_notify(2.4d0(10) epoch 11702) -9007> 2014-09-18 18:08:23.995578 7f4c9f251700 5 -- op tracker -- , seq: 85625, time: 2014-09-18 18:08:23.995576, event: done, op: pg_notify(2.4d0(10) epoch 11702) -9006> 2014-09-18 18:08:23.995880 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 298 ==== pg_info(1 pgs e11702:2.756) v4 ==== 738+0+0 (4109743787 0 0) 0xe797a20 con 0x140f95a0 -9005> 2014-09-18 18:08:23.995895 7f4c9f251700 5 -- op tracker -- , seq: 85626, time: 2014-09-18 18:08:23.995827, event: header_read, op: pg_info(1 pgs e11702:2.756) -9004> 2014-09-18 18:08:23.995904 7f4c9f251700 5 -- op tracker -- , seq: 85626, time: 2014-09-18 18:08:23.995828, event: throttled, op: pg_info(1 pgs e11702:2.756) -9003> 2014-09-18 18:08:23.995910 7f4c9f251700 5 -- op tracker -- , seq: 85626, time: 2014-09-18 18:08:23.995868, event: all_read, op: pg_info(1 pgs e11702:2.756) -9002> 2014-09-18 18:08:23.995916 7f4c9f251700 5 -- op tracker -- , seq: 85626, time: 2014-09-18 18:08:23.995892, event: dispatched, op: pg_info(1 pgs e11702:2.756) -9001> 2014-09-18 18:08:23.995922 7f4c9f251700 5 -- op tracker -- , seq: 85626, time: 2014-09-18 18:08:23.995921, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.756) -9000> 2014-09-18 18:08:23.995930 7f4c9f251700 5 -- op tracker -- , seq: 85626, time: 2014-09-18 18:08:23.995928, event: started, op: pg_info(1 pgs e11702:2.756) -8999> 2014-09-18 18:08:23.995946 7f4c9f251700 5 -- op tracker -- , seq: 85626, time: 2014-09-18 18:08:23.995945, event: done, op: pg_info(1 pgs e11702:2.756) -8998> 2014-09-18 18:08:24.002098 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 516 ==== pg_info(1 pgs e11702:2.1d8) v4 ==== 742+0+0 (1811721802 0 0) 0xf344b20 con 0x1395ef20 -8997> 2014-09-18 18:08:24.002117 7f4c9f251700 5 -- op tracker -- , seq: 85627, time: 2014-09-18 18:08:24.002056, event: header_read, op: pg_info(1 pgs e11702:2.1d8) -8996> 2014-09-18 18:08:24.002127 7f4c9f251700 5 -- op tracker -- , seq: 85627, time: 2014-09-18 18:08:24.002057, event: throttled, op: pg_info(1 pgs e11702:2.1d8) -8995> 2014-09-18 18:08:24.002134 7f4c9f251700 5 -- op tracker -- , seq: 85627, time: 2014-09-18 18:08:24.002081, event: all_read, op: pg_info(1 pgs e11702:2.1d8) -8994> 2014-09-18 18:08:24.002139 7f4c9f251700 5 -- op tracker -- , seq: 85627, time: 2014-09-18 18:08:24.002113, event: dispatched, op: pg_info(1 pgs e11702:2.1d8) -8993> 2014-09-18 18:08:24.002146 7f4c9f251700 5 -- op tracker -- , seq: 85627, time: 2014-09-18 18:08:24.002145, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.1d8) -8992> 2014-09-18 18:08:24.002154 7f4c9f251700 5 -- op tracker -- , seq: 85627, time: 2014-09-18 18:08:24.002152, event: started, op: pg_info(1 pgs e11702:2.1d8) -8991> 2014-09-18 18:08:24.002172 7f4c9f251700 5 -- op tracker -- , seq: 85627, time: 2014-09-18 18:08:24.002170, event: done, op: pg_info(1 pgs e11702:2.1d8) -8990> 2014-09-18 18:08:24.002187 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6815/37668 -- pg_info(1 pgs e11702:2.1d8) v4 -- ?+0 0xe795680 con 0x1088b440 -8989> 2014-09-18 18:08:24.002207 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11702:2.1d8) v4 -- ?+0 0xe797840 con 0x1395f8c0 -8988> 2014-09-18 18:08:24.002218 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_info(1 pgs e11702:2.1d8) v4 -- ?+0 0xe7961c0 con 0x1395ef20 -8987> 2014-09-18 18:08:24.002382 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6829/19404 -- pg_info(1 pgs e11702:2.1d8) v4 -- ?+0 0xf0b8ee0 con 0xe2d9a20 -8986> 2014-09-18 18:08:24.002590 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11702 n=492 ec=188 les/c 11702/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 active+remapped] on activate: bft=22,27 from 0//0//-1 -8985> 2014-09-18 18:08:24.002666 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11702 n=492 ec=188 les/c 11702/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 active+remapped] target shard 22 from 0//0//-1 -8984> 2014-09-18 18:08:24.002686 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11702 n=492 ec=188 les/c 11702/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 active+remapped] target shard 27 from 0//0//-1 -8983> 2014-09-18 18:08:24.002705 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11702 n=492 ec=188 les/c 11702/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 active+remapped] exit Started/Primary/Active/Activating 0.079411 14 0.000940 -8982> 2014-09-18 18:08:24.002718 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.1d8( v 6865'30228 (4155'27332,6865'30228] local-les=11702 n=492 ec=188 les/c 11702/11160 11701/11701/11159) [42,22,27]/[42,32,55] r=0 lpr=11701 pi=11159-11700/9 bft=22,27 crt=6865'30228 mlcod 0'0 active+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -8981> 2014-09-18 18:08:24.034313 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.24 10.242.42.172:0/5612 110 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:13.138272) v2 ==== 47+0+0 (2145336586 0 0) 0xf107c20 con 0x10a5bb80 -8980> 2014-09-18 18:08:24.034332 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:13.138272) v2 -- ?+0 0xf09cd00 con 0x10a5bb80 -8979> 2014-09-18 18:08:24.034414 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.24 10.242.42.172:0/5612 110 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:13.138272) v2 ==== 47+0+0 (2145336586 0 0) 0xf1e8760 con 0x110bb020 -8978> 2014-09-18 18:08:24.034444 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:13.138272) v2 -- ?+0 0xe76eb20 con 0x110bb020 -8977> 2014-09-18 18:08:24.069249 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.37 10.242.42.168:6809/18319 10 ==== pg_info(1 pgs e11702:2.6a6) v4 ==== 738+0+0 (2027628711 0 0) 0xf0b6960 con 0x1395eb00 -8976> 2014-09-18 18:08:24.069269 7f4c9f251700 5 -- op tracker -- , seq: 85628, time: 2014-09-18 18:08:24.069120, event: header_read, op: pg_info(1 pgs e11702:2.6a6) -8975> 2014-09-18 18:08:24.069280 7f4c9f251700 5 -- op tracker -- , seq: 85628, time: 2014-09-18 18:08:24.069121, event: throttled, op: pg_info(1 pgs e11702:2.6a6) -8974> 2014-09-18 18:08:24.069288 7f4c9f251700 5 -- op tracker -- , seq: 85628, time: 2014-09-18 18:08:24.069147, event: all_read, op: pg_info(1 pgs e11702:2.6a6) -8973> 2014-09-18 18:08:24.069295 7f4c9f251700 5 -- op tracker -- , seq: 85628, time: 2014-09-18 18:08:24.069264, event: dispatched, op: pg_info(1 pgs e11702:2.6a6) -8972> 2014-09-18 18:08:24.069304 7f4c9f251700 5 -- op tracker -- , seq: 85628, time: 2014-09-18 18:08:24.069303, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.6a6) -8971> 2014-09-18 18:08:24.069314 7f4c9f251700 5 -- op tracker -- , seq: 85628, time: 2014-09-18 18:08:24.069312, event: started, op: pg_info(1 pgs e11702:2.6a6) -8970> 2014-09-18 18:08:24.069332 7f4c9f251700 5 -- op tracker -- , seq: 85628, time: 2014-09-18 18:08:24.069330, event: done, op: pg_info(1 pgs e11702:2.6a6) -8969> 2014-09-18 18:08:24.194730 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3584 ==== pg_notify(2.78f(54) epoch 11702) v5 ==== 3616+0+0 (3419009632 0 0) 0xf275de0 con 0xcbecec0 -8968> 2014-09-18 18:08:24.194761 7f4c9f251700 5 -- op tracker -- , seq: 85629, time: 2014-09-18 18:08:24.194593, event: header_read, op: pg_notify(2.78f(54) epoch 11702) -8967> 2014-09-18 18:08:24.194780 7f4c9f251700 5 -- op tracker -- , seq: 85629, time: 2014-09-18 18:08:24.194596, event: throttled, op: pg_notify(2.78f(54) epoch 11702) -8966> 2014-09-18 18:08:24.194791 7f4c9f251700 5 -- op tracker -- , seq: 85629, time: 2014-09-18 18:08:24.194705, event: all_read, op: pg_notify(2.78f(54) epoch 11702) -8965> 2014-09-18 18:08:24.194800 7f4c9f251700 5 -- op tracker -- , seq: 85629, time: 2014-09-18 18:08:24.194753, event: dispatched, op: pg_notify(2.78f(54) epoch 11702) -8964> 2014-09-18 18:08:24.194811 7f4c9f251700 5 -- op tracker -- , seq: 85629, time: 2014-09-18 18:08:24.194809, event: waiting_for_osdmap, op: pg_notify(2.78f(54) epoch 11702) -8963> 2014-09-18 18:08:24.194824 7f4c9f251700 5 -- op tracker -- , seq: 85629, time: 2014-09-18 18:08:24.194822, event: started, op: pg_notify(2.78f(54) epoch 11702) -8962> 2014-09-18 18:08:24.194847 7f4c9f251700 5 -- op tracker -- , seq: 85629, time: 2014-09-18 18:08:24.194846, event: done, op: pg_notify(2.78f(54) epoch 11702) -8961> 2014-09-18 18:08:24.194880 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.275290 7 0.000188 -8960> 2014-09-18 18:08:24.194924 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -8959> 2014-09-18 18:08:24.195421 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000497 0 0.000000 -8958> 2014-09-18 18:08:24.195638 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -8957> 2014-09-18 18:08:24.195743 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000105 0 0.000000 -8956> 2014-09-18 18:08:24.195802 7f4c96a40700 5 osd.42 pg_epoch: 11702 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -8955> 2014-09-18 18:08:24.196541 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3585 ==== pg_notify(2.78f(54) epoch 11702) v5 ==== 3616+0+0 (3419009632 0 0) 0xf4f0580 con 0xcbecec0 -8954> 2014-09-18 18:08:24.196569 7f4c9f251700 5 -- op tracker -- , seq: 85630, time: 2014-09-18 18:08:24.196463, event: header_read, op: pg_notify(2.78f(54) epoch 11702) -8953> 2014-09-18 18:08:24.196580 7f4c9f251700 5 -- op tracker -- , seq: 85630, time: 2014-09-18 18:08:24.196465, event: throttled, op: pg_notify(2.78f(54) epoch 11702) -8952> 2014-09-18 18:08:24.196586 7f4c9f251700 5 -- op tracker -- , seq: 85630, time: 2014-09-18 18:08:24.196525, event: all_read, op: pg_notify(2.78f(54) epoch 11702) -8951> 2014-09-18 18:08:24.196592 7f4c9f251700 5 -- op tracker -- , seq: 85630, time: 2014-09-18 18:08:24.196565, event: dispatched, op: pg_notify(2.78f(54) epoch 11702) -8950> 2014-09-18 18:08:24.196597 7f4c9f251700 5 -- op tracker -- , seq: 85630, time: 2014-09-18 18:08:24.196596, event: waiting_for_osdmap, op: pg_notify(2.78f(54) epoch 11702) -8949> 2014-09-18 18:08:24.196605 7f4c9f251700 5 -- op tracker -- , seq: 85630, time: 2014-09-18 18:08:24.196603, event: started, op: pg_notify(2.78f(54) epoch 11702) -8948> 2014-09-18 18:08:24.196623 7f4c9f251700 5 -- op tracker -- , seq: 85630, time: 2014-09-18 18:08:24.196622, event: done, op: pg_notify(2.78f(54) epoch 11702) -8947> 2014-09-18 18:08:24.252605 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.57 10.242.42.176:0/988 117 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:29.000763) v2 ==== 47+0+0 (4257315090 0 0) 0xf726000 con 0x12abfc80 -8946> 2014-09-18 18:08:24.252642 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/988 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:29.000763) v2 -- ?+0 0xf107c20 con 0x12abfc80 -8945> 2014-09-18 18:08:24.252642 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.57 10.242.42.176:0/988 117 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:29.000763) v2 ==== 47+0+0 (4257315090 0 0) 0xf15f860 con 0x12561e40 -8944> 2014-09-18 18:08:24.252678 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/988 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:29.000763) v2 -- ?+0 0xf1e8760 con 0x12561e40 -8943> 2014-09-18 18:08:24.382347 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.22 10.242.42.172:0/37668 438 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:13.486909) v2 ==== 47+0+0 (291219903 0 0) 0xe957a40 con 0x109caf20 -8942> 2014-09-18 18:08:24.382414 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:13.486909) v2 -- ?+0 0xf15f860 con 0x109caf20 -8941> 2014-09-18 18:08:24.383026 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.22 10.242.42.172:0/37668 438 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:13.486909) v2 ==== 47+0+0 (291219903 0 0) 0xf93da20 con 0x14393340 -8940> 2014-09-18 18:08:24.383080 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:13.486909) v2 -- ?+0 0xf726000 con 0x14393340 -8939> 2014-09-18 18:08:24.401152 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.51 10.242.42.168:0/15056 147 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:24.268110) v2 ==== 47+0+0 (2214454941 0 0) 0x10508580 con 0xcc76f20 -8938> 2014-09-18 18:08:24.401196 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/15056 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.268110) v2 -- ?+0 0xf93da20 con 0xcc76f20 -8937> 2014-09-18 18:08:24.401205 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.51 10.242.42.168:0/15056 147 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:24.268110) v2 ==== 47+0+0 (2214454941 0 0) 0xf7883a0 con 0x17de92e0 -8936> 2014-09-18 18:08:24.401235 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/15056 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.268110) v2 -- ?+0 0xe957a40 con 0x17de92e0 -8935> 2014-09-18 18:08:24.476054 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.30 10.242.42.176:0/1204 118 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:29.223902) v2 ==== 47+0+0 (3557823597 0 0) 0xf188ee0 con 0x110bbc80 -8934> 2014-09-18 18:08:24.476075 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.30 10.242.42.176:0/1204 118 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:29.223902) v2 ==== 47+0+0 (3557823597 0 0) 0xeb7a1c0 con 0x12abd1e0 -8933> 2014-09-18 18:08:24.476114 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/1204 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:29.223902) v2 -- ?+0 0xf7883a0 con 0x110bbc80 -8932> 2014-09-18 18:08:24.476150 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/1204 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:29.223902) v2 -- ?+0 0x10508580 con 0x12abd1e0 -8931> 2014-09-18 18:08:24.583909 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.35 10.242.42.164:0/17421 163 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:24.582365) v2 ==== 47+0+0 (2280920946 0 0) 0xddf52a0 con 0x1d5bcd60 -8930> 2014-09-18 18:08:24.583909 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.35 10.242.42.164:0/17421 163 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:24.582365) v2 ==== 47+0+0 (2280920946 0 0) 0xe76fde0 con 0x142e9700 -8929> 2014-09-18 18:08:24.583955 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/17421 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.582365) v2 -- ?+0 0xeb7a1c0 con 0x1d5bcd60 -8928> 2014-09-18 18:08:24.583984 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/17421 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.582365) v2 -- ?+0 0xf188ee0 con 0x142e9700 -8927> 2014-09-18 18:08:24.606407 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.23 10.242.42.172:0/11463 1547 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:13.709138) v2 ==== 47+0+0 (3631144158 0 0) 0xfcc94a0 con 0xe567080 -8926> 2014-09-18 18:08:24.606439 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/11463 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:13.709138) v2 -- ?+0 0xe76fde0 con 0xe567080 -8925> 2014-09-18 18:08:24.606683 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.23 10.242.42.172:0/11463 1547 ==== osd_ping(ping e11702 stamp 2014-09-18 23:37:13.709138) v2 ==== 47+0+0 (3631144158 0 0) 0xe1a2960 con 0xec5c160 -8924> 2014-09-18 18:08:24.606702 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/11463 -- osd_ping(ping_reply e11702 stamp 2014-09-18 23:37:13.709138) v2 -- ?+0 0xddf52a0 con 0xec5c160 -8923> 2014-09-18 18:08:24.704003 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.43 10.242.42.178:0/4511 144 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:24.703301) v2 ==== 47+0+0 (4207462296 0 0) 0x21cfb0e0 con 0x17de6000 -8922> 2014-09-18 18:08:24.704038 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.43 10.242.42.178:0/4511 144 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:24.703301) v2 ==== 47+0+0 (4207462296 0 0) 0x19f25a20 con 0xcc76dc0 -8921> 2014-09-18 18:08:24.704058 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/4511 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.703301) v2 -- ?+0 0xfcc94a0 con 0x17de6000 -8920> 2014-09-18 18:08:24.704251 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/4511 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.703301) v2 -- ?+0 0xe1a2960 con 0xcc76dc0 -8919> 2014-09-18 18:08:24.733596 7f4caae41700 5 osd.42 11702 tick -8918> 2014-09-18 18:08:24.735875 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.14 10.242.42.168:0/16237 139 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:24.602588) v2 ==== 47+0+0 (1278992556 0 0) 0xf7a70e0 con 0xcc49a20 -8917> 2014-09-18 18:08:24.735923 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.602588) v2 -- ?+0 0x21cfb0e0 con 0xcc49a20 -8916> 2014-09-18 18:08:24.735922 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.14 10.242.42.168:0/16237 139 ==== osd_ping(ping e11702 stamp 2014-09-18 18:08:24.602588) v2 ==== 47+0+0 (1278992556 0 0) 0xe0bcb20 con 0x101cc3c0 -8915> 2014-09-18 18:08:24.735952 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.602588) v2 -- ?+0 0x19f25a20 con 0x101cc3c0 -8914> 2014-09-18 18:08:24.763585 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5052 ==== pg_notify(2.62(44) epoch 11701) v5 ==== 3220+0+0 (903907195 0 0) 0xf6dfa40 con 0x1088aec0 -8913> 2014-09-18 18:08:24.763614 7f4c9f251700 5 -- op tracker -- , seq: 85631, time: 2014-09-18 18:08:24.763426, event: header_read, op: pg_notify(2.62(44) epoch 11701) -8912> 2014-09-18 18:08:24.763627 7f4c9f251700 5 -- op tracker -- , seq: 85631, time: 2014-09-18 18:08:24.763428, event: throttled, op: pg_notify(2.62(44) epoch 11701) -8911> 2014-09-18 18:08:24.763632 7f4c9f251700 5 -- op tracker -- , seq: 85631, time: 2014-09-18 18:08:24.763563, event: all_read, op: pg_notify(2.62(44) epoch 11701) -8910> 2014-09-18 18:08:24.763641 7f4c9f251700 5 -- op tracker -- , seq: 85631, time: 2014-09-18 18:08:24.763606, event: dispatched, op: pg_notify(2.62(44) epoch 11701) -8909> 2014-09-18 18:08:24.763649 7f4c9f251700 5 -- op tracker -- , seq: 85631, time: 2014-09-18 18:08:24.763647, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11701) -8908> 2014-09-18 18:08:24.763659 7f4c9f251700 5 -- op tracker -- , seq: 85631, time: 2014-09-18 18:08:24.763657, event: started, op: pg_notify(2.62(44) epoch 11701) -8907> 2014-09-18 18:08:24.763683 7f4c9f251700 5 -- op tracker -- , seq: 85631, time: 2014-09-18 18:08:24.763681, event: done, op: pg_notify(2.62(44) epoch 11701) -8906> 2014-09-18 18:08:24.772482 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5053 ==== pg_notify(2.3b6(10) epoch 11701) v5 ==== 1260+0+0 (245422889 0 0) 0xfa29de0 con 0x1088aec0 -8905> 2014-09-18 18:08:24.772511 7f4c9f251700 5 -- op tracker -- , seq: 85632, time: 2014-09-18 18:08:24.772387, event: header_read, op: pg_notify(2.3b6(10) epoch 11701) -8904> 2014-09-18 18:08:24.772523 7f4c9f251700 5 -- op tracker -- , seq: 85632, time: 2014-09-18 18:08:24.772389, event: throttled, op: pg_notify(2.3b6(10) epoch 11701) -8903> 2014-09-18 18:08:24.772530 7f4c9f251700 5 -- op tracker -- , seq: 85632, time: 2014-09-18 18:08:24.772442, event: all_read, op: pg_notify(2.3b6(10) epoch 11701) -8902> 2014-09-18 18:08:24.772536 7f4c9f251700 5 -- op tracker -- , seq: 85632, time: 2014-09-18 18:08:24.772504, event: dispatched, op: pg_notify(2.3b6(10) epoch 11701) -8901> 2014-09-18 18:08:24.772542 7f4c9f251700 5 -- op tracker -- , seq: 85632, time: 2014-09-18 18:08:24.772541, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11701) -8900> 2014-09-18 18:08:24.772552 7f4c9f251700 5 -- op tracker -- , seq: 85632, time: 2014-09-18 18:08:24.772550, event: started, op: pg_notify(2.3b6(10) epoch 11701) -8899> 2014-09-18 18:08:24.772574 7f4c9f251700 5 -- op tracker -- , seq: 85632, time: 2014-09-18 18:08:24.772572, event: done, op: pg_notify(2.3b6(10) epoch 11701) -8898> 2014-09-18 18:08:24.814207 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.0 10.242.42.164:0/15878 158 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:24.813476) v2 ==== 47+0+0 (1716337737 0 0) 0x12138f00 con 0x1d5bc7e0 -8897> 2014-09-18 18:08:24.814236 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.0 10.242.42.164:0/15878 158 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:24.813476) v2 ==== 47+0+0 (1716337737 0 0) 0xde5f0e0 con 0x142e8260 -8896> 2014-09-18 18:08:24.814250 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.813476) v2 -- ?+0 0xe0bcb20 con 0x1d5bc7e0 -8895> 2014-09-18 18:08:24.814276 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11702 stamp 2014-09-18 18:08:24.813476) v2 -- ?+0 0xf7a70e0 con 0x142e8260 -8894> 2014-09-18 18:08:24.906641 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 25 ==== osd_map(11702..11703 src has 1..11703) v3 ==== 3197+0+0 (1251843143 0 0) 0xe2ee640 con 0x111771e0 -8893> 2014-09-18 18:08:24.906765 7f4ca0a54700 3 osd.42 11702 handle_osd_map epochs [11702,11703], i have 11702, src has [1,11703] -8892> 2014-09-18 18:08:24.909122 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0x139a6d80 con 0x1395ec60 -8891> 2014-09-18 18:08:24.909163 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.3b5(65) epoch 11703) v5 -- ?+0 0xf2b8760 con 0x1395ec60 -8890> 2014-09-18 18:08:24.909928 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0x2fb9e000 con 0x140f95a0 -8889> 2014-09-18 18:08:24.909967 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11703) v5 -- ?+0 0x2b93bfe0 con 0x140f95a0 -8888> 2014-09-18 18:08:24.910026 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0x1182a480 con 0x1395edc0 -8887> 2014-09-18 18:08:24.910037 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11703) v5 -- ?+0 0x2b93be00 con 0x1395edc0 -8886> 2014-09-18 18:08:24.910084 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0x1182a000 con 0x1395f1e0 -8885> 2014-09-18 18:08:24.910096 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11703) v5 -- ?+0 0x2b93a3c0 con 0x1395f1e0 -8884> 2014-09-18 18:08:24.910449 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11703) v5 -- ?+0 0xe1aa960 con 0x140f95a0 -8883> 2014-09-18 18:08:24.910592 7f4c96a40700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.714790 4 0.001017 -8882> 2014-09-18 18:08:24.910661 7f4c96a40700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.991085 0 0.000000 -8881> 2014-09-18 18:08:24.910686 7f4c96a40700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11698 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -8880> 2014-09-18 18:08:24.911329 7f4c96a40700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11698/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -8879> 2014-09-18 18:08:24.911542 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0x1182bf80 con 0x23a74f20 -8878> 2014-09-18 18:08:24.911556 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11703) v5 -- ?+0 0xde5f4a0 con 0x23a74f20 -8877> 2014-09-18 18:08:24.911742 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0xe339a80 con 0x1395ef20 -8876> 2014-09-18 18:08:24.911775 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(10) epoch 11703) v5 -- ?+0 0xe1acb20 con 0x1395ef20 -8875> 2014-09-18 18:08:24.911799 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6884/38815 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0xe338640 con 0x12b3a680 -8874> 2014-09-18 18:08:24.911855 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6884/38815 -- pg_info(1 pgs e11703:2.78f(153)) v4 -- ?+0 0xe1ad660 con 0x12b3a680 -8873> 2014-09-18 18:08:24.911912 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0x24dbd440 con 0x1395f8c0 -8872> 2014-09-18 18:08:24.911992 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11703:2.78f(153)) v4 -- ?+0 0xf0b9de0 con 0x1395f8c0 -8871> 2014-09-18 18:08:24.912095 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11703..11703 src has 1..11703) v3 -- ?+0 0x24dbc240 con 0xcbecec0 -8870> 2014-09-18 18:08:24.912145 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11703:2.78f(153)) v4 -- ?+0 0x21e10d20 con 0xcbecec0 -8869> 2014-09-18 18:08:24.913660 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 26 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x2d7fc640 con 0x111771e0 -8868> 2014-09-18 18:08:24.913691 7f4ca0a54700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8867> 2014-09-18 18:08:24.914285 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3270 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x24d1d600 con 0x4900580 -8866> 2014-09-18 18:08:24.914307 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8865> 2014-09-18 18:08:24.914600 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3271 ==== pg_notify(2.4f6(184) epoch 11703) v5 ==== 11296+0+0 (3203324680 0 0) 0xfc58760 con 0x4900580 -8864> 2014-09-18 18:08:24.914618 7f4c9f251700 5 -- op tracker -- , seq: 85633, time: 2014-09-18 18:08:24.914368, event: header_read, op: pg_notify(2.4f6(184) epoch 11703) -8863> 2014-09-18 18:08:24.914629 7f4c9f251700 5 -- op tracker -- , seq: 85633, time: 2014-09-18 18:08:24.914369, event: throttled, op: pg_notify(2.4f6(184) epoch 11703) -8862> 2014-09-18 18:08:24.914636 7f4c9f251700 5 -- op tracker -- , seq: 85633, time: 2014-09-18 18:08:24.914586, event: all_read, op: pg_notify(2.4f6(184) epoch 11703) -8861> 2014-09-18 18:08:24.914642 7f4c9f251700 5 -- op tracker -- , seq: 85633, time: 2014-09-18 18:08:24.914613, event: dispatched, op: pg_notify(2.4f6(184) epoch 11703) -8860> 2014-09-18 18:08:24.914648 7f4c9f251700 5 -- op tracker -- , seq: 85633, time: 2014-09-18 18:08:24.914647, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11703) -8859> 2014-09-18 18:08:24.914656 7f4c9f251700 5 -- op tracker -- , seq: 85633, time: 2014-09-18 18:08:24.914655, event: started, op: pg_notify(2.4f6(184) epoch 11703) -8858> 2014-09-18 18:08:24.914675 7f4c9f251700 5 -- op tracker -- , seq: 85633, time: 2014-09-18 18:08:24.914674, event: done, op: pg_notify(2.4f6(184) epoch 11703) -8857> 2014-09-18 18:08:24.914848 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 909 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x24dbd440 con 0x1395f8c0 -8856> 2014-09-18 18:08:24.914862 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8855> 2014-09-18 18:08:24.915720 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1026 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0xe939180 con 0x14358260 -8854> 2014-09-18 18:08:24.915761 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8853> 2014-09-18 18:08:24.915790 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 910 ==== pg_notify(2.78f(153) epoch 11703) v5 ==== 9229+0+0 (3248059949 0 0) 0xf0b9de0 con 0x1395f8c0 -8852> 2014-09-18 18:08:24.915810 7f4c9f251700 5 -- op tracker -- , seq: 85634, time: 2014-09-18 18:08:24.915610, event: header_read, op: pg_notify(2.78f(153) epoch 11703) -8851> 2014-09-18 18:08:24.915824 7f4c9f251700 5 -- op tracker -- , seq: 85634, time: 2014-09-18 18:08:24.915611, event: throttled, op: pg_notify(2.78f(153) epoch 11703) -8850> 2014-09-18 18:08:24.915862 7f4c9f251700 5 -- op tracker -- , seq: 85634, time: 2014-09-18 18:08:24.915772, event: all_read, op: pg_notify(2.78f(153) epoch 11703) -8849> 2014-09-18 18:08:24.915880 7f4c9f251700 5 -- op tracker -- , seq: 85634, time: 2014-09-18 18:08:24.915805, event: dispatched, op: pg_notify(2.78f(153) epoch 11703) -8848> 2014-09-18 18:08:24.915891 7f4c9f251700 5 -- op tracker -- , seq: 85634, time: 2014-09-18 18:08:24.915889, event: waiting_for_osdmap, op: pg_notify(2.78f(153) epoch 11703) -8847> 2014-09-18 18:08:24.915904 7f4c9f251700 5 -- op tracker -- , seq: 85634, time: 2014-09-18 18:08:24.915901, event: started, op: pg_notify(2.78f(153) epoch 11703) -8846> 2014-09-18 18:08:24.915932 7f4c9f251700 5 -- op tracker -- , seq: 85634, time: 2014-09-18 18:08:24.915930, event: done, op: pg_notify(2.78f(153) epoch 11703) -8845> 2014-09-18 18:08:24.915959 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1027 ==== pg_notify(2.719(17) epoch 11703) v5 ==== 1601+0+0 (306271361 0 0) 0x2b9412a0 con 0x14358260 -8844> 2014-09-18 18:08:24.915976 7f4c9f251700 5 -- op tracker -- , seq: 85635, time: 2014-09-18 18:08:24.915716, event: header_read, op: pg_notify(2.719(17) epoch 11703) -8843> 2014-09-18 18:08:24.915986 7f4c9f251700 5 -- op tracker -- , seq: 85635, time: 2014-09-18 18:08:24.915717, event: throttled, op: pg_notify(2.719(17) epoch 11703) -8842> 2014-09-18 18:08:24.915996 7f4c9f251700 5 -- op tracker -- , seq: 85635, time: 2014-09-18 18:08:24.915783, event: all_read, op: pg_notify(2.719(17) epoch 11703) -8841> 2014-09-18 18:08:24.916005 7f4c9f251700 5 -- op tracker -- , seq: 85635, time: 2014-09-18 18:08:24.915972, event: dispatched, op: pg_notify(2.719(17) epoch 11703) -8840> 2014-09-18 18:08:24.916015 7f4c9f251700 5 -- op tracker -- , seq: 85635, time: 2014-09-18 18:08:24.916012, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11703) -8839> 2014-09-18 18:08:24.916026 7f4c9f251700 5 -- op tracker -- , seq: 85635, time: 2014-09-18 18:08:24.916023, event: started, op: pg_notify(2.719(17) epoch 11703) -8838> 2014-09-18 18:08:24.916052 7f4c9f251700 5 -- op tracker -- , seq: 85635, time: 2014-09-18 18:08:24.916049, event: done, op: pg_notify(2.719(17) epoch 11703) -8837> 2014-09-18 18:08:24.916464 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3272 ==== pg_notify(2.17e(17) epoch 11703) v5 ==== 1593+0+0 (2625060459 0 0) 0xfc58580 con 0x4900580 -8836> 2014-09-18 18:08:24.916485 7f4c9f251700 5 -- op tracker -- , seq: 85636, time: 2014-09-18 18:08:24.916402, event: header_read, op: pg_notify(2.17e(17) epoch 11703) -8835> 2014-09-18 18:08:24.916498 7f4c9f251700 5 -- op tracker -- , seq: 85636, time: 2014-09-18 18:08:24.916403, event: throttled, op: pg_notify(2.17e(17) epoch 11703) -8834> 2014-09-18 18:08:24.916508 7f4c9f251700 5 -- op tracker -- , seq: 85636, time: 2014-09-18 18:08:24.916446, event: all_read, op: pg_notify(2.17e(17) epoch 11703) -8833> 2014-09-18 18:08:24.916515 7f4c9f251700 5 -- op tracker -- , seq: 85636, time: 2014-09-18 18:08:24.916480, event: dispatched, op: pg_notify(2.17e(17) epoch 11703) -8832> 2014-09-18 18:08:24.916521 7f4c9f251700 5 -- op tracker -- , seq: 85636, time: 2014-09-18 18:08:24.916519, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11703) -8831> 2014-09-18 18:08:24.916533 7f4c9f251700 5 -- op tracker -- , seq: 85636, time: 2014-09-18 18:08:24.916531, event: started, op: pg_notify(2.17e(17) epoch 11703) -8830> 2014-09-18 18:08:24.916561 7f4c9f251700 5 -- op tracker -- , seq: 85636, time: 2014-09-18 18:08:24.916558, event: done, op: pg_notify(2.17e(17) epoch 11703) -8829> 2014-09-18 18:08:24.917303 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1252 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0xeee8fc0 con 0xe8db760 -8828> 2014-09-18 18:08:24.917325 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8827> 2014-09-18 18:08:24.917376 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1253 ==== pg_notify(2.17e(17) epoch 11703) v5 ==== 1593+0+0 (2625060459 0 0) 0xf1063c0 con 0xe8db760 -8826> 2014-09-18 18:08:24.917397 7f4c9f251700 5 -- op tracker -- , seq: 85637, time: 2014-09-18 18:08:24.917303, event: header_read, op: pg_notify(2.17e(17) epoch 11703) -8825> 2014-09-18 18:08:24.917407 7f4c9f251700 5 -- op tracker -- , seq: 85637, time: 2014-09-18 18:08:24.917303, event: throttled, op: pg_notify(2.17e(17) epoch 11703) -8824> 2014-09-18 18:08:24.917413 7f4c9f251700 5 -- op tracker -- , seq: 85637, time: 2014-09-18 18:08:24.917351, event: all_read, op: pg_notify(2.17e(17) epoch 11703) -8823> 2014-09-18 18:08:24.917419 7f4c9f251700 5 -- op tracker -- , seq: 85637, time: 2014-09-18 18:08:24.917391, event: dispatched, op: pg_notify(2.17e(17) epoch 11703) -8822> 2014-09-18 18:08:24.917431 7f4c9f251700 5 -- op tracker -- , seq: 85637, time: 2014-09-18 18:08:24.917429, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11703) -8821> 2014-09-18 18:08:24.917441 7f4c9f251700 5 -- op tracker -- , seq: 85637, time: 2014-09-18 18:08:24.917438, event: started, op: pg_notify(2.17e(17) epoch 11703) -8820> 2014-09-18 18:08:24.917464 7f4c9f251700 5 -- op tracker -- , seq: 85637, time: 2014-09-18 18:08:24.917462, event: done, op: pg_notify(2.17e(17) epoch 11703) -8819> 2014-09-18 18:08:24.917969 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3273 ==== pg_notify(2.1bb(9) epoch 11703) v5 ==== 1189+0+0 (3533779923 0 0) 0x1368c1e0 con 0x4900580 -8818> 2014-09-18 18:08:24.917989 7f4c9f251700 5 -- op tracker -- , seq: 85638, time: 2014-09-18 18:08:24.917896, event: header_read, op: pg_notify(2.1bb(9) epoch 11703) -8817> 2014-09-18 18:08:24.918001 7f4c9f251700 5 -- op tracker -- , seq: 85638, time: 2014-09-18 18:08:24.917897, event: throttled, op: pg_notify(2.1bb(9) epoch 11703) -8816> 2014-09-18 18:08:24.918008 7f4c9f251700 5 -- op tracker -- , seq: 85638, time: 2014-09-18 18:08:24.917936, event: all_read, op: pg_notify(2.1bb(9) epoch 11703) -8815> 2014-09-18 18:08:24.918015 7f4c9f251700 5 -- op tracker -- , seq: 85638, time: 2014-09-18 18:08:24.917984, event: dispatched, op: pg_notify(2.1bb(9) epoch 11703) -8814> 2014-09-18 18:08:24.918021 7f4c9f251700 5 -- op tracker -- , seq: 85638, time: 2014-09-18 18:08:24.918019, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11703) -8813> 2014-09-18 18:08:24.918029 7f4c9f251700 5 -- op tracker -- , seq: 85638, time: 2014-09-18 18:08:24.918027, event: started, op: pg_notify(2.1bb(9) epoch 11703) -8812> 2014-09-18 18:08:24.918052 7f4c9f251700 5 -- op tracker -- , seq: 85638, time: 2014-09-18 18:08:24.918049, event: done, op: pg_notify(2.1bb(9) epoch 11703) -8811> 2014-09-18 18:08:24.918572 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3586 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x24dbc240 con 0xcbecec0 -8810> 2014-09-18 18:08:24.918597 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8809> 2014-09-18 18:08:24.918612 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3587 ==== pg_notify(2.4f6(12) epoch 11703) v5 ==== 1382+0+0 (2043682523 0 0) 0x21e10d20 con 0xcbecec0 -8808> 2014-09-18 18:08:24.918630 7f4c9f251700 5 -- op tracker -- , seq: 85639, time: 2014-09-18 18:08:24.918563, event: header_read, op: pg_notify(2.4f6(12) epoch 11703) -8807> 2014-09-18 18:08:24.918645 7f4c9f251700 5 -- op tracker -- , seq: 85639, time: 2014-09-18 18:08:24.918563, event: throttled, op: pg_notify(2.4f6(12) epoch 11703) -8806> 2014-09-18 18:08:24.918656 7f4c9f251700 5 -- op tracker -- , seq: 85639, time: 2014-09-18 18:08:24.918604, event: all_read, op: pg_notify(2.4f6(12) epoch 11703) -8805> 2014-09-18 18:08:24.918665 7f4c9f251700 5 -- op tracker -- , seq: 85639, time: 2014-09-18 18:08:24.918625, event: dispatched, op: pg_notify(2.4f6(12) epoch 11703) -8804> 2014-09-18 18:08:24.918675 7f4c9f251700 5 -- op tracker -- , seq: 85639, time: 2014-09-18 18:08:24.918673, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11703) -8803> 2014-09-18 18:08:24.918689 7f4c9f251700 5 -- op tracker -- , seq: 85639, time: 2014-09-18 18:08:24.918686, event: started, op: pg_notify(2.4f6(12) epoch 11703) -8802> 2014-09-18 18:08:24.918717 7f4c9f251700 5 -- op tracker -- , seq: 85639, time: 2014-09-18 18:08:24.918715, event: done, op: pg_notify(2.4f6(12) epoch 11703) -8801> 2014-09-18 18:08:24.919061 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 6 ==== pg_notify(2.120(15) epoch 11703) v5 ==== 1535+0+0 (3181740986 0 0) 0xe1ad660 con 0x12b3a680 -8800> 2014-09-18 18:08:24.919082 7f4c9f251700 5 -- op tracker -- , seq: 85640, time: 2014-09-18 18:08:24.919009, event: header_read, op: pg_notify(2.120(15) epoch 11703) -8799> 2014-09-18 18:08:24.919093 7f4c9f251700 5 -- op tracker -- , seq: 85640, time: 2014-09-18 18:08:24.919010, event: throttled, op: pg_notify(2.120(15) epoch 11703) -8798> 2014-09-18 18:08:24.919103 7f4c9f251700 5 -- op tracker -- , seq: 85640, time: 2014-09-18 18:08:24.919044, event: all_read, op: pg_notify(2.120(15) epoch 11703) -8797> 2014-09-18 18:08:24.919108 7f4c9f251700 5 -- op tracker -- , seq: 85640, time: 2014-09-18 18:08:24.919077, event: dispatched, op: pg_notify(2.120(15) epoch 11703) -8796> 2014-09-18 18:08:24.919113 7f4c9f251700 5 -- op tracker -- , seq: 85640, time: 2014-09-18 18:08:24.919112, event: waiting_for_osdmap, op: pg_notify(2.120(15) epoch 11703) -8795> 2014-09-18 18:08:24.919130 7f4c9f251700 5 -- op tracker -- , seq: 85640, time: 2014-09-18 18:08:24.919117, event: started, op: pg_notify(2.120(15) epoch 11703) -8794> 2014-09-18 18:08:24.919165 7f4c9f251700 5 -- op tracker -- , seq: 85640, time: 2014-09-18 18:08:24.919162, event: done, op: pg_notify(2.120(15) epoch 11703) -8793> 2014-09-18 18:08:24.919420 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 774 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0xe2ee880 con 0xe9a27e0 -8792> 2014-09-18 18:08:24.919435 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8791> 2014-09-18 18:08:24.919585 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 7 ==== pg_notify(2.40a(162) epoch 11703) v5 ==== 9736+0+0 (205615364 0 0) 0xe146960 con 0x12b3a680 -8790> 2014-09-18 18:08:24.919601 7f4c9f251700 5 -- op tracker -- , seq: 85641, time: 2014-09-18 18:08:24.919446, event: header_read, op: pg_notify(2.40a(162) epoch 11703) -8789> 2014-09-18 18:08:24.919610 7f4c9f251700 5 -- op tracker -- , seq: 85641, time: 2014-09-18 18:08:24.919447, event: throttled, op: pg_notify(2.40a(162) epoch 11703) -8788> 2014-09-18 18:08:24.919617 7f4c9f251700 5 -- op tracker -- , seq: 85641, time: 2014-09-18 18:08:24.919573, event: all_read, op: pg_notify(2.40a(162) epoch 11703) -8787> 2014-09-18 18:08:24.919623 7f4c9f251700 5 -- op tracker -- , seq: 85641, time: 2014-09-18 18:08:24.919597, event: dispatched, op: pg_notify(2.40a(162) epoch 11703) -8786> 2014-09-18 18:08:24.919630 7f4c9f251700 5 -- op tracker -- , seq: 85641, time: 2014-09-18 18:08:24.919628, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11703) -8785> 2014-09-18 18:08:24.919637 7f4c9f251700 5 -- op tracker -- , seq: 85641, time: 2014-09-18 18:08:24.919636, event: started, op: pg_notify(2.40a(162) epoch 11703) -8784> 2014-09-18 18:08:24.919655 7f4c9f251700 5 -- op tracker -- , seq: 85641, time: 2014-09-18 18:08:24.919654, event: done, op: pg_notify(2.40a(162) epoch 11703) -8783> 2014-09-18 18:08:24.919754 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 775 ==== pg_notify(2.2d8(173) epoch 11703) v5 ==== 10757+0+0 (3511634240 0 0) 0xe0c92a0 con 0xe9a27e0 -8782> 2014-09-18 18:08:24.919767 7f4c9f251700 5 -- op tracker -- , seq: 85642, time: 2014-09-18 18:08:24.919577, event: header_read, op: pg_notify(2.2d8(173) epoch 11703) -8781> 2014-09-18 18:08:24.919775 7f4c9f251700 5 -- op tracker -- , seq: 85642, time: 2014-09-18 18:08:24.919578, event: throttled, op: pg_notify(2.2d8(173) epoch 11703) -8780> 2014-09-18 18:08:24.919781 7f4c9f251700 5 -- op tracker -- , seq: 85642, time: 2014-09-18 18:08:24.919743, event: all_read, op: pg_notify(2.2d8(173) epoch 11703) -8779> 2014-09-18 18:08:24.919787 7f4c9f251700 5 -- op tracker -- , seq: 85642, time: 2014-09-18 18:08:24.919764, event: dispatched, op: pg_notify(2.2d8(173) epoch 11703) -8778> 2014-09-18 18:08:24.919794 7f4c9f251700 5 -- op tracker -- , seq: 85642, time: 2014-09-18 18:08:24.919792, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11703) -8777> 2014-09-18 18:08:24.919800 7f4c9f251700 5 -- op tracker -- , seq: 85642, time: 2014-09-18 18:08:24.919798, event: started, op: pg_notify(2.2d8(173) epoch 11703) -8776> 2014-09-18 18:08:24.919816 7f4c9f251700 5 -- op tracker -- , seq: 85642, time: 2014-09-18 18:08:24.919815, event: done, op: pg_notify(2.2d8(173) epoch 11703) -8775> 2014-09-18 18:08:24.920255 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 517 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0xe339a80 con 0x1395ef20 -8774> 2014-09-18 18:08:24.920276 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8773> 2014-09-18 18:08:24.920286 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 518 ==== pg_notify(2.4d0(10) epoch 11703) v5 ==== 1292+0+0 (1682702670 0 0) 0xe1acb20 con 0x1395ef20 -8772> 2014-09-18 18:08:24.920302 7f4c9f251700 5 -- op tracker -- , seq: 85643, time: 2014-09-18 18:08:24.920255, event: header_read, op: pg_notify(2.4d0(10) epoch 11703) -8771> 2014-09-18 18:08:24.920313 7f4c9f251700 5 -- op tracker -- , seq: 85643, time: 2014-09-18 18:08:24.920255, event: throttled, op: pg_notify(2.4d0(10) epoch 11703) -8770> 2014-09-18 18:08:24.920322 7f4c9f251700 5 -- op tracker -- , seq: 85643, time: 2014-09-18 18:08:24.920277, event: all_read, op: pg_notify(2.4d0(10) epoch 11703) -8769> 2014-09-18 18:08:24.920331 7f4c9f251700 5 -- op tracker -- , seq: 85643, time: 2014-09-18 18:08:24.920298, event: dispatched, op: pg_notify(2.4d0(10) epoch 11703) -8768> 2014-09-18 18:08:24.920342 7f4c9f251700 5 -- op tracker -- , seq: 85643, time: 2014-09-18 18:08:24.920340, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11703) -8767> 2014-09-18 18:08:24.920353 7f4c9f251700 5 -- op tracker -- , seq: 85643, time: 2014-09-18 18:08:24.920351, event: started, op: pg_notify(2.4d0(10) epoch 11703) -8766> 2014-09-18 18:08:24.920377 7f4c9f251700 5 -- op tracker -- , seq: 85643, time: 2014-09-18 18:08:24.920375, event: done, op: pg_notify(2.4d0(10) epoch 11703) -8765> 2014-09-18 18:08:24.920406 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2333 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0xe2666c0 con 0xdd8c940 -8764> 2014-09-18 18:08:24.920420 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8763> 2014-09-18 18:08:24.920483 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2334 ==== pg_notify(2.457(21) epoch 11703) v5 ==== 1875+0+0 (2894570949 0 0) 0xf1081c0 con 0xdd8c940 -8762> 2014-09-18 18:08:24.920500 7f4c9f251700 5 -- op tracker -- , seq: 85644, time: 2014-09-18 18:08:24.920420, event: header_read, op: pg_notify(2.457(21) epoch 11703) -8761> 2014-09-18 18:08:24.920510 7f4c9f251700 5 -- op tracker -- , seq: 85644, time: 2014-09-18 18:08:24.920421, event: throttled, op: pg_notify(2.457(21) epoch 11703) -8760> 2014-09-18 18:08:24.920518 7f4c9f251700 5 -- op tracker -- , seq: 85644, time: 2014-09-18 18:08:24.920470, event: all_read, op: pg_notify(2.457(21) epoch 11703) -8759> 2014-09-18 18:08:24.920526 7f4c9f251700 5 -- op tracker -- , seq: 85644, time: 2014-09-18 18:08:24.920495, event: dispatched, op: pg_notify(2.457(21) epoch 11703) -8758> 2014-09-18 18:08:24.920534 7f4c9f251700 5 -- op tracker -- , seq: 85644, time: 2014-09-18 18:08:24.920532, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11703) -8757> 2014-09-18 18:08:24.920545 7f4c9f251700 5 -- op tracker -- , seq: 85644, time: 2014-09-18 18:08:24.920542, event: started, op: pg_notify(2.457(21) epoch 11703) -8756> 2014-09-18 18:08:24.920570 7f4c9f251700 5 -- op tracker -- , seq: 85644, time: 2014-09-18 18:08:24.920567, event: done, op: pg_notify(2.457(21) epoch 11703) -8755> 2014-09-18 18:08:24.920773 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 30 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x12bf18c0 con 0x1395f340 -8754> 2014-09-18 18:08:24.920791 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8753> 2014-09-18 18:08:24.920827 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 31 ==== pg_notify(2.1d8(9) epoch 11703) v5 ==== 1261+0+0 (4040286175 0 0) 0xf0b90c0 con 0x1395f340 -8752> 2014-09-18 18:08:24.920844 7f4c9f251700 5 -- op tracker -- , seq: 85645, time: 2014-09-18 18:08:24.920776, event: header_read, op: pg_notify(2.1d8(9) epoch 11703) -8751> 2014-09-18 18:08:24.920856 7f4c9f251700 5 -- op tracker -- , seq: 85645, time: 2014-09-18 18:08:24.920777, event: throttled, op: pg_notify(2.1d8(9) epoch 11703) -8750> 2014-09-18 18:08:24.920865 7f4c9f251700 5 -- op tracker -- , seq: 85645, time: 2014-09-18 18:08:24.920813, event: all_read, op: pg_notify(2.1d8(9) epoch 11703) -8749> 2014-09-18 18:08:24.920875 7f4c9f251700 5 -- op tracker -- , seq: 85645, time: 2014-09-18 18:08:24.920840, event: dispatched, op: pg_notify(2.1d8(9) epoch 11703) -8748> 2014-09-18 18:08:24.920886 7f4c9f251700 5 -- op tracker -- , seq: 85645, time: 2014-09-18 18:08:24.920883, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11703) -8747> 2014-09-18 18:08:24.920895 7f4c9f251700 5 -- op tracker -- , seq: 85645, time: 2014-09-18 18:08:24.920893, event: started, op: pg_notify(2.1d8(9) epoch 11703) -8746> 2014-09-18 18:08:24.920918 7f4c9f251700 5 -- op tracker -- , seq: 85645, time: 2014-09-18 18:08:24.920916, event: done, op: pg_notify(2.1d8(9) epoch 11703) -8745> 2014-09-18 18:08:24.921125 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2335 ==== pg_notify(2.2c7(139) epoch 11703) v5 ==== 8651+0+0 (954609441 0 0) 0xf109660 con 0xdd8c940 -8744> 2014-09-18 18:08:24.921138 7f4c9f251700 5 -- op tracker -- , seq: 85646, time: 2014-09-18 18:08:24.920960, event: header_read, op: pg_notify(2.2c7(139) epoch 11703) -8743> 2014-09-18 18:08:24.921146 7f4c9f251700 5 -- op tracker -- , seq: 85646, time: 2014-09-18 18:08:24.920961, event: throttled, op: pg_notify(2.2c7(139) epoch 11703) -8742> 2014-09-18 18:08:24.921152 7f4c9f251700 5 -- op tracker -- , seq: 85646, time: 2014-09-18 18:08:24.921111, event: all_read, op: pg_notify(2.2c7(139) epoch 11703) -8741> 2014-09-18 18:08:24.921158 7f4c9f251700 5 -- op tracker -- , seq: 85646, time: 2014-09-18 18:08:24.921135, event: dispatched, op: pg_notify(2.2c7(139) epoch 11703) -8740> 2014-09-18 18:08:24.921164 7f4c9f251700 5 -- op tracker -- , seq: 85646, time: 2014-09-18 18:08:24.921163, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11703) -8739> 2014-09-18 18:08:24.921172 7f4c9f251700 5 -- op tracker -- , seq: 85646, time: 2014-09-18 18:08:24.921171, event: started, op: pg_notify(2.2c7(139) epoch 11703) -8738> 2014-09-18 18:08:24.921190 7f4c9f251700 5 -- op tracker -- , seq: 85646, time: 2014-09-18 18:08:24.921188, event: done, op: pg_notify(2.2c7(139) epoch 11703) -8737> 2014-09-18 18:08:24.921231 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2584 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x19df2f40 con 0x4900b00 -8736> 2014-09-18 18:08:24.921246 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8735> 2014-09-18 18:08:24.921255 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2585 ==== pg_notify(2.71c(12) epoch 11703) v5 ==== 1424+0+0 (1811905584 0 0) 0xf6de960 con 0x4900b00 -8734> 2014-09-18 18:08:24.921271 7f4c9f251700 5 -- op tracker -- , seq: 85647, time: 2014-09-18 18:08:24.921146, event: header_read, op: pg_notify(2.71c(12) epoch 11703) -8733> 2014-09-18 18:08:24.921284 7f4c9f251700 5 -- op tracker -- , seq: 85647, time: 2014-09-18 18:08:24.921146, event: throttled, op: pg_notify(2.71c(12) epoch 11703) -8732> 2014-09-18 18:08:24.921294 7f4c9f251700 5 -- op tracker -- , seq: 85647, time: 2014-09-18 18:08:24.921177, event: all_read, op: pg_notify(2.71c(12) epoch 11703) -8731> 2014-09-18 18:08:24.921305 7f4c9f251700 5 -- op tracker -- , seq: 85647, time: 2014-09-18 18:08:24.921266, event: dispatched, op: pg_notify(2.71c(12) epoch 11703) -8730> 2014-09-18 18:08:24.921313 7f4c9f251700 5 -- op tracker -- , seq: 85647, time: 2014-09-18 18:08:24.921312, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11703) -8729> 2014-09-18 18:08:24.921324 7f4c9f251700 5 -- op tracker -- , seq: 85647, time: 2014-09-18 18:08:24.921323, event: started, op: pg_notify(2.71c(12) epoch 11703) -8728> 2014-09-18 18:08:24.921360 7f4c9f251700 5 -- op tracker -- , seq: 85647, time: 2014-09-18 18:08:24.921358, event: done, op: pg_notify(2.71c(12) epoch 11703) -8727> 2014-09-18 18:08:24.921371 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 911 ==== pg_notify(2.410(11) epoch 11703) v5 ==== 1287+0+0 (3802948368 0 0) 0xe797840 con 0x1395f8c0 -8726> 2014-09-18 18:08:24.921381 7f4c9f251700 5 -- op tracker -- , seq: 85648, time: 2014-09-18 18:08:24.921235, event: header_read, op: pg_notify(2.410(11) epoch 11703) -8725> 2014-09-18 18:08:24.921387 7f4c9f251700 5 -- op tracker -- , seq: 85648, time: 2014-09-18 18:08:24.921237, event: throttled, op: pg_notify(2.410(11) epoch 11703) -8724> 2014-09-18 18:08:24.921393 7f4c9f251700 5 -- op tracker -- , seq: 85648, time: 2014-09-18 18:08:24.921283, event: all_read, op: pg_notify(2.410(11) epoch 11703) -8723> 2014-09-18 18:08:24.921399 7f4c9f251700 5 -- op tracker -- , seq: 85648, time: 2014-09-18 18:08:24.921378, event: dispatched, op: pg_notify(2.410(11) epoch 11703) -8722> 2014-09-18 18:08:24.921405 7f4c9f251700 5 -- op tracker -- , seq: 85648, time: 2014-09-18 18:08:24.921404, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11703) -8721> 2014-09-18 18:08:24.921413 7f4c9f251700 5 -- op tracker -- , seq: 85648, time: 2014-09-18 18:08:24.921411, event: started, op: pg_notify(2.410(11) epoch 11703) -8720> 2014-09-18 18:08:24.921428 7f4c9f251700 5 -- op tracker -- , seq: 85648, time: 2014-09-18 18:08:24.921427, event: done, op: pg_notify(2.410(11) epoch 11703) -8719> 2014-09-18 18:08:24.921949 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1184 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x139a8d00 con 0x1088b440 -8718> 2014-09-18 18:08:24.921969 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8717> 2014-09-18 18:08:24.922394 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1185 ==== pg_notify(2.2d8(173) epoch 11703) v5 ==== 10757+0+0 (3511634240 0 0) 0xe795680 con 0x1088b440 -8716> 2014-09-18 18:08:24.922416 7f4c9f251700 5 -- op tracker -- , seq: 85649, time: 2014-09-18 18:08:24.922206, event: header_read, op: pg_notify(2.2d8(173) epoch 11703) -8715> 2014-09-18 18:08:24.922429 7f4c9f251700 5 -- op tracker -- , seq: 85649, time: 2014-09-18 18:08:24.922206, event: throttled, op: pg_notify(2.2d8(173) epoch 11703) -8714> 2014-09-18 18:08:24.922439 7f4c9f251700 5 -- op tracker -- , seq: 85649, time: 2014-09-18 18:08:24.922378, event: all_read, op: pg_notify(2.2d8(173) epoch 11703) -8713> 2014-09-18 18:08:24.922446 7f4c9f251700 5 -- op tracker -- , seq: 85649, time: 2014-09-18 18:08:24.922410, event: dispatched, op: pg_notify(2.2d8(173) epoch 11703) -8712> 2014-09-18 18:08:24.922456 7f4c9f251700 5 -- op tracker -- , seq: 85649, time: 2014-09-18 18:08:24.922454, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11703) -8711> 2014-09-18 18:08:24.922467 7f4c9f251700 5 -- op tracker -- , seq: 85649, time: 2014-09-18 18:08:24.922465, event: started, op: pg_notify(2.2d8(173) epoch 11703) -8710> 2014-09-18 18:08:24.922494 7f4c9f251700 5 -- op tracker -- , seq: 85649, time: 2014-09-18 18:08:24.922492, event: done, op: pg_notify(2.2d8(173) epoch 11703) -8709> 2014-09-18 18:08:24.923431 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3588 ==== pg_notify(2.3b6(10) epoch 11703) v5 ==== 1260+0+0 (1378060784 0 0) 0xf4f1a20 con 0xcbecec0 -8708> 2014-09-18 18:08:24.923448 7f4c9f251700 5 -- op tracker -- , seq: 85650, time: 2014-09-18 18:08:24.923336, event: header_read, op: pg_notify(2.3b6(10) epoch 11703) -8707> 2014-09-18 18:08:24.923457 7f4c9f251700 5 -- op tracker -- , seq: 85650, time: 2014-09-18 18:08:24.923337, event: throttled, op: pg_notify(2.3b6(10) epoch 11703) -8706> 2014-09-18 18:08:24.923463 7f4c9f251700 5 -- op tracker -- , seq: 85650, time: 2014-09-18 18:08:24.923367, event: all_read, op: pg_notify(2.3b6(10) epoch 11703) -8705> 2014-09-18 18:08:24.923469 7f4c9f251700 5 -- op tracker -- , seq: 85650, time: 2014-09-18 18:08:24.923444, event: dispatched, op: pg_notify(2.3b6(10) epoch 11703) -8704> 2014-09-18 18:08:24.923476 7f4c9f251700 5 -- op tracker -- , seq: 85650, time: 2014-09-18 18:08:24.923474, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11703) -8703> 2014-09-18 18:08:24.923484 7f4c9f251700 5 -- op tracker -- , seq: 85650, time: 2014-09-18 18:08:24.923482, event: started, op: pg_notify(2.3b6(10) epoch 11703) -8702> 2014-09-18 18:08:24.923502 7f4c9f251700 5 -- op tracker -- , seq: 85650, time: 2014-09-18 18:08:24.923501, event: done, op: pg_notify(2.3b6(10) epoch 11703) -8701> 2014-09-18 18:08:24.925167 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 8 ==== pg_notify(2.1a2(20) epoch 11703) v5 ==== 1798+0+0 (2468172093 0 0) 0xe2870e0 con 0x12b3a680 -8700> 2014-09-18 18:08:24.925188 7f4c9f251700 5 -- op tracker -- , seq: 85651, time: 2014-09-18 18:08:24.924250, event: header_read, op: pg_notify(2.1a2(20) epoch 11703) -8699> 2014-09-18 18:08:24.925199 7f4c9f251700 5 -- op tracker -- , seq: 85651, time: 2014-09-18 18:08:24.924251, event: throttled, op: pg_notify(2.1a2(20) epoch 11703) -8698> 2014-09-18 18:08:24.925203 7f4c9f251700 5 -- op tracker -- , seq: 85651, time: 2014-09-18 18:08:24.924291, event: all_read, op: pg_notify(2.1a2(20) epoch 11703) -8697> 2014-09-18 18:08:24.925208 7f4c9f251700 5 -- op tracker -- , seq: 85651, time: 2014-09-18 18:08:24.925183, event: dispatched, op: pg_notify(2.1a2(20) epoch 11703) -8696> 2014-09-18 18:08:24.925212 7f4c9f251700 5 -- op tracker -- , seq: 85651, time: 2014-09-18 18:08:24.925211, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11703) -8695> 2014-09-18 18:08:24.925219 7f4c9f251700 5 -- op tracker -- , seq: 85651, time: 2014-09-18 18:08:24.925218, event: started, op: pg_notify(2.1a2(20) epoch 11703) -8694> 2014-09-18 18:08:24.925243 7f4c9f251700 5 -- op tracker -- , seq: 85651, time: 2014-09-18 18:08:24.925241, event: done, op: pg_notify(2.1a2(20) epoch 11703) -8693> 2014-09-18 18:08:24.925261 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1711 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x10e23d40 con 0x4900000 -8692> 2014-09-18 18:08:24.925275 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8691> 2014-09-18 18:08:24.925282 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 1999 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x1182cf40 con 0xe2d9a20 -8690> 2014-09-18 18:08:24.925290 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8689> 2014-09-18 18:08:24.925294 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1712 ==== pg_notify(2.e0(79) epoch 11703) v5 ==== 5119+0+0 (466078690 0 0) 0xf1eb0e0 con 0x4900000 -8688> 2014-09-18 18:08:24.925304 7f4c9f251700 5 -- op tracker -- , seq: 85652, time: 2014-09-18 18:08:24.924758, event: header_read, op: pg_notify(2.e0(79) epoch 11703) -8687> 2014-09-18 18:08:24.925311 7f4c9f251700 5 -- op tracker -- , seq: 85652, time: 2014-09-18 18:08:24.924759, event: throttled, op: pg_notify(2.e0(79) epoch 11703) -8686> 2014-09-18 18:08:24.925317 7f4c9f251700 5 -- op tracker -- , seq: 85652, time: 2014-09-18 18:08:24.924881, event: all_read, op: pg_notify(2.e0(79) epoch 11703) -8685> 2014-09-18 18:08:24.925323 7f4c9f251700 5 -- op tracker -- , seq: 85652, time: 2014-09-18 18:08:24.925301, event: dispatched, op: pg_notify(2.e0(79) epoch 11703) -8684> 2014-09-18 18:08:24.925330 7f4c9f251700 5 -- op tracker -- , seq: 85652, time: 2014-09-18 18:08:24.925328, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11703) -8683> 2014-09-18 18:08:24.925337 7f4c9f251700 5 -- op tracker -- , seq: 85652, time: 2014-09-18 18:08:24.925335, event: started, op: pg_notify(2.e0(79) epoch 11703) -8682> 2014-09-18 18:08:24.925355 7f4c9f251700 5 -- op tracker -- , seq: 85652, time: 2014-09-18 18:08:24.925353, event: done, op: pg_notify(2.e0(79) epoch 11703) -8681> 2014-09-18 18:08:24.925380 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2000 ==== pg_notify(2.26d(9) epoch 11703) v5 ==== 1257+0+0 (2264663145 0 0) 0xf0b8ee0 con 0xe2d9a20 -8680> 2014-09-18 18:08:24.925390 7f4c9f251700 5 -- op tracker -- , seq: 85653, time: 2014-09-18 18:08:24.925152, event: header_read, op: pg_notify(2.26d(9) epoch 11703) -8679> 2014-09-18 18:08:24.925397 7f4c9f251700 5 -- op tracker -- , seq: 85653, time: 2014-09-18 18:08:24.925152, event: throttled, op: pg_notify(2.26d(9) epoch 11703) -8678> 2014-09-18 18:08:24.925403 7f4c9f251700 5 -- op tracker -- , seq: 85653, time: 2014-09-18 18:08:24.925186, event: all_read, op: pg_notify(2.26d(9) epoch 11703) -8677> 2014-09-18 18:08:24.925409 7f4c9f251700 5 -- op tracker -- , seq: 85653, time: 2014-09-18 18:08:24.925388, event: dispatched, op: pg_notify(2.26d(9) epoch 11703) -8676> 2014-09-18 18:08:24.925416 7f4c9f251700 5 -- op tracker -- , seq: 85653, time: 2014-09-18 18:08:24.925414, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11703) -8675> 2014-09-18 18:08:24.925423 7f4c9f251700 5 -- op tracker -- , seq: 85653, time: 2014-09-18 18:08:24.925421, event: started, op: pg_notify(2.26d(9) epoch 11703) -8674> 2014-09-18 18:08:24.925439 7f4c9f251700 5 -- op tracker -- , seq: 85653, time: 2014-09-18 18:08:24.925438, event: done, op: pg_notify(2.26d(9) epoch 11703) -8673> 2014-09-18 18:08:24.926430 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3589 ==== pg_notify(2.517(11) epoch 11703) v5 ==== 1343+0+0 (500133569 0 0) 0xf4eeb40 con 0xcbecec0 -8672> 2014-09-18 18:08:24.926444 7f4c9f251700 5 -- op tracker -- , seq: 85654, time: 2014-09-18 18:08:24.926372, event: header_read, op: pg_notify(2.517(11) epoch 11703) -8671> 2014-09-18 18:08:24.926452 7f4c9f251700 5 -- op tracker -- , seq: 85654, time: 2014-09-18 18:08:24.926373, event: throttled, op: pg_notify(2.517(11) epoch 11703) -8670> 2014-09-18 18:08:24.926458 7f4c9f251700 5 -- op tracker -- , seq: 85654, time: 2014-09-18 18:08:24.926413, event: all_read, op: pg_notify(2.517(11) epoch 11703) -8669> 2014-09-18 18:08:24.926464 7f4c9f251700 5 -- op tracker -- , seq: 85654, time: 2014-09-18 18:08:24.926440, event: dispatched, op: pg_notify(2.517(11) epoch 11703) -8668> 2014-09-18 18:08:24.926471 7f4c9f251700 5 -- op tracker -- , seq: 85654, time: 2014-09-18 18:08:24.926469, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11703) -8667> 2014-09-18 18:08:24.926478 7f4c9f251700 5 -- op tracker -- , seq: 85654, time: 2014-09-18 18:08:24.926477, event: started, op: pg_notify(2.517(11) epoch 11703) -8666> 2014-09-18 18:08:24.926495 7f4c9f251700 5 -- op tracker -- , seq: 85654, time: 2014-09-18 18:08:24.926493, event: done, op: pg_notify(2.517(11) epoch 11703) -8665> 2014-09-18 18:08:24.926920 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 9 ==== pg_notify(2.78f(153),2.4c6(15) epoch 11703) v5 ==== 10728+0+0 (754904056 0 0) 0x2b92ef00 con 0x12b3a680 -8664> 2014-09-18 18:08:24.926935 7f4c9f251700 5 -- op tracker -- , seq: 85655, time: 2014-09-18 18:08:24.926708, event: header_read, op: pg_notify(2.78f(153),2.4c6(15) epoch 11703) -8663> 2014-09-18 18:08:24.926944 7f4c9f251700 5 -- op tracker -- , seq: 85655, time: 2014-09-18 18:08:24.926709, event: throttled, op: pg_notify(2.78f(153),2.4c6(15) epoch 11703) -8662> 2014-09-18 18:08:24.926950 7f4c9f251700 5 -- op tracker -- , seq: 85655, time: 2014-09-18 18:08:24.926904, event: all_read, op: pg_notify(2.78f(153),2.4c6(15) epoch 11703) -8661> 2014-09-18 18:08:24.926956 7f4c9f251700 5 -- op tracker -- , seq: 85655, time: 2014-09-18 18:08:24.926932, event: dispatched, op: pg_notify(2.78f(153),2.4c6(15) epoch 11703) -8660> 2014-09-18 18:08:24.926963 7f4c9f251700 5 -- op tracker -- , seq: 85655, time: 2014-09-18 18:08:24.926961, event: waiting_for_osdmap, op: pg_notify(2.78f(153),2.4c6(15) epoch 11703) -8659> 2014-09-18 18:08:24.926971 7f4c9f251700 5 -- op tracker -- , seq: 85655, time: 2014-09-18 18:08:24.926969, event: started, op: pg_notify(2.78f(153),2.4c6(15) epoch 11703) -8658> 2014-09-18 18:08:24.927010 7f4c9f251700 5 -- op tracker -- , seq: 85655, time: 2014-09-18 18:08:24.927008, event: done, op: pg_notify(2.78f(153),2.4c6(15) epoch 11703) -8657> 2014-09-18 18:08:24.928215 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3590 ==== pg_notify(2.2c7(139) epoch 11703) v5 ==== 8651+0+0 (4182888650 0 0) 0xf4ee960 con 0xcbecec0 -8656> 2014-09-18 18:08:24.928230 7f4c9f251700 5 -- op tracker -- , seq: 85656, time: 2014-09-18 18:08:24.928079, event: header_read, op: pg_notify(2.2c7(139) epoch 11703) -8655> 2014-09-18 18:08:24.928250 7f4c9f251700 5 -- op tracker -- , seq: 85656, time: 2014-09-18 18:08:24.928080, event: throttled, op: pg_notify(2.2c7(139) epoch 11703) -8654> 2014-09-18 18:08:24.928254 7f4c9f251700 5 -- op tracker -- , seq: 85656, time: 2014-09-18 18:08:24.928200, event: all_read, op: pg_notify(2.2c7(139) epoch 11703) -8653> 2014-09-18 18:08:24.928258 7f4c9f251700 5 -- op tracker -- , seq: 85656, time: 2014-09-18 18:08:24.928226, event: dispatched, op: pg_notify(2.2c7(139) epoch 11703) -8652> 2014-09-18 18:08:24.928263 7f4c9f251700 5 -- op tracker -- , seq: 85656, time: 2014-09-18 18:08:24.928261, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11703) -8651> 2014-09-18 18:08:24.928280 7f4c9f251700 5 -- op tracker -- , seq: 85656, time: 2014-09-18 18:08:24.928278, event: started, op: pg_notify(2.2c7(139) epoch 11703) -8650> 2014-09-18 18:08:24.928301 7f4c9f251700 5 -- op tracker -- , seq: 85656, time: 2014-09-18 18:08:24.928299, event: done, op: pg_notify(2.2c7(139) epoch 11703) -8649> 2014-09-18 18:08:24.928907 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 912 ==== pg_info(1 pgs e11703:2.78f) v4 ==== 738+0+0 (182277239 0 0) 0xe958d00 con 0x1395f8c0 -8648> 2014-09-18 18:08:24.928920 7f4c9f251700 5 -- op tracker -- , seq: 85657, time: 2014-09-18 18:08:24.928854, event: header_read, op: pg_info(1 pgs e11703:2.78f) -8647> 2014-09-18 18:08:24.928939 7f4c9f251700 5 -- op tracker -- , seq: 85657, time: 2014-09-18 18:08:24.928855, event: throttled, op: pg_info(1 pgs e11703:2.78f) -8646> 2014-09-18 18:08:24.928944 7f4c9f251700 5 -- op tracker -- , seq: 85657, time: 2014-09-18 18:08:24.928888, event: all_read, op: pg_info(1 pgs e11703:2.78f) -8645> 2014-09-18 18:08:24.928948 7f4c9f251700 5 -- op tracker -- , seq: 85657, time: 2014-09-18 18:08:24.928917, event: dispatched, op: pg_info(1 pgs e11703:2.78f) -8644> 2014-09-18 18:08:24.928952 7f4c9f251700 5 -- op tracker -- , seq: 85657, time: 2014-09-18 18:08:24.928951, event: waiting_for_osdmap, op: pg_info(1 pgs e11703:2.78f) -8643> 2014-09-18 18:08:24.928960 7f4c9f251700 5 -- op tracker -- , seq: 85657, time: 2014-09-18 18:08:24.928958, event: started, op: pg_info(1 pgs e11703:2.78f) -8642> 2014-09-18 18:08:24.928978 7f4c9f251700 5 -- op tracker -- , seq: 85657, time: 2014-09-18 18:08:24.928976, event: done, op: pg_info(1 pgs e11703:2.78f) -8641> 2014-09-18 18:08:24.950546 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6884/38815 10 ==== pg_info(1 pgs e11703:2.78f) v4 ==== 738+0+0 (182277239 0 0) 0xe794f00 con 0x12b3a680 -8640> 2014-09-18 18:08:24.950588 7f4c9f251700 5 -- op tracker -- , seq: 85658, time: 2014-09-18 18:08:24.950443, event: header_read, op: pg_info(1 pgs e11703:2.78f) -8639> 2014-09-18 18:08:24.950598 7f4c9f251700 5 -- op tracker -- , seq: 85658, time: 2014-09-18 18:08:24.950457, event: throttled, op: pg_info(1 pgs e11703:2.78f) -8638> 2014-09-18 18:08:24.950602 7f4c9f251700 5 -- op tracker -- , seq: 85658, time: 2014-09-18 18:08:24.950511, event: all_read, op: pg_info(1 pgs e11703:2.78f) -8637> 2014-09-18 18:08:24.950607 7f4c9f251700 5 -- op tracker -- , seq: 85658, time: 2014-09-18 18:08:24.950581, event: dispatched, op: pg_info(1 pgs e11703:2.78f) -8636> 2014-09-18 18:08:24.950612 7f4c9f251700 5 -- op tracker -- , seq: 85658, time: 2014-09-18 18:08:24.950611, event: waiting_for_osdmap, op: pg_info(1 pgs e11703:2.78f) -8635> 2014-09-18 18:08:24.950619 7f4c9f251700 5 -- op tracker -- , seq: 85658, time: 2014-09-18 18:08:24.950618, event: started, op: pg_info(1 pgs e11703:2.78f) -8634> 2014-09-18 18:08:24.950847 7f4c9f251700 5 -- op tracker -- , seq: 85658, time: 2014-09-18 18:08:24.950845, event: done, op: pg_info(1 pgs e11703:2.78f) -8633> 2014-09-18 18:08:24.965585 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5054 ==== pg_notify(2.517(11) epoch 11701) v5 ==== 1335+0+0 (877857142 0 0) 0xfa26780 con 0x1088aec0 -8632> 2014-09-18 18:08:24.965604 7f4c9f251700 5 -- op tracker -- , seq: 85659, time: 2014-09-18 18:08:24.965450, event: header_read, op: pg_notify(2.517(11) epoch 11701) -8631> 2014-09-18 18:08:24.965625 7f4c9f251700 5 -- op tracker -- , seq: 85659, time: 2014-09-18 18:08:24.965453, event: throttled, op: pg_notify(2.517(11) epoch 11701) -8630> 2014-09-18 18:08:24.965630 7f4c9f251700 5 -- op tracker -- , seq: 85659, time: 2014-09-18 18:08:24.965525, event: all_read, op: pg_notify(2.517(11) epoch 11701) -8629> 2014-09-18 18:08:24.965634 7f4c9f251700 5 -- op tracker -- , seq: 85659, time: 2014-09-18 18:08:24.965599, event: dispatched, op: pg_notify(2.517(11) epoch 11701) -8628> 2014-09-18 18:08:24.965657 7f4c9f251700 5 -- op tracker -- , seq: 85659, time: 2014-09-18 18:08:24.965655, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11701) -8627> 2014-09-18 18:08:24.965669 7f4c9f251700 5 -- op tracker -- , seq: 85659, time: 2014-09-18 18:08:24.965668, event: started, op: pg_notify(2.517(11) epoch 11701) -8626> 2014-09-18 18:08:24.965692 7f4c9f251700 5 -- op tracker -- , seq: 85659, time: 2014-09-18 18:08:24.965690, event: done, op: pg_notify(2.517(11) epoch 11701) -8625> 2014-09-18 18:08:24.968601 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3591 ==== pg_notify(2.40a(162) epoch 11703) v5 ==== 9736+0+0 (4062519330 0 0) 0xf4ee000 con 0xcbecec0 -8624> 2014-09-18 18:08:24.968637 7f4c9f251700 5 -- op tracker -- , seq: 85660, time: 2014-09-18 18:08:24.968400, event: header_read, op: pg_notify(2.40a(162) epoch 11703) -8623> 2014-09-18 18:08:24.968649 7f4c9f251700 5 -- op tracker -- , seq: 85660, time: 2014-09-18 18:08:24.968402, event: throttled, op: pg_notify(2.40a(162) epoch 11703) -8622> 2014-09-18 18:08:24.968656 7f4c9f251700 5 -- op tracker -- , seq: 85660, time: 2014-09-18 18:08:24.968586, event: all_read, op: pg_notify(2.40a(162) epoch 11703) -8621> 2014-09-18 18:08:24.968662 7f4c9f251700 5 -- op tracker -- , seq: 85660, time: 2014-09-18 18:08:24.968631, event: dispatched, op: pg_notify(2.40a(162) epoch 11703) -8620> 2014-09-18 18:08:24.968669 7f4c9f251700 5 -- op tracker -- , seq: 85660, time: 2014-09-18 18:08:24.968667, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11703) -8619> 2014-09-18 18:08:24.968679 7f4c9f251700 5 -- op tracker -- , seq: 85660, time: 2014-09-18 18:08:24.968677, event: started, op: pg_notify(2.40a(162) epoch 11703) -8618> 2014-09-18 18:08:24.968701 7f4c9f251700 5 -- op tracker -- , seq: 85660, time: 2014-09-18 18:08:24.968699, event: done, op: pg_notify(2.40a(162) epoch 11703) -8617> 2014-09-18 18:08:24.969361 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3140 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x139a6d80 con 0x1395ec60 -8616> 2014-09-18 18:08:24.969377 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11703,11703], i have 11703, src has [1,11703] -8615> 2014-09-18 18:08:24.969410 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3141 ==== pg_notify(2.71(19) epoch 11703) v5 ==== 1795+0+0 (1176093192 0 0) 0xf2b8760 con 0x1395ec60 -8614> 2014-09-18 18:08:24.969420 7f4c9f251700 5 -- op tracker -- , seq: 85661, time: 2014-09-18 18:08:24.969368, event: header_read, op: pg_notify(2.71(19) epoch 11703) -8613> 2014-09-18 18:08:24.969429 7f4c9f251700 5 -- op tracker -- , seq: 85661, time: 2014-09-18 18:08:24.969368, event: throttled, op: pg_notify(2.71(19) epoch 11703) -8612> 2014-09-18 18:08:24.969434 7f4c9f251700 5 -- op tracker -- , seq: 85661, time: 2014-09-18 18:08:24.969400, event: all_read, op: pg_notify(2.71(19) epoch 11703) -8611> 2014-09-18 18:08:24.969439 7f4c9f251700 5 -- op tracker -- , seq: 85661, time: 2014-09-18 18:08:24.969418, event: dispatched, op: pg_notify(2.71(19) epoch 11703) -8610> 2014-09-18 18:08:24.969446 7f4c9f251700 5 -- op tracker -- , seq: 85661, time: 2014-09-18 18:08:24.969444, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11703) -8609> 2014-09-18 18:08:24.969453 7f4c9f251700 5 -- op tracker -- , seq: 85661, time: 2014-09-18 18:08:24.969452, event: started, op: pg_notify(2.71(19) epoch 11703) -8608> 2014-09-18 18:08:24.969471 7f4c9f251700 5 -- op tracker -- , seq: 85661, time: 2014-09-18 18:08:24.969469, event: done, op: pg_notify(2.71(19) epoch 11703) -8607> 2014-09-18 18:08:24.969761 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3592 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11703) v5 ==== 6174+0+0 (2468550970 0 0) 0xf4efe00 con 0xcbecec0 -8606> 2014-09-18 18:08:24.969775 7f4c9f251700 5 -- op tracker -- , seq: 85662, time: 2014-09-18 18:08:24.969671, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11703) -8605> 2014-09-18 18:08:24.969783 7f4c9f251700 5 -- op tracker -- , seq: 85662, time: 2014-09-18 18:08:24.969672, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11703) -8604> 2014-09-18 18:08:24.969790 7f4c9f251700 5 -- op tracker -- , seq: 85662, time: 2014-09-18 18:08:24.969749, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11703) -8603> 2014-09-18 18:08:24.969797 7f4c9f251700 5 -- op tracker -- , seq: 85662, time: 2014-09-18 18:08:24.969771, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11703) -8602> 2014-09-18 18:08:24.969803 7f4c9f251700 5 -- op tracker -- , seq: 85662, time: 2014-09-18 18:08:24.969801, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11703) -8601> 2014-09-18 18:08:24.969811 7f4c9f251700 5 -- op tracker -- , seq: 85662, time: 2014-09-18 18:08:24.969809, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11703) -8600> 2014-09-18 18:08:24.969852 7f4c9f251700 5 -- op tracker -- , seq: 85662, time: 2014-09-18 18:08:24.969850, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11703) -8599> 2014-09-18 18:08:24.970572 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3593 ==== pg_notify(2.1bb(9) epoch 11703) v5 ==== 1189+0+0 (2116583872 0 0) 0xf1ea780 con 0xcbecec0 -8598> 2014-09-18 18:08:24.970614 7f4c9f251700 5 -- op tracker -- , seq: 85663, time: 2014-09-18 18:08:24.970523, event: header_read, op: pg_notify(2.1bb(9) epoch 11703) -8597> 2014-09-18 18:08:24.970623 7f4c9f251700 5 -- op tracker -- , seq: 85663, time: 2014-09-18 18:08:24.970524, event: throttled, op: pg_notify(2.1bb(9) epoch 11703) -8596> 2014-09-18 18:08:24.970628 7f4c9f251700 5 -- op tracker -- , seq: 85663, time: 2014-09-18 18:08:24.970560, event: all_read, op: pg_notify(2.1bb(9) epoch 11703) -8595> 2014-09-18 18:08:24.970634 7f4c9f251700 5 -- op tracker -- , seq: 85663, time: 2014-09-18 18:08:24.970610, event: dispatched, op: pg_notify(2.1bb(9) epoch 11703) -8594> 2014-09-18 18:08:24.970640 7f4c9f251700 5 -- op tracker -- , seq: 85663, time: 2014-09-18 18:08:24.970638, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11703) -8593> 2014-09-18 18:08:24.970647 7f4c9f251700 5 -- op tracker -- , seq: 85663, time: 2014-09-18 18:08:24.970646, event: started, op: pg_notify(2.1bb(9) epoch 11703) -8592> 2014-09-18 18:08:24.970664 7f4c9f251700 5 -- op tracker -- , seq: 85663, time: 2014-09-18 18:08:24.970663, event: done, op: pg_notify(2.1bb(9) epoch 11703) -8591> 2014-09-18 18:08:24.971676 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3594 ==== pg_notify(2.78f(54) epoch 11703) v5 ==== 3616+0+0 (1439037366 0 0) 0xf1ea1e0 con 0xcbecec0 -8590> 2014-09-18 18:08:24.971690 7f4c9f251700 5 -- op tracker -- , seq: 85664, time: 2014-09-18 18:08:24.971535, event: header_read, op: pg_notify(2.78f(54) epoch 11703) -8589> 2014-09-18 18:08:24.971699 7f4c9f251700 5 -- op tracker -- , seq: 85664, time: 2014-09-18 18:08:24.971536, event: throttled, op: pg_notify(2.78f(54) epoch 11703) -8588> 2014-09-18 18:08:24.971705 7f4c9f251700 5 -- op tracker -- , seq: 85664, time: 2014-09-18 18:08:24.971663, event: all_read, op: pg_notify(2.78f(54) epoch 11703) -8587> 2014-09-18 18:08:24.971711 7f4c9f251700 5 -- op tracker -- , seq: 85664, time: 2014-09-18 18:08:24.971687, event: dispatched, op: pg_notify(2.78f(54) epoch 11703) -8586> 2014-09-18 18:08:24.971717 7f4c9f251700 5 -- op tracker -- , seq: 85664, time: 2014-09-18 18:08:24.971716, event: waiting_for_osdmap, op: pg_notify(2.78f(54) epoch 11703) -8585> 2014-09-18 18:08:24.971725 7f4c9f251700 5 -- op tracker -- , seq: 85664, time: 2014-09-18 18:08:24.971723, event: started, op: pg_notify(2.78f(54) epoch 11703) -8584> 2014-09-18 18:08:24.971747 7f4c9f251700 5 -- op tracker -- , seq: 85664, time: 2014-09-18 18:08:24.971746, event: done, op: pg_notify(2.78f(54) epoch 11703) -8583> 2014-09-18 18:08:24.975161 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3142 ==== pg_info(1 pgs e11703:2.3b5(2)) v4 ==== 860+0+0 (1036496118 0 0) 0x130412c0 con 0x1395ec60 -8582> 2014-09-18 18:08:24.975176 7f4c9f251700 5 -- op tracker -- , seq: 85665, time: 2014-09-18 18:08:24.975105, event: header_read, op: pg_info(1 pgs e11703:2.3b5(2)) -8581> 2014-09-18 18:08:24.975184 7f4c9f251700 5 -- op tracker -- , seq: 85665, time: 2014-09-18 18:08:24.975106, event: throttled, op: pg_info(1 pgs e11703:2.3b5(2)) -8580> 2014-09-18 18:08:24.975190 7f4c9f251700 5 -- op tracker -- , seq: 85665, time: 2014-09-18 18:08:24.975136, event: all_read, op: pg_info(1 pgs e11703:2.3b5(2)) -8579> 2014-09-18 18:08:24.975196 7f4c9f251700 5 -- op tracker -- , seq: 85665, time: 2014-09-18 18:08:24.975172, event: dispatched, op: pg_info(1 pgs e11703:2.3b5(2)) -8578> 2014-09-18 18:08:24.975201 7f4c9f251700 5 -- op tracker -- , seq: 85665, time: 2014-09-18 18:08:24.975199, event: waiting_for_osdmap, op: pg_info(1 pgs e11703:2.3b5(2)) -8577> 2014-09-18 18:08:24.975207 7f4c9f251700 5 -- op tracker -- , seq: 85665, time: 2014-09-18 18:08:24.975205, event: started, op: pg_info(1 pgs e11703:2.3b5(2)) -8576> 2014-09-18 18:08:24.975222 7f4c9f251700 5 -- op tracker -- , seq: 85665, time: 2014-09-18 18:08:24.975220, event: done, op: pg_info(1 pgs e11703:2.3b5(2)) -8575> 2014-09-18 18:08:24.975231 7f4c97241700 5 osd.42 pg_epoch: 11703 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] exit Started/Stray 1.059394 7 0.000709 -8574> 2014-09-18 18:08:24.975253 7f4c97241700 5 osd.42 pg_epoch: 11703 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -8573> 2014-09-18 18:08:24.975282 7f4c97241700 5 osd.42 pg_epoch: 11703 pg[2.3b5( v 6760'24794 (4083'21161,6760'24794] local-les=11491 n=522 ec=188 les/c 11491/11491 11701/11702/11702) [24,27,42]/[9,27,42] r=2 lpr=11702 pi=2630-11701/65 crt=6750'24785 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -8572> 2014-09-18 18:08:24.976614 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11703:2.3b5) v4 -- ?+0 0xf160d00 con 0x1395ec60 -8571> 2014-09-18 18:08:24.979634 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3143 ==== pg_notify(2.410(11) epoch 11703) v5 ==== 1325+0+0 (4020947656 0 0) 0xf160d00 con 0x1395ec60 -8570> 2014-09-18 18:08:24.979653 7f4c9f251700 5 -- op tracker -- , seq: 85666, time: 2014-09-18 18:08:24.979515, event: header_read, op: pg_notify(2.410(11) epoch 11703) -8569> 2014-09-18 18:08:24.979661 7f4c9f251700 5 -- op tracker -- , seq: 85666, time: 2014-09-18 18:08:24.979516, event: throttled, op: pg_notify(2.410(11) epoch 11703) -8568> 2014-09-18 18:08:24.979683 7f4c9f251700 5 -- op tracker -- , seq: 85666, time: 2014-09-18 18:08:24.979566, event: all_read, op: pg_notify(2.410(11) epoch 11703) -8567> 2014-09-18 18:08:24.979687 7f4c9f251700 5 -- op tracker -- , seq: 85666, time: 2014-09-18 18:08:24.979649, event: dispatched, op: pg_notify(2.410(11) epoch 11703) -8566> 2014-09-18 18:08:24.979692 7f4c9f251700 5 -- op tracker -- , seq: 85666, time: 2014-09-18 18:08:24.979690, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11703) -8565> 2014-09-18 18:08:24.979707 7f4c9f251700 5 -- op tracker -- , seq: 85666, time: 2014-09-18 18:08:24.979706, event: started, op: pg_notify(2.410(11) epoch 11703) -8564> 2014-09-18 18:08:24.979732 7f4c9f251700 5 -- op tracker -- , seq: 85666, time: 2014-09-18 18:08:24.979731, event: done, op: pg_notify(2.410(11) epoch 11703) -8563> 2014-09-18 18:08:24.981460 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3144 ==== pg_notify(2.74e(11) epoch 11703) v5 ==== 1303+0+0 (482774630 0 0) 0x1368d680 con 0x1395ec60 -8562> 2014-09-18 18:08:24.981479 7f4c9f251700 5 -- op tracker -- , seq: 85667, time: 2014-09-18 18:08:24.981390, event: header_read, op: pg_notify(2.74e(11) epoch 11703) -8561> 2014-09-18 18:08:24.981488 7f4c9f251700 5 -- op tracker -- , seq: 85667, time: 2014-09-18 18:08:24.981391, event: throttled, op: pg_notify(2.74e(11) epoch 11703) -8560> 2014-09-18 18:08:24.981492 7f4c9f251700 5 -- op tracker -- , seq: 85667, time: 2014-09-18 18:08:24.981423, event: all_read, op: pg_notify(2.74e(11) epoch 11703) -8559> 2014-09-18 18:08:24.981496 7f4c9f251700 5 -- op tracker -- , seq: 85667, time: 2014-09-18 18:08:24.981475, event: dispatched, op: pg_notify(2.74e(11) epoch 11703) -8558> 2014-09-18 18:08:24.981500 7f4c9f251700 5 -- op tracker -- , seq: 85667, time: 2014-09-18 18:08:24.981499, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11703) -8557> 2014-09-18 18:08:24.981514 7f4c9f251700 5 -- op tracker -- , seq: 85667, time: 2014-09-18 18:08:24.981512, event: started, op: pg_notify(2.74e(11) epoch 11703) -8556> 2014-09-18 18:08:24.981531 7f4c9f251700 5 -- op tracker -- , seq: 85667, time: 2014-09-18 18:08:24.981529, event: done, op: pg_notify(2.74e(11) epoch 11703) -8555> 2014-09-18 18:08:24.985249 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3145 ==== pg_notify(2.186(12) epoch 11703) v5 ==== 1386+0+0 (4219269959 0 0) 0xf91c760 con 0x1395ec60 -8554> 2014-09-18 18:08:24.985268 7f4c9f251700 5 -- op tracker -- , seq: 85668, time: 2014-09-18 18:08:24.985162, event: header_read, op: pg_notify(2.186(12) epoch 11703) -8553> 2014-09-18 18:08:24.985276 7f4c9f251700 5 -- op tracker -- , seq: 85668, time: 2014-09-18 18:08:24.985162, event: throttled, op: pg_notify(2.186(12) epoch 11703) -8552> 2014-09-18 18:08:24.985280 7f4c9f251700 5 -- op tracker -- , seq: 85668, time: 2014-09-18 18:08:24.985227, event: all_read, op: pg_notify(2.186(12) epoch 11703) -8551> 2014-09-18 18:08:24.985284 7f4c9f251700 5 -- op tracker -- , seq: 85668, time: 2014-09-18 18:08:24.985263, event: dispatched, op: pg_notify(2.186(12) epoch 11703) -8550> 2014-09-18 18:08:24.985289 7f4c9f251700 5 -- op tracker -- , seq: 85668, time: 2014-09-18 18:08:24.985287, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11703) -8549> 2014-09-18 18:08:24.985295 7f4c9f251700 5 -- op tracker -- , seq: 85668, time: 2014-09-18 18:08:24.985293, event: started, op: pg_notify(2.186(12) epoch 11703) -8548> 2014-09-18 18:08:24.985320 7f4c9f251700 5 -- op tracker -- , seq: 85668, time: 2014-09-18 18:08:24.985319, event: done, op: pg_notify(2.186(12) epoch 11703) -8547> 2014-09-18 18:08:24.985540 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3146 ==== pg_notify(2.4b1(13) epoch 11703) v5 ==== 1441+0+0 (1970999695 0 0) 0xf7310c0 con 0x1395ec60 -8546> 2014-09-18 18:08:24.985552 7f4c9f251700 5 -- op tracker -- , seq: 85669, time: 2014-09-18 18:08:24.985494, event: header_read, op: pg_notify(2.4b1(13) epoch 11703) -8545> 2014-09-18 18:08:24.985565 7f4c9f251700 5 -- op tracker -- , seq: 85669, time: 2014-09-18 18:08:24.985495, event: throttled, op: pg_notify(2.4b1(13) epoch 11703) -8544> 2014-09-18 18:08:24.985569 7f4c9f251700 5 -- op tracker -- , seq: 85669, time: 2014-09-18 18:08:24.985528, event: all_read, op: pg_notify(2.4b1(13) epoch 11703) -8543> 2014-09-18 18:08:24.985578 7f4c9f251700 5 -- op tracker -- , seq: 85669, time: 2014-09-18 18:08:24.985550, event: dispatched, op: pg_notify(2.4b1(13) epoch 11703) -8542> 2014-09-18 18:08:24.985582 7f4c9f251700 5 -- op tracker -- , seq: 85669, time: 2014-09-18 18:08:24.985581, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11703) -8541> 2014-09-18 18:08:24.985592 7f4c9f251700 5 -- op tracker -- , seq: 85669, time: 2014-09-18 18:08:24.985590, event: started, op: pg_notify(2.4b1(13) epoch 11703) -8540> 2014-09-18 18:08:24.985609 7f4c9f251700 5 -- op tracker -- , seq: 85669, time: 2014-09-18 18:08:24.985608, event: done, op: pg_notify(2.4b1(13) epoch 11703) -8539> 2014-09-18 18:08:24.986278 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3147 ==== pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11703) v5 ==== 6445+0+0 (2842586409 0 0) 0xf810ee0 con 0x1395ec60 -8538> 2014-09-18 18:08:24.986299 7f4c9f251700 5 -- op tracker -- , seq: 85670, time: 2014-09-18 18:08:24.986160, event: header_read, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11703) -8537> 2014-09-18 18:08:24.986341 7f4c9f251700 5 -- op tracker -- , seq: 85670, time: 2014-09-18 18:08:24.986160, event: throttled, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11703) -8536> 2014-09-18 18:08:24.986346 7f4c9f251700 5 -- op tracker -- , seq: 85670, time: 2014-09-18 18:08:24.986242, event: all_read, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11703) -8535> 2014-09-18 18:08:24.986353 7f4c9f251700 5 -- op tracker -- , seq: 85670, time: 2014-09-18 18:08:24.986294, event: dispatched, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11703) -8534> 2014-09-18 18:08:24.986366 7f4c9f251700 5 -- op tracker -- , seq: 85670, time: 2014-09-18 18:08:24.986364, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11703) -8533> 2014-09-18 18:08:24.986375 7f4c9f251700 5 -- op tracker -- , seq: 85670, time: 2014-09-18 18:08:24.986373, event: started, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11703) -8532> 2014-09-18 18:08:24.986444 7f4c9f251700 5 -- op tracker -- , seq: 85670, time: 2014-09-18 18:08:24.986442, event: done, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11703) -8531> 2014-09-18 18:08:24.987316 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3148 ==== pg_notify(2.799(9),2.4d0(10) epoch 11703) v5 ==== 2509+0+0 (3394740035 0 0) 0xf80e780 con 0x1395ec60 -8530> 2014-09-18 18:08:24.987336 7f4c9f251700 5 -- op tracker -- , seq: 85671, time: 2014-09-18 18:08:24.987233, event: header_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11703) -8529> 2014-09-18 18:08:24.987366 7f4c9f251700 5 -- op tracker -- , seq: 85671, time: 2014-09-18 18:08:24.987233, event: throttled, op: pg_notify(2.799(9),2.4d0(10) epoch 11703) -8528> 2014-09-18 18:08:24.987371 7f4c9f251700 5 -- op tracker -- , seq: 85671, time: 2014-09-18 18:08:24.987266, event: all_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11703) -8527> 2014-09-18 18:08:24.987378 7f4c9f251700 5 -- op tracker -- , seq: 85671, time: 2014-09-18 18:08:24.987332, event: dispatched, op: pg_notify(2.799(9),2.4d0(10) epoch 11703) -8526> 2014-09-18 18:08:24.987384 7f4c9f251700 5 -- op tracker -- , seq: 85671, time: 2014-09-18 18:08:24.987381, event: waiting_for_osdmap, op: pg_notify(2.799(9),2.4d0(10) epoch 11703) -8525> 2014-09-18 18:08:24.987392 7f4c9f251700 5 -- op tracker -- , seq: 85671, time: 2014-09-18 18:08:24.987390, event: started, op: pg_notify(2.799(9),2.4d0(10) epoch 11703) -8524> 2014-09-18 18:08:24.987434 7f4c9f251700 5 -- op tracker -- , seq: 85671, time: 2014-09-18 18:08:24.987432, event: done, op: pg_notify(2.799(9),2.4d0(10) epoch 11703) -8523> 2014-09-18 18:08:25.007222 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3149 ==== pg_info(1 pgs e11703:2.3b5) v4 ==== 742+0+0 (3459908903 0 0) 0xf80fa40 con 0x1395ec60 -8522> 2014-09-18 18:08:25.007247 7f4c9f251700 5 -- op tracker -- , seq: 85672, time: 2014-09-18 18:08:25.007138, event: header_read, op: pg_info(1 pgs e11703:2.3b5) -8521> 2014-09-18 18:08:25.007283 7f4c9f251700 5 -- op tracker -- , seq: 85672, time: 2014-09-18 18:08:25.007140, event: throttled, op: pg_info(1 pgs e11703:2.3b5) -8520> 2014-09-18 18:08:25.007289 7f4c9f251700 5 -- op tracker -- , seq: 85672, time: 2014-09-18 18:08:25.007186, event: all_read, op: pg_info(1 pgs e11703:2.3b5) -8519> 2014-09-18 18:08:25.007303 7f4c9f251700 5 -- op tracker -- , seq: 85672, time: 2014-09-18 18:08:25.007241, event: dispatched, op: pg_info(1 pgs e11703:2.3b5) -8518> 2014-09-18 18:08:25.007318 7f4c9f251700 5 -- op tracker -- , seq: 85672, time: 2014-09-18 18:08:25.007316, event: waiting_for_osdmap, op: pg_info(1 pgs e11703:2.3b5) -8517> 2014-09-18 18:08:25.007326 7f4c9f251700 5 -- op tracker -- , seq: 85672, time: 2014-09-18 18:08:25.007324, event: started, op: pg_info(1 pgs e11703:2.3b5) -8516> 2014-09-18 18:08:25.007349 7f4c9f251700 5 -- op tracker -- , seq: 85672, time: 2014-09-18 18:08:25.007348, event: done, op: pg_info(1 pgs e11703:2.3b5) -8515> 2014-09-18 18:08:25.056333 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3595 ==== pg_info(1 pgs e11703:2.78f) v4 ==== 738+0+0 (2995097951 0 0) 0xeb7a580 con 0xcbecec0 -8514> 2014-09-18 18:08:25.056357 7f4c9f251700 5 -- op tracker -- , seq: 85673, time: 2014-09-18 18:08:25.056258, event: header_read, op: pg_info(1 pgs e11703:2.78f) -8513> 2014-09-18 18:08:25.056376 7f4c9f251700 5 -- op tracker -- , seq: 85673, time: 2014-09-18 18:08:25.056260, event: throttled, op: pg_info(1 pgs e11703:2.78f) -8512> 2014-09-18 18:08:25.056381 7f4c9f251700 5 -- op tracker -- , seq: 85673, time: 2014-09-18 18:08:25.056309, event: all_read, op: pg_info(1 pgs e11703:2.78f) -8511> 2014-09-18 18:08:25.056385 7f4c9f251700 5 -- op tracker -- , seq: 85673, time: 2014-09-18 18:08:25.056351, event: dispatched, op: pg_info(1 pgs e11703:2.78f) -8510> 2014-09-18 18:08:25.056392 7f4c9f251700 5 -- op tracker -- , seq: 85673, time: 2014-09-18 18:08:25.056389, event: waiting_for_osdmap, op: pg_info(1 pgs e11703:2.78f) -8509> 2014-09-18 18:08:25.056407 7f4c9f251700 5 -- op tracker -- , seq: 85673, time: 2014-09-18 18:08:25.056405, event: started, op: pg_info(1 pgs e11703:2.78f) -8508> 2014-09-18 18:08:25.056432 7f4c9f251700 5 -- op tracker -- , seq: 85673, time: 2014-09-18 18:08:25.056431, event: done, op: pg_info(1 pgs e11703:2.78f) -8507> 2014-09-18 18:08:25.056462 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6884/38815 -- pg_info(1 pgs e11703:2.78f) v4 -- ?+0 0xeb7b480 con 0x12b3a680 -8506> 2014-09-18 18:08:25.056488 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11703:2.78f) v4 -- ?+0 0xeb7b0c0 con 0x1395f8c0 -8505> 2014-09-18 18:08:25.056505 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11703:2.78f) v4 -- ?+0 0xeb7b660 con 0xcbecec0 -8504> 2014-09-18 18:08:25.056582 7f4c97241700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=1,27 from 0//0//-1 -8503> 2014-09-18 18:08:25.056708 7f4c97241700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 1 from 0//0//-1 -8502> 2014-09-18 18:08:25.056730 7f4c97241700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 27 from 0//0//-1 -8501> 2014-09-18 18:08:25.056751 7f4c97241700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 0.145422 11 0.001140 -8500> 2014-09-18 18:08:25.056766 7f4c97241700 5 osd.42 pg_epoch: 11703 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -8499> 2014-09-18 18:08:25.144689 7f4c7f55f700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3a680).reader couldn't read tag, (0) Success -8498> 2014-09-18 18:08:25.144727 7f4c7f55f700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3a680).fault (0) Success -8497> 2014-09-18 18:08:25.144743 7f4c7f55f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3a680).fault, initiating reconnect -8496> 2014-09-18 18:08:25.144853 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).connect error 10.242.42.164:6884/38815, (111) Connection refused -8495> 2014-09-18 18:08:25.144874 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).fault (111) Connection refused -8494> 2014-09-18 18:08:25.144892 7f4c775e0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).fault -8493> 2014-09-18 18:08:25.144945 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).connect error 10.242.42.164:6884/38815, (111) Connection refused -8492> 2014-09-18 18:08:25.144960 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).fault (111) Connection refused -8491> 2014-09-18 18:08:25.145127 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0x14b50c00 sd=127 :42788 s=2 pgs=27 cs=1 l=1 c=0x1d7c92e0).reader couldn't read tag, (0) Success -8490> 2014-09-18 18:08:25.145195 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0x14b50c00 sd=127 :42788 s=2 pgs=27 cs=1 l=1 c=0x1d7c92e0).fault (0) Success -8489> 2014-09-18 18:08:25.145191 7f4ca853b700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0x13d4e840 sd=102 :35093 s=2 pgs=27 cs=1 l=1 c=0x1d7c78c0).reader couldn't read tag, (0) Success -8488> 2014-09-18 18:08:25.145224 7f4ca853b700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0x13d4e840 sd=102 :35093 s=2 pgs=27 cs=1 l=1 c=0x1d7c78c0).fault (0) Success -8487> 2014-09-18 18:08:25.145242 7f4c9da4e700 1 -- 10.242.42.178:0/2483 mark_down 0x1d7c78c0 -- 0x13d4e840 -8486> 2014-09-18 18:08:25.145357 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).connect error 10.242.42.164:6885/38815, (111) Connection refused -8485> 2014-09-18 18:08:25.145376 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=167 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).connect error 10.242.42.164:6886/38815, (111) Connection refused -8484> 2014-09-18 18:08:25.145407 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=167 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault (111) Connection refused -8483> 2014-09-18 18:08:25.145404 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault (111) Connection refused -8482> 2014-09-18 18:08:25.145416 7f4c84bb5700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=167 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault -8481> 2014-09-18 18:08:25.145417 7f4c82b95700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault -8480> 2014-09-18 18:08:25.145467 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).connect error 10.242.42.164:6886/38815, (111) Connection refused -8479> 2014-09-18 18:08:25.145484 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault (111) Connection refused -8478> 2014-09-18 18:08:25.145473 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).connect error 10.242.42.164:6885/38815, (111) Connection refused -8477> 2014-09-18 18:08:25.145495 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault (111) Connection refused -8476> 2014-09-18 18:08:25.194560 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.46 10.242.42.178:0/4507 116 ==== osd_ping(ping e11703 stamp 2014-09-18 18:08:25.193920) v2 ==== 47+0+0 (3088513171 0 0) 0xf4ef680 con 0x10a5c3c0 -8475> 2014-09-18 18:08:25.194566 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.46 10.242.42.178:0/4507 116 ==== osd_ping(ping e11703 stamp 2014-09-18 18:08:25.193920) v2 ==== 47+0+0 (3088513171 0 0) 0xf2b5480 con 0x110bb5a0 -8474> 2014-09-18 18:08:25.194614 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/4507 -- osd_ping(ping_reply e11703 stamp 2014-09-18 18:08:25.193920) v2 -- ?+0 0x12138f00 con 0x10a5c3c0 -8473> 2014-09-18 18:08:25.194645 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/4507 -- osd_ping(ping_reply e11703 stamp 2014-09-18 18:08:25.193920) v2 -- ?+0 0xde5f0e0 con 0x110bb5a0 -8472> 2014-09-18 18:08:25.255206 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.45 10.242.42.178:0/7086 104 ==== osd_ping(ping e11703 stamp 2014-09-18 18:08:25.254564) v2 ==== 47+0+0 (1146715058 0 0) 0xfccab20 con 0x110bb700 -8471> 2014-09-18 18:08:25.255211 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.45 10.242.42.178:0/7086 104 ==== osd_ping(ping e11703 stamp 2014-09-18 18:08:25.254564) v2 ==== 47+0+0 (1146715058 0 0) 0x1173a940 con 0x10a5d9c0 -8470> 2014-09-18 18:08:25.255242 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11703 stamp 2014-09-18 18:08:25.254564) v2 -- ?+0 0xf2b5480 con 0x110bb700 -8469> 2014-09-18 18:08:25.255273 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11703 stamp 2014-09-18 18:08:25.254564) v2 -- ?+0 0xf4ef680 con 0x10a5d9c0 -8468> 2014-09-18 18:08:25.276268 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1267 ==== pg_notify(2.4f6(15) epoch 11700) v5 ==== 1543+0+0 (3694782794 0 0) 0xe1ab0e0 con 0x1f102840 -8467> 2014-09-18 18:08:25.276295 7f4c9f251700 5 -- op tracker -- , seq: 85674, time: 2014-09-18 18:08:25.276164, event: header_read, op: pg_notify(2.4f6(15) epoch 11700) -8466> 2014-09-18 18:08:25.276307 7f4c9f251700 5 -- op tracker -- , seq: 85674, time: 2014-09-18 18:08:25.276179, event: throttled, op: pg_notify(2.4f6(15) epoch 11700) -8465> 2014-09-18 18:08:25.276317 7f4c9f251700 5 -- op tracker -- , seq: 85674, time: 2014-09-18 18:08:25.276248, event: all_read, op: pg_notify(2.4f6(15) epoch 11700) -8464> 2014-09-18 18:08:25.276326 7f4c9f251700 5 -- op tracker -- , seq: 85674, time: 2014-09-18 18:08:25.276289, event: dispatched, op: pg_notify(2.4f6(15) epoch 11700) -8463> 2014-09-18 18:08:25.276347 7f4c9f251700 5 -- op tracker -- , seq: 85674, time: 2014-09-18 18:08:25.276345, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11700) -8462> 2014-09-18 18:08:25.276361 7f4c9f251700 5 -- op tracker -- , seq: 85674, time: 2014-09-18 18:08:25.276359, event: started, op: pg_notify(2.4f6(15) epoch 11700) -8461> 2014-09-18 18:08:25.276385 7f4c9f251700 5 -- op tracker -- , seq: 85674, time: 2014-09-18 18:08:25.276383, event: done, op: pg_notify(2.4f6(15) epoch 11700) -8460> 2014-09-18 18:08:25.296043 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1268 ==== osd_map(11702..11703 src has 1..11703) v3 ==== 3197+0+0 (1251843143 0 0) 0x10e23200 con 0x1f102840 -8459> 2014-09-18 18:08:25.296071 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11702,11703], i have 11703, src has [1,11703] -8458> 2014-09-18 18:08:25.296094 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1269 ==== pg_notify(2.4f6(15) epoch 11703) v5 ==== 1543+0+0 (881465680 0 0) 0xf488b20 con 0x1f102840 -8457> 2014-09-18 18:08:25.296106 7f4c9f251700 5 -- op tracker -- , seq: 85675, time: 2014-09-18 18:08:25.296031, event: header_read, op: pg_notify(2.4f6(15) epoch 11703) -8456> 2014-09-18 18:08:25.296117 7f4c9f251700 5 -- op tracker -- , seq: 85675, time: 2014-09-18 18:08:25.296031, event: throttled, op: pg_notify(2.4f6(15) epoch 11703) -8455> 2014-09-18 18:08:25.296125 7f4c9f251700 5 -- op tracker -- , seq: 85675, time: 2014-09-18 18:08:25.296078, event: all_read, op: pg_notify(2.4f6(15) epoch 11703) -8454> 2014-09-18 18:08:25.296132 7f4c9f251700 5 -- op tracker -- , seq: 85675, time: 2014-09-18 18:08:25.296102, event: dispatched, op: pg_notify(2.4f6(15) epoch 11703) -8453> 2014-09-18 18:08:25.296138 7f4c9f251700 5 -- op tracker -- , seq: 85675, time: 2014-09-18 18:08:25.296137, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11703) -8452> 2014-09-18 18:08:25.296146 7f4c9f251700 5 -- op tracker -- , seq: 85675, time: 2014-09-18 18:08:25.296145, event: started, op: pg_notify(2.4f6(15) epoch 11703) -8451> 2014-09-18 18:08:25.296166 7f4c9f251700 5 -- op tracker -- , seq: 85675, time: 2014-09-18 18:08:25.296164, event: done, op: pg_notify(2.4f6(15) epoch 11703) -8450> 2014-09-18 18:08:25.337691 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.37 10.242.42.168:0/18319 103 ==== osd_ping(ping e11703 stamp 2014-09-18 18:08:25.204247) v2 ==== 47+0+0 (2505717594 0 0) 0xf578760 con 0x11ffdce0 -8449> 2014-09-18 18:08:25.337730 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11703 stamp 2014-09-18 18:08:25.204247) v2 -- ?+0 0x1173a940 con 0x11ffdce0 -8448> 2014-09-18 18:08:25.337759 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.37 10.242.42.168:0/18319 103 ==== osd_ping(ping e11703 stamp 2014-09-18 18:08:25.204247) v2 ==== 47+0+0 (2505717594 0 0) 0xf0a12a0 con 0x110bb860 -8447> 2014-09-18 18:08:25.337791 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11703 stamp 2014-09-18 18:08:25.204247) v2 -- ?+0 0xfccab20 con 0x110bb860 -8446> 2014-09-18 18:08:25.345092 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).connect error 10.242.42.164:6884/38815, (111) Connection refused -8445> 2014-09-18 18:08:25.345122 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=142 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).fault (111) Connection refused -8444> 2014-09-18 18:08:25.345612 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).connect error 10.242.42.164:6886/38815, (111) Connection refused -8443> 2014-09-18 18:08:25.345621 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).connect error 10.242.42.164:6885/38815, (111) Connection refused -8442> 2014-09-18 18:08:25.345645 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault (111) Connection refused -8441> 2014-09-18 18:08:25.345655 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault (111) Connection refused -8440> 2014-09-18 18:08:25.483863 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.22 10.242.42.172:0/37668 439 ==== osd_ping(ping e11703 stamp 2014-09-18 23:37:14.588420) v2 ==== 47+0+0 (1459258187 0 0) 0xf93d480 con 0x14393340 -8439> 2014-09-18 18:08:25.483903 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11703 stamp 2014-09-18 23:37:14.588420) v2 -- ?+0 0xf578760 con 0x14393340 -8438> 2014-09-18 18:08:25.484126 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.22 10.242.42.172:0/37668 439 ==== osd_ping(ping e11703 stamp 2014-09-18 23:37:14.588420) v2 ==== 47+0+0 (1459258187 0 0) 0xe794000 con 0x109caf20 -8437> 2014-09-18 18:08:25.484160 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11703 stamp 2014-09-18 23:37:14.588420) v2 -- ?+0 0xf0a12a0 con 0x109caf20 -8436> 2014-09-18 18:08:25.672372 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2586 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x19df2d00 con 0x4900b00 -8435> 2014-09-18 18:08:25.672419 7f4c9f251700 3 osd.42 11703 handle_osd_map epochs [11704,11704], i have 11703, src has [1,11704] -8434> 2014-09-18 18:08:25.675371 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2587 ==== pg_notify(2.71c(12) epoch 11704) v5 ==== 1424+0+0 (2556049624 0 0) 0xf6e1840 con 0x4900b00 -8433> 2014-09-18 18:08:25.675386 7f4c9f251700 5 -- op tracker -- , seq: 85676, time: 2014-09-18 18:08:25.672387, event: header_read, op: pg_notify(2.71c(12) epoch 11704) -8432> 2014-09-18 18:08:25.675397 7f4c9f251700 5 -- op tracker -- , seq: 85676, time: 2014-09-18 18:08:25.672388, event: throttled, op: pg_notify(2.71c(12) epoch 11704) -8431> 2014-09-18 18:08:25.675402 7f4c9f251700 5 -- op tracker -- , seq: 85676, time: 2014-09-18 18:08:25.672427, event: all_read, op: pg_notify(2.71c(12) epoch 11704) -8430> 2014-09-18 18:08:25.675408 7f4c9f251700 5 -- op tracker -- , seq: 85676, time: 2014-09-18 18:08:25.675382, event: dispatched, op: pg_notify(2.71c(12) epoch 11704) -8429> 2014-09-18 18:08:25.675415 7f4c9f251700 5 -- op tracker -- , seq: 85676, time: 2014-09-18 18:08:25.675414, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11704) -8428> 2014-09-18 18:08:25.675504 7f4c9f251700 5 -- op tracker -- , seq: 85676, time: 2014-09-18 18:08:25.675501, event: started, op: pg_notify(2.71c(12) epoch 11704) -8427> 2014-09-18 18:08:25.675528 7f4c9f251700 5 -- op tracker -- , seq: 85676, time: 2014-09-18 18:08:25.675527, event: done, op: pg_notify(2.71c(12) epoch 11704) -8426> 2014-09-18 18:08:25.675542 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1254 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0xeeead00 con 0xe8db760 -8425> 2014-09-18 18:08:25.675553 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8424> 2014-09-18 18:08:25.675559 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1028 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0xe936480 con 0x14358260 -8423> 2014-09-18 18:08:25.675567 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8422> 2014-09-18 18:08:25.675571 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1255 ==== pg_notify(2.17e(17) epoch 11704) v5 ==== 1593+0+0 (689004329 0 0) 0xf1065a0 con 0xe8db760 -8421> 2014-09-18 18:08:25.675580 7f4c9f251700 5 -- op tracker -- , seq: 85677, time: 2014-09-18 18:08:25.673593, event: header_read, op: pg_notify(2.17e(17) epoch 11704) -8420> 2014-09-18 18:08:25.675587 7f4c9f251700 5 -- op tracker -- , seq: 85677, time: 2014-09-18 18:08:25.673594, event: throttled, op: pg_notify(2.17e(17) epoch 11704) -8419> 2014-09-18 18:08:25.675593 7f4c9f251700 5 -- op tracker -- , seq: 85677, time: 2014-09-18 18:08:25.673659, event: all_read, op: pg_notify(2.17e(17) epoch 11704) -8418> 2014-09-18 18:08:25.675599 7f4c9f251700 5 -- op tracker -- , seq: 85677, time: 2014-09-18 18:08:25.675577, event: dispatched, op: pg_notify(2.17e(17) epoch 11704) -8417> 2014-09-18 18:08:25.675605 7f4c9f251700 5 -- op tracker -- , seq: 85677, time: 2014-09-18 18:08:25.675604, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11704) -8416> 2014-09-18 18:08:25.675612 7f4c9f251700 5 -- op tracker -- , seq: 85677, time: 2014-09-18 18:08:25.675611, event: started, op: pg_notify(2.17e(17) epoch 11704) -8415> 2014-09-18 18:08:25.675694 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11704..11704 src has 1..11704) v3 -- ?+0 0xe337440 con 0x140f95a0 -8414> 2014-09-18 18:08:25.675705 7f4c9f251700 5 -- op tracker -- , seq: 85677, time: 2014-09-18 18:08:25.675703, event: done, op: pg_notify(2.17e(17) epoch 11704) -8413> 2014-09-18 18:08:25.675712 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11704) v5 -- ?+0 0xf2b5840 con 0x140f95a0 -8412> 2014-09-18 18:08:25.675716 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1029 ==== pg_notify(2.719(17) epoch 11704) v5 ==== 1601+0+0 (4219519929 0 0) 0xf731de0 con 0x14358260 -8411> 2014-09-18 18:08:25.675727 7f4c9f251700 5 -- op tracker -- , seq: 85678, time: 2014-09-18 18:08:25.674880, event: header_read, op: pg_notify(2.719(17) epoch 11704) -8410> 2014-09-18 18:08:25.675734 7f4c9f251700 5 -- op tracker -- , seq: 85678, time: 2014-09-18 18:08:25.674881, event: throttled, op: pg_notify(2.719(17) epoch 11704) -8409> 2014-09-18 18:08:25.675740 7f4c9f251700 5 -- op tracker -- , seq: 85678, time: 2014-09-18 18:08:25.674918, event: all_read, op: pg_notify(2.719(17) epoch 11704) -8408> 2014-09-18 18:08:25.675746 7f4c9f251700 5 -- op tracker -- , seq: 85678, time: 2014-09-18 18:08:25.675725, event: dispatched, op: pg_notify(2.719(17) epoch 11704) -8407> 2014-09-18 18:08:25.675753 7f4c9f251700 5 -- op tracker -- , seq: 85678, time: 2014-09-18 18:08:25.675752, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11704) -8406> 2014-09-18 18:08:25.675761 7f4c9f251700 5 -- op tracker -- , seq: 85678, time: 2014-09-18 18:08:25.675759, event: started, op: pg_notify(2.719(17) epoch 11704) -8405> 2014-09-18 18:08:25.675774 7f4c9f251700 5 -- op tracker -- , seq: 85678, time: 2014-09-18 18:08:25.675773, event: done, op: pg_notify(2.719(17) epoch 11704) -8404> 2014-09-18 18:08:25.675802 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11704..11704 src has 1..11704) v3 -- ?+0 0x1e282fc0 con 0x1395edc0 -8403> 2014-09-18 18:08:25.675822 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11704) v5 -- ?+0 0xf186780 con 0x1395edc0 -8402> 2014-09-18 18:08:25.675856 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11704..11704 src has 1..11704) v3 -- ?+0 0x2db45b00 con 0x1395f1e0 -8401> 2014-09-18 18:08:25.675881 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11704) v5 -- ?+0 0x21cfc1c0 con 0x1395f1e0 -8400> 2014-09-18 18:08:25.676142 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 913 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x2db47840 con 0x1395f8c0 -8399> 2014-09-18 18:08:25.676164 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8398> 2014-09-18 18:08:25.676193 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 914 ==== pg_notify(2.410(11) epoch 11704) v5 ==== 1287+0+0 (3955550183 0 0) 0xeb7b0c0 con 0x1395f8c0 -8397> 2014-09-18 18:08:25.676208 7f4c9f251700 5 -- op tracker -- , seq: 85679, time: 2014-09-18 18:08:25.676150, event: header_read, op: pg_notify(2.410(11) epoch 11704) -8396> 2014-09-18 18:08:25.676218 7f4c9f251700 5 -- op tracker -- , seq: 85679, time: 2014-09-18 18:08:25.676151, event: throttled, op: pg_notify(2.410(11) epoch 11704) -8395> 2014-09-18 18:08:25.676224 7f4c9f251700 5 -- op tracker -- , seq: 85679, time: 2014-09-18 18:08:25.676182, event: all_read, op: pg_notify(2.410(11) epoch 11704) -8394> 2014-09-18 18:08:25.676230 7f4c9f251700 5 -- op tracker -- , seq: 85679, time: 2014-09-18 18:08:25.676204, event: dispatched, op: pg_notify(2.410(11) epoch 11704) -8393> 2014-09-18 18:08:25.676236 7f4c9f251700 5 -- op tracker -- , seq: 85679, time: 2014-09-18 18:08:25.676235, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11704) -8392> 2014-09-18 18:08:25.676244 7f4c9f251700 5 -- op tracker -- , seq: 85679, time: 2014-09-18 18:08:25.676242, event: started, op: pg_notify(2.410(11) epoch 11704) -8391> 2014-09-18 18:08:25.676259 7f4c9f251700 5 -- op tracker -- , seq: 85679, time: 2014-09-18 18:08:25.676257, event: done, op: pg_notify(2.410(11) epoch 11704) -8390> 2014-09-18 18:08:25.676395 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11704) v5 -- ?+0 0xf4f12a0 con 0x140f95a0 -8389> 2014-09-18 18:08:25.676751 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11704..11704 src has 1..11704) v3 -- ?+0 0xe2edf80 con 0x1395ef20 -8388> 2014-09-18 18:08:25.676860 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(10) epoch 11704) v5 -- ?+0 0xf1ea1e0 con 0x1395ef20 -8387> 2014-09-18 18:08:25.677448 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11704..11704 src has 1..11704) v3 -- ?+0 0xe2ec000 con 0x23a74f20 -8386> 2014-09-18 18:08:25.677463 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11704) v5 -- ?+0 0xf1ea780 con 0x23a74f20 -8385> 2014-09-18 18:08:25.678437 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 32 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0xd7c5600 con 0x1395f340 -8384> 2014-09-18 18:08:25.678451 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8383> 2014-09-18 18:08:25.678479 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 33 ==== pg_notify(2.1d8(9) epoch 11704) v5 ==== 1261+0+0 (230384592 0 0) 0xf488940 con 0x1395f340 -8382> 2014-09-18 18:08:25.678498 7f4c9f251700 5 -- op tracker -- , seq: 85680, time: 2014-09-18 18:08:25.678440, event: header_read, op: pg_notify(2.1d8(9) epoch 11704) -8381> 2014-09-18 18:08:25.678508 7f4c9f251700 5 -- op tracker -- , seq: 85680, time: 2014-09-18 18:08:25.678440, event: throttled, op: pg_notify(2.1d8(9) epoch 11704) -8380> 2014-09-18 18:08:25.678513 7f4c9f251700 5 -- op tracker -- , seq: 85680, time: 2014-09-18 18:08:25.678467, event: all_read, op: pg_notify(2.1d8(9) epoch 11704) -8379> 2014-09-18 18:08:25.678520 7f4c9f251700 5 -- op tracker -- , seq: 85680, time: 2014-09-18 18:08:25.678493, event: dispatched, op: pg_notify(2.1d8(9) epoch 11704) -8378> 2014-09-18 18:08:25.678526 7f4c9f251700 5 -- op tracker -- , seq: 85680, time: 2014-09-18 18:08:25.678525, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11704) -8377> 2014-09-18 18:08:25.678534 7f4c9f251700 5 -- op tracker -- , seq: 85680, time: 2014-09-18 18:08:25.678533, event: started, op: pg_notify(2.1d8(9) epoch 11704) -8376> 2014-09-18 18:08:25.678561 7f4c9f251700 5 -- op tracker -- , seq: 85680, time: 2014-09-18 18:08:25.678559, event: done, op: pg_notify(2.1d8(9) epoch 11704) -8375> 2014-09-18 18:08:25.678726 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 776 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x12330480 con 0xe9a27e0 -8374> 2014-09-18 18:08:25.678751 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8373> 2014-09-18 18:08:25.678966 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 777 ==== pg_notify(2.2d8(173) epoch 11704) v5 ==== 10757+0+0 (535386036 0 0) 0x1421ef00 con 0xe9a27e0 -8372> 2014-09-18 18:08:25.678977 7f4c9f251700 5 -- op tracker -- , seq: 85681, time: 2014-09-18 18:08:25.678814, event: header_read, op: pg_notify(2.2d8(173) epoch 11704) -8371> 2014-09-18 18:08:25.678987 7f4c9f251700 5 -- op tracker -- , seq: 85681, time: 2014-09-18 18:08:25.678814, event: throttled, op: pg_notify(2.2d8(173) epoch 11704) -8370> 2014-09-18 18:08:25.678993 7f4c9f251700 5 -- op tracker -- , seq: 85681, time: 2014-09-18 18:08:25.678955, event: all_read, op: pg_notify(2.2d8(173) epoch 11704) -8369> 2014-09-18 18:08:25.678999 7f4c9f251700 5 -- op tracker -- , seq: 85681, time: 2014-09-18 18:08:25.678974, event: dispatched, op: pg_notify(2.2d8(173) epoch 11704) -8368> 2014-09-18 18:08:25.679005 7f4c9f251700 5 -- op tracker -- , seq: 85681, time: 2014-09-18 18:08:25.679004, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11704) -8367> 2014-09-18 18:08:25.679013 7f4c9f251700 5 -- op tracker -- , seq: 85681, time: 2014-09-18 18:08:25.679012, event: started, op: pg_notify(2.2d8(173) epoch 11704) -8366> 2014-09-18 18:08:25.679030 7f4c9f251700 5 -- op tracker -- , seq: 85681, time: 2014-09-18 18:08:25.679029, event: done, op: pg_notify(2.2d8(173) epoch 11704) -8365> 2014-09-18 18:08:25.679064 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 519 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0xe2edf80 con 0x1395ef20 -8364> 2014-09-18 18:08:25.679075 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8363> 2014-09-18 18:08:25.679098 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 520 ==== pg_notify(2.4d0(10) epoch 11704) v5 ==== 1292+0+0 (1611206639 0 0) 0xf1ea1e0 con 0x1395ef20 -8362> 2014-09-18 18:08:25.679108 7f4c9f251700 5 -- op tracker -- , seq: 85682, time: 2014-09-18 18:08:25.679066, event: header_read, op: pg_notify(2.4d0(10) epoch 11704) -8361> 2014-09-18 18:08:25.679115 7f4c9f251700 5 -- op tracker -- , seq: 85682, time: 2014-09-18 18:08:25.679066, event: throttled, op: pg_notify(2.4d0(10) epoch 11704) -8360> 2014-09-18 18:08:25.679121 7f4c9f251700 5 -- op tracker -- , seq: 85682, time: 2014-09-18 18:08:25.679089, event: all_read, op: pg_notify(2.4d0(10) epoch 11704) -8359> 2014-09-18 18:08:25.679127 7f4c9f251700 5 -- op tracker -- , seq: 85682, time: 2014-09-18 18:08:25.679105, event: dispatched, op: pg_notify(2.4d0(10) epoch 11704) -8358> 2014-09-18 18:08:25.679133 7f4c9f251700 5 -- op tracker -- , seq: 85682, time: 2014-09-18 18:08:25.679132, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11704) -8357> 2014-09-18 18:08:25.679140 7f4c9f251700 5 -- op tracker -- , seq: 85682, time: 2014-09-18 18:08:25.679139, event: started, op: pg_notify(2.4d0(10) epoch 11704) -8356> 2014-09-18 18:08:25.679155 7f4c9f251700 5 -- op tracker -- , seq: 85682, time: 2014-09-18 18:08:25.679154, event: done, op: pg_notify(2.4d0(10) epoch 11704) -8355> 2014-09-18 18:08:25.679812 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2001 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x1182a6c0 con 0xe2d9a20 -8354> 2014-09-18 18:08:25.679829 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8353> 2014-09-18 18:08:25.679849 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2002 ==== pg_notify(2.26d(9) epoch 11704) v5 ==== 1257+0+0 (2748585893 0 0) 0x2b93c1c0 con 0xe2d9a20 -8352> 2014-09-18 18:08:25.679860 7f4c9f251700 5 -- op tracker -- , seq: 85683, time: 2014-09-18 18:08:25.679811, event: header_read, op: pg_notify(2.26d(9) epoch 11704) -8351> 2014-09-18 18:08:25.679870 7f4c9f251700 5 -- op tracker -- , seq: 85683, time: 2014-09-18 18:08:25.679812, event: throttled, op: pg_notify(2.26d(9) epoch 11704) -8350> 2014-09-18 18:08:25.679876 7f4c9f251700 5 -- op tracker -- , seq: 85683, time: 2014-09-18 18:08:25.679839, event: all_read, op: pg_notify(2.26d(9) epoch 11704) -8349> 2014-09-18 18:08:25.679882 7f4c9f251700 5 -- op tracker -- , seq: 85683, time: 2014-09-18 18:08:25.679856, event: dispatched, op: pg_notify(2.26d(9) epoch 11704) -8348> 2014-09-18 18:08:25.679889 7f4c9f251700 5 -- op tracker -- , seq: 85683, time: 2014-09-18 18:08:25.679887, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11704) -8347> 2014-09-18 18:08:25.679896 7f4c9f251700 5 -- op tracker -- , seq: 85683, time: 2014-09-18 18:08:25.679895, event: started, op: pg_notify(2.26d(9) epoch 11704) -8346> 2014-09-18 18:08:25.679913 7f4c9f251700 5 -- op tracker -- , seq: 85683, time: 2014-09-18 18:08:25.679912, event: done, op: pg_notify(2.26d(9) epoch 11704) -8345> 2014-09-18 18:08:25.680114 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3596 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x139a8880 con 0xcbecec0 -8344> 2014-09-18 18:08:25.680130 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8343> 2014-09-18 18:08:25.680156 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3597 ==== pg_notify(2.4f6(12) epoch 11704) v5 ==== 1382+0+0 (2065896923 0 0) 0xeb7b660 con 0xcbecec0 -8342> 2014-09-18 18:08:25.680168 7f4c9f251700 5 -- op tracker -- , seq: 85684, time: 2014-09-18 18:08:25.680114, event: header_read, op: pg_notify(2.4f6(12) epoch 11704) -8341> 2014-09-18 18:08:25.680177 7f4c9f251700 5 -- op tracker -- , seq: 85684, time: 2014-09-18 18:08:25.680114, event: throttled, op: pg_notify(2.4f6(12) epoch 11704) -8340> 2014-09-18 18:08:25.680183 7f4c9f251700 5 -- op tracker -- , seq: 85684, time: 2014-09-18 18:08:25.680146, event: all_read, op: pg_notify(2.4f6(12) epoch 11704) -8339> 2014-09-18 18:08:25.680189 7f4c9f251700 5 -- op tracker -- , seq: 85684, time: 2014-09-18 18:08:25.680165, event: dispatched, op: pg_notify(2.4f6(12) epoch 11704) -8338> 2014-09-18 18:08:25.680195 7f4c9f251700 5 -- op tracker -- , seq: 85684, time: 2014-09-18 18:08:25.680194, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11704) -8337> 2014-09-18 18:08:25.680204 7f4c9f251700 5 -- op tracker -- , seq: 85684, time: 2014-09-18 18:08:25.680202, event: started, op: pg_notify(2.4f6(12) epoch 11704) -8336> 2014-09-18 18:08:25.680221 7f4c9f251700 5 -- op tracker -- , seq: 85684, time: 2014-09-18 18:08:25.680220, event: done, op: pg_notify(2.4f6(12) epoch 11704) -8335> 2014-09-18 18:08:25.680354 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2336 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0xe269180 con 0xdd8c940 -8334> 2014-09-18 18:08:25.680370 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8333> 2014-09-18 18:08:25.680413 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2337 ==== pg_notify(2.457(21) epoch 11704) v5 ==== 1875+0+0 (3751260036 0 0) 0xf1072c0 con 0xdd8c940 -8332> 2014-09-18 18:08:25.680433 7f4c9f251700 5 -- op tracker -- , seq: 85685, time: 2014-09-18 18:08:25.680367, event: header_read, op: pg_notify(2.457(21) epoch 11704) -8331> 2014-09-18 18:08:25.680446 7f4c9f251700 5 -- op tracker -- , seq: 85685, time: 2014-09-18 18:08:25.680367, event: throttled, op: pg_notify(2.457(21) epoch 11704) -8330> 2014-09-18 18:08:25.680456 7f4c9f251700 5 -- op tracker -- , seq: 85685, time: 2014-09-18 18:08:25.680401, event: all_read, op: pg_notify(2.457(21) epoch 11704) -8329> 2014-09-18 18:08:25.680466 7f4c9f251700 5 -- op tracker -- , seq: 85685, time: 2014-09-18 18:08:25.680428, event: dispatched, op: pg_notify(2.457(21) epoch 11704) -8328> 2014-09-18 18:08:25.680476 7f4c9f251700 5 -- op tracker -- , seq: 85685, time: 2014-09-18 18:08:25.680474, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11704) -8327> 2014-09-18 18:08:25.680484 7f4c9f251700 5 -- op tracker -- , seq: 85685, time: 2014-09-18 18:08:25.680482, event: started, op: pg_notify(2.457(21) epoch 11704) -8326> 2014-09-18 18:08:25.680567 7f4c9f251700 5 -- op tracker -- , seq: 85685, time: 2014-09-18 18:08:25.680565, event: done, op: pg_notify(2.457(21) epoch 11704) -8325> 2014-09-18 18:08:25.681710 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2338 ==== pg_notify(2.2c7(139) epoch 11704) v5 ==== 8651+0+0 (2283672900 0 0) 0xf108580 con 0xdd8c940 -8324> 2014-09-18 18:08:25.681723 7f4c9f251700 5 -- op tracker -- , seq: 85686, time: 2014-09-18 18:08:25.681589, event: header_read, op: pg_notify(2.2c7(139) epoch 11704) -8323> 2014-09-18 18:08:25.681731 7f4c9f251700 5 -- op tracker -- , seq: 85686, time: 2014-09-18 18:08:25.681590, event: throttled, op: pg_notify(2.2c7(139) epoch 11704) -8322> 2014-09-18 18:08:25.681737 7f4c9f251700 5 -- op tracker -- , seq: 85686, time: 2014-09-18 18:08:25.681698, event: all_read, op: pg_notify(2.2c7(139) epoch 11704) -8321> 2014-09-18 18:08:25.681745 7f4c9f251700 5 -- op tracker -- , seq: 85686, time: 2014-09-18 18:08:25.681720, event: dispatched, op: pg_notify(2.2c7(139) epoch 11704) -8320> 2014-09-18 18:08:25.681754 7f4c9f251700 5 -- op tracker -- , seq: 85686, time: 2014-09-18 18:08:25.681752, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11704) -8319> 2014-09-18 18:08:25.681763 7f4c9f251700 5 -- op tracker -- , seq: 85686, time: 2014-09-18 18:08:25.681761, event: started, op: pg_notify(2.2c7(139) epoch 11704) -8318> 2014-09-18 18:08:25.681782 7f4c9f251700 5 -- op tracker -- , seq: 85686, time: 2014-09-18 18:08:25.681780, event: done, op: pg_notify(2.2c7(139) epoch 11704) -8317> 2014-09-18 18:08:25.682278 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3274 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x24d1a000 con 0x4900580 -8316> 2014-09-18 18:08:25.682298 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8315> 2014-09-18 18:08:25.682627 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3275 ==== pg_notify(2.4f6(184) epoch 11704) v5 ==== 11296+0+0 (3320916039 0 0) 0x1368e580 con 0x4900580 -8314> 2014-09-18 18:08:25.682641 7f4c9f251700 5 -- op tracker -- , seq: 85687, time: 2014-09-18 18:08:25.682454, event: header_read, op: pg_notify(2.4f6(184) epoch 11704) -8313> 2014-09-18 18:08:25.682650 7f4c9f251700 5 -- op tracker -- , seq: 85687, time: 2014-09-18 18:08:25.682455, event: throttled, op: pg_notify(2.4f6(184) epoch 11704) -8312> 2014-09-18 18:08:25.682656 7f4c9f251700 5 -- op tracker -- , seq: 85687, time: 2014-09-18 18:08:25.682613, event: all_read, op: pg_notify(2.4f6(184) epoch 11704) -8311> 2014-09-18 18:08:25.682662 7f4c9f251700 5 -- op tracker -- , seq: 85687, time: 2014-09-18 18:08:25.682638, event: dispatched, op: pg_notify(2.4f6(184) epoch 11704) -8310> 2014-09-18 18:08:25.682668 7f4c9f251700 5 -- op tracker -- , seq: 85687, time: 2014-09-18 18:08:25.682667, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11704) -8309> 2014-09-18 18:08:25.682676 7f4c9f251700 5 -- op tracker -- , seq: 85687, time: 2014-09-18 18:08:25.682674, event: started, op: pg_notify(2.4f6(184) epoch 11704) -8308> 2014-09-18 18:08:25.682692 7f4c9f251700 5 -- op tracker -- , seq: 85687, time: 2014-09-18 18:08:25.682691, event: done, op: pg_notify(2.4f6(184) epoch 11704) -8307> 2014-09-18 18:08:25.682974 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1186 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x1182b8c0 con 0x1088b440 -8306> 2014-09-18 18:08:25.682986 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8305> 2014-09-18 18:08:25.683313 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1187 ==== pg_notify(2.2d8(173) epoch 11704) v5 ==== 10757+0+0 (535386036 0 0) 0x100eef00 con 0x1088b440 -8304> 2014-09-18 18:08:25.683333 7f4c9f251700 5 -- op tracker -- , seq: 85688, time: 2014-09-18 18:08:25.683133, event: header_read, op: pg_notify(2.2d8(173) epoch 11704) -8303> 2014-09-18 18:08:25.683345 7f4c9f251700 5 -- op tracker -- , seq: 85688, time: 2014-09-18 18:08:25.683134, event: throttled, op: pg_notify(2.2d8(173) epoch 11704) -8302> 2014-09-18 18:08:25.683365 7f4c9f251700 5 -- op tracker -- , seq: 85688, time: 2014-09-18 18:08:25.683299, event: all_read, op: pg_notify(2.2d8(173) epoch 11704) -8301> 2014-09-18 18:08:25.683375 7f4c9f251700 5 -- op tracker -- , seq: 85688, time: 2014-09-18 18:08:25.683327, event: dispatched, op: pg_notify(2.2d8(173) epoch 11704) -8300> 2014-09-18 18:08:25.683386 7f4c9f251700 5 -- op tracker -- , seq: 85688, time: 2014-09-18 18:08:25.683384, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11704) -8299> 2014-09-18 18:08:25.683398 7f4c9f251700 5 -- op tracker -- , seq: 85688, time: 2014-09-18 18:08:25.683395, event: started, op: pg_notify(2.2d8(173) epoch 11704) -8298> 2014-09-18 18:08:25.683420 7f4c9f251700 5 -- op tracker -- , seq: 85688, time: 2014-09-18 18:08:25.683419, event: done, op: pg_notify(2.2d8(173) epoch 11704) -8297> 2014-09-18 18:08:25.683667 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3276 ==== pg_notify(2.17e(17) epoch 11704) v5 ==== 1593+0+0 (689004329 0 0) 0x19f23680 con 0x4900580 -8296> 2014-09-18 18:08:25.683681 7f4c9f251700 5 -- op tracker -- , seq: 85689, time: 2014-09-18 18:08:25.683623, event: header_read, op: pg_notify(2.17e(17) epoch 11704) -8295> 2014-09-18 18:08:25.683693 7f4c9f251700 5 -- op tracker -- , seq: 85689, time: 2014-09-18 18:08:25.683623, event: throttled, op: pg_notify(2.17e(17) epoch 11704) -8294> 2014-09-18 18:08:25.683703 7f4c9f251700 5 -- op tracker -- , seq: 85689, time: 2014-09-18 18:08:25.683656, event: all_read, op: pg_notify(2.17e(17) epoch 11704) -8293> 2014-09-18 18:08:25.683712 7f4c9f251700 5 -- op tracker -- , seq: 85689, time: 2014-09-18 18:08:25.683676, event: dispatched, op: pg_notify(2.17e(17) epoch 11704) -8292> 2014-09-18 18:08:25.683722 7f4c9f251700 5 -- op tracker -- , seq: 85689, time: 2014-09-18 18:08:25.683720, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11704) -8291> 2014-09-18 18:08:25.683734 7f4c9f251700 5 -- op tracker -- , seq: 85689, time: 2014-09-18 18:08:25.683731, event: started, op: pg_notify(2.17e(17) epoch 11704) -8290> 2014-09-18 18:08:25.683756 7f4c9f251700 5 -- op tracker -- , seq: 85689, time: 2014-09-18 18:08:25.683755, event: done, op: pg_notify(2.17e(17) epoch 11704) -8289> 2014-09-18 18:08:25.684118 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3598 ==== pg_notify(2.3b6(10) epoch 11704) v5 ==== 1260+0+0 (3518116266 0 0) 0xeb785a0 con 0xcbecec0 -8288> 2014-09-18 18:08:25.684200 7f4c9f251700 5 -- op tracker -- , seq: 85690, time: 2014-09-18 18:08:25.684074, event: header_read, op: pg_notify(2.3b6(10) epoch 11704) -8287> 2014-09-18 18:08:25.684212 7f4c9f251700 5 -- op tracker -- , seq: 85690, time: 2014-09-18 18:08:25.684075, event: throttled, op: pg_notify(2.3b6(10) epoch 11704) -8286> 2014-09-18 18:08:25.684221 7f4c9f251700 5 -- op tracker -- , seq: 85690, time: 2014-09-18 18:08:25.684107, event: all_read, op: pg_notify(2.3b6(10) epoch 11704) -8285> 2014-09-18 18:08:25.684230 7f4c9f251700 5 -- op tracker -- , seq: 85690, time: 2014-09-18 18:08:25.684194, event: dispatched, op: pg_notify(2.3b6(10) epoch 11704) -8284> 2014-09-18 18:08:25.684241 7f4c9f251700 5 -- op tracker -- , seq: 85690, time: 2014-09-18 18:08:25.684238, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11704) -8283> 2014-09-18 18:08:25.684253 7f4c9f251700 5 -- op tracker -- , seq: 85690, time: 2014-09-18 18:08:25.684250, event: started, op: pg_notify(2.3b6(10) epoch 11704) -8282> 2014-09-18 18:08:25.684276 7f4c9f251700 5 -- op tracker -- , seq: 85690, time: 2014-09-18 18:08:25.684274, event: done, op: pg_notify(2.3b6(10) epoch 11704) -8281> 2014-09-18 18:08:25.684753 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3277 ==== pg_notify(2.1bb(9) epoch 11704) v5 ==== 1189+0+0 (1935154184 0 0) 0x19f250c0 con 0x4900580 -8280> 2014-09-18 18:08:25.684766 7f4c9f251700 5 -- op tracker -- , seq: 85691, time: 2014-09-18 18:08:25.684716, event: header_read, op: pg_notify(2.1bb(9) epoch 11704) -8279> 2014-09-18 18:08:25.684777 7f4c9f251700 5 -- op tracker -- , seq: 85691, time: 2014-09-18 18:08:25.684717, event: throttled, op: pg_notify(2.1bb(9) epoch 11704) -8278> 2014-09-18 18:08:25.684786 7f4c9f251700 5 -- op tracker -- , seq: 85691, time: 2014-09-18 18:08:25.684743, event: all_read, op: pg_notify(2.1bb(9) epoch 11704) -8277> 2014-09-18 18:08:25.684796 7f4c9f251700 5 -- op tracker -- , seq: 85691, time: 2014-09-18 18:08:25.684762, event: dispatched, op: pg_notify(2.1bb(9) epoch 11704) -8276> 2014-09-18 18:08:25.684806 7f4c9f251700 5 -- op tracker -- , seq: 85691, time: 2014-09-18 18:08:25.684803, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11704) -8275> 2014-09-18 18:08:25.684817 7f4c9f251700 5 -- op tracker -- , seq: 85691, time: 2014-09-18 18:08:25.684814, event: started, op: pg_notify(2.1bb(9) epoch 11704) -8274> 2014-09-18 18:08:25.684838 7f4c9f251700 5 -- op tracker -- , seq: 85691, time: 2014-09-18 18:08:25.684836, event: done, op: pg_notify(2.1bb(9) epoch 11704) -8273> 2014-09-18 18:08:25.686686 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3599 ==== pg_notify(2.517(11) epoch 11704) v5 ==== 1343+0+0 (2225297396 0 0) 0x24f6c5a0 con 0xcbecec0 -8272> 2014-09-18 18:08:25.686701 7f4c9f251700 5 -- op tracker -- , seq: 85692, time: 2014-09-18 18:08:25.686629, event: header_read, op: pg_notify(2.517(11) epoch 11704) -8271> 2014-09-18 18:08:25.686711 7f4c9f251700 5 -- op tracker -- , seq: 85692, time: 2014-09-18 18:08:25.686630, event: throttled, op: pg_notify(2.517(11) epoch 11704) -8270> 2014-09-18 18:08:25.686721 7f4c9f251700 5 -- op tracker -- , seq: 85692, time: 2014-09-18 18:08:25.686673, event: all_read, op: pg_notify(2.517(11) epoch 11704) -8269> 2014-09-18 18:08:25.686731 7f4c9f251700 5 -- op tracker -- , seq: 85692, time: 2014-09-18 18:08:25.686696, event: dispatched, op: pg_notify(2.517(11) epoch 11704) -8268> 2014-09-18 18:08:25.686741 7f4c9f251700 5 -- op tracker -- , seq: 85692, time: 2014-09-18 18:08:25.686739, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11704) -8267> 2014-09-18 18:08:25.686753 7f4c9f251700 5 -- op tracker -- , seq: 85692, time: 2014-09-18 18:08:25.686750, event: started, op: pg_notify(2.517(11) epoch 11704) -8266> 2014-09-18 18:08:25.686793 7f4c9f251700 5 -- op tracker -- , seq: 85692, time: 2014-09-18 18:08:25.686791, event: done, op: pg_notify(2.517(11) epoch 11704) -8265> 2014-09-18 18:08:25.687656 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3600 ==== pg_notify(2.2c7(139) epoch 11704) v5 ==== 8651+0+0 (1235872431 0 0) 0xe76c5a0 con 0xcbecec0 -8264> 2014-09-18 18:08:25.687669 7f4c9f251700 5 -- op tracker -- , seq: 85693, time: 2014-09-18 18:08:25.687545, event: header_read, op: pg_notify(2.2c7(139) epoch 11704) -8263> 2014-09-18 18:08:25.687680 7f4c9f251700 5 -- op tracker -- , seq: 85693, time: 2014-09-18 18:08:25.687546, event: throttled, op: pg_notify(2.2c7(139) epoch 11704) -8262> 2014-09-18 18:08:25.687690 7f4c9f251700 5 -- op tracker -- , seq: 85693, time: 2014-09-18 18:08:25.687644, event: all_read, op: pg_notify(2.2c7(139) epoch 11704) -8261> 2014-09-18 18:08:25.687699 7f4c9f251700 5 -- op tracker -- , seq: 85693, time: 2014-09-18 18:08:25.687665, event: dispatched, op: pg_notify(2.2c7(139) epoch 11704) -8260> 2014-09-18 18:08:25.687709 7f4c9f251700 5 -- op tracker -- , seq: 85693, time: 2014-09-18 18:08:25.687706, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11704) -8259> 2014-09-18 18:08:25.687720 7f4c9f251700 5 -- op tracker -- , seq: 85693, time: 2014-09-18 18:08:25.687718, event: started, op: pg_notify(2.2c7(139) epoch 11704) -8258> 2014-09-18 18:08:25.687741 7f4c9f251700 5 -- op tracker -- , seq: 85693, time: 2014-09-18 18:08:25.687740, event: done, op: pg_notify(2.2c7(139) epoch 11704) -8257> 2014-09-18 18:08:25.688577 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.13 10.242.42.168:0/10879 692 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:25.555429) v2 ==== 47+0+0 (1475626566 0 0) 0x100f0760 con 0x109f9de0 -8256> 2014-09-18 18:08:25.688596 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.13 10.242.42.168:0/10879 692 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:25.555429) v2 ==== 47+0+0 (1475626566 0 0) 0x11fd10e0 con 0x140f7b80 -8255> 2014-09-18 18:08:25.688602 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.555429) v2 -- ?+0 0xe794000 con 0x109f9de0 -8254> 2014-09-18 18:08:25.688652 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.555429) v2 -- ?+0 0xf93d480 con 0x140f7b80 -8253> 2014-09-18 18:08:25.689923 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3601 ==== pg_notify(2.40a(162) epoch 11704) v5 ==== 9736+0+0 (4009354838 0 0) 0x146e1de0 con 0xcbecec0 -8252> 2014-09-18 18:08:25.689939 7f4c9f251700 5 -- op tracker -- , seq: 85694, time: 2014-09-18 18:08:25.689787, event: header_read, op: pg_notify(2.40a(162) epoch 11704) -8251> 2014-09-18 18:08:25.689951 7f4c9f251700 5 -- op tracker -- , seq: 85694, time: 2014-09-18 18:08:25.689788, event: throttled, op: pg_notify(2.40a(162) epoch 11704) -8250> 2014-09-18 18:08:25.689960 7f4c9f251700 5 -- op tracker -- , seq: 85694, time: 2014-09-18 18:08:25.689909, event: all_read, op: pg_notify(2.40a(162) epoch 11704) -8249> 2014-09-18 18:08:25.689970 7f4c9f251700 5 -- op tracker -- , seq: 85694, time: 2014-09-18 18:08:25.689934, event: dispatched, op: pg_notify(2.40a(162) epoch 11704) -8248> 2014-09-18 18:08:25.689981 7f4c9f251700 5 -- op tracker -- , seq: 85694, time: 2014-09-18 18:08:25.689978, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11704) -8247> 2014-09-18 18:08:25.689993 7f4c9f251700 5 -- op tracker -- , seq: 85694, time: 2014-09-18 18:08:25.689990, event: started, op: pg_notify(2.40a(162) epoch 11704) -8246> 2014-09-18 18:08:25.690015 7f4c9f251700 5 -- op tracker -- , seq: 85694, time: 2014-09-18 18:08:25.690013, event: done, op: pg_notify(2.40a(162) epoch 11704) -8245> 2014-09-18 18:08:25.690948 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3602 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11704) v5 ==== 6174+0+0 (2606927056 0 0) 0xf15e5a0 con 0xcbecec0 -8244> 2014-09-18 18:08:25.690963 7f4c9f251700 5 -- op tracker -- , seq: 85695, time: 2014-09-18 18:08:25.690859, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11704) -8243> 2014-09-18 18:08:25.691000 7f4c9f251700 5 -- op tracker -- , seq: 85695, time: 2014-09-18 18:08:25.690860, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11704) -8242> 2014-09-18 18:08:25.691011 7f4c9f251700 5 -- op tracker -- , seq: 85695, time: 2014-09-18 18:08:25.690936, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11704) -8241> 2014-09-18 18:08:25.691021 7f4c9f251700 5 -- op tracker -- , seq: 85695, time: 2014-09-18 18:08:25.690958, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11704) -8240> 2014-09-18 18:08:25.691032 7f4c9f251700 5 -- op tracker -- , seq: 85695, time: 2014-09-18 18:08:25.691028, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11704) -8239> 2014-09-18 18:08:25.691044 7f4c9f251700 5 -- op tracker -- , seq: 85695, time: 2014-09-18 18:08:25.691040, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11704) -8238> 2014-09-18 18:08:25.691083 7f4c9f251700 5 -- op tracker -- , seq: 85695, time: 2014-09-18 18:08:25.691081, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11704) -8237> 2014-09-18 18:08:25.691617 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3603 ==== pg_notify(2.1bb(9) epoch 11704) v5 ==== 1189+0+0 (3755060763 0 0) 0xf15e780 con 0xcbecec0 -8236> 2014-09-18 18:08:25.691632 7f4c9f251700 5 -- op tracker -- , seq: 85696, time: 2014-09-18 18:08:25.691579, event: header_read, op: pg_notify(2.1bb(9) epoch 11704) -8235> 2014-09-18 18:08:25.691643 7f4c9f251700 5 -- op tracker -- , seq: 85696, time: 2014-09-18 18:08:25.691579, event: throttled, op: pg_notify(2.1bb(9) epoch 11704) -8234> 2014-09-18 18:08:25.691652 7f4c9f251700 5 -- op tracker -- , seq: 85696, time: 2014-09-18 18:08:25.691606, event: all_read, op: pg_notify(2.1bb(9) epoch 11704) -8233> 2014-09-18 18:08:25.691662 7f4c9f251700 5 -- op tracker -- , seq: 85696, time: 2014-09-18 18:08:25.691627, event: dispatched, op: pg_notify(2.1bb(9) epoch 11704) -8232> 2014-09-18 18:08:25.691673 7f4c9f251700 5 -- op tracker -- , seq: 85696, time: 2014-09-18 18:08:25.691670, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11704) -8231> 2014-09-18 18:08:25.691684 7f4c9f251700 5 -- op tracker -- , seq: 85696, time: 2014-09-18 18:08:25.691681, event: started, op: pg_notify(2.1bb(9) epoch 11704) -8230> 2014-09-18 18:08:25.691782 7f4c9f251700 5 -- op tracker -- , seq: 85696, time: 2014-09-18 18:08:25.691780, event: done, op: pg_notify(2.1bb(9) epoch 11704) -8229> 2014-09-18 18:08:25.694996 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.11 10.242.42.168:0/17298 124 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:25.562106) v2 ==== 47+0+0 (2561197482 0 0) 0xf09c940 con 0x101cd180 -8228> 2014-09-18 18:08:25.695015 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.562106) v2 -- ?+0 0x11fd10e0 con 0x101cd180 -8227> 2014-09-18 18:08:25.695012 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.11 10.242.42.168:0/17298 124 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:25.562106) v2 ==== 47+0+0 (2561197482 0 0) 0xe76fc00 con 0xcc49e40 -8226> 2014-09-18 18:08:25.695035 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.562106) v2 -- ?+0 0x100f0760 con 0xcc49e40 -8225> 2014-09-18 18:08:25.699779 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1713 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x10e24f40 con 0x4900000 -8224> 2014-09-18 18:08:25.699795 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8223> 2014-09-18 18:08:25.699938 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1714 ==== pg_notify(2.e0(79) epoch 11704) v5 ==== 5119+0+0 (236777662 0 0) 0xf1ed2a0 con 0x4900000 -8222> 2014-09-18 18:08:25.699951 7f4c9f251700 5 -- op tracker -- , seq: 85697, time: 2014-09-18 18:08:25.699856, event: header_read, op: pg_notify(2.e0(79) epoch 11704) -8221> 2014-09-18 18:08:25.699963 7f4c9f251700 5 -- op tracker -- , seq: 85697, time: 2014-09-18 18:08:25.699857, event: throttled, op: pg_notify(2.e0(79) epoch 11704) -8220> 2014-09-18 18:08:25.699974 7f4c9f251700 5 -- op tracker -- , seq: 85697, time: 2014-09-18 18:08:25.699927, event: all_read, op: pg_notify(2.e0(79) epoch 11704) -8219> 2014-09-18 18:08:25.699985 7f4c9f251700 5 -- op tracker -- , seq: 85697, time: 2014-09-18 18:08:25.699947, event: dispatched, op: pg_notify(2.e0(79) epoch 11704) -8218> 2014-09-18 18:08:25.699994 7f4c9f251700 5 -- op tracker -- , seq: 85697, time: 2014-09-18 18:08:25.699992, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11704) -8217> 2014-09-18 18:08:25.700006 7f4c9f251700 5 -- op tracker -- , seq: 85697, time: 2014-09-18 18:08:25.700003, event: started, op: pg_notify(2.e0(79) epoch 11704) -8216> 2014-09-18 18:08:25.700028 7f4c9f251700 5 -- op tracker -- , seq: 85697, time: 2014-09-18 18:08:25.700026, event: done, op: pg_notify(2.e0(79) epoch 11704) -8215> 2014-09-18 18:08:25.733908 7f4caae41700 5 osd.42 11704 tick -8214> 2014-09-18 18:08:25.743185 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3150 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x1182cf40 con 0x1395ec60 -8213> 2014-09-18 18:08:25.743206 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8212> 2014-09-18 18:08:25.743228 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3151 ==== pg_notify(2.71(19) epoch 11704) v5 ==== 1795+0+0 (318159760 0 0) 0xf811480 con 0x1395ec60 -8211> 2014-09-18 18:08:25.743240 7f4c9f251700 5 -- op tracker -- , seq: 85698, time: 2014-09-18 18:08:25.743179, event: header_read, op: pg_notify(2.71(19) epoch 11704) -8210> 2014-09-18 18:08:25.743252 7f4c9f251700 5 -- op tracker -- , seq: 85698, time: 2014-09-18 18:08:25.743180, event: throttled, op: pg_notify(2.71(19) epoch 11704) -8209> 2014-09-18 18:08:25.743260 7f4c9f251700 5 -- op tracker -- , seq: 85698, time: 2014-09-18 18:08:25.743217, event: all_read, op: pg_notify(2.71(19) epoch 11704) -8208> 2014-09-18 18:08:25.743268 7f4c9f251700 5 -- op tracker -- , seq: 85698, time: 2014-09-18 18:08:25.743236, event: dispatched, op: pg_notify(2.71(19) epoch 11704) -8207> 2014-09-18 18:08:25.743276 7f4c9f251700 5 -- op tracker -- , seq: 85698, time: 2014-09-18 18:08:25.743275, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11704) -8206> 2014-09-18 18:08:25.743285 7f4c9f251700 5 -- op tracker -- , seq: 85698, time: 2014-09-18 18:08:25.743283, event: started, op: pg_notify(2.71(19) epoch 11704) -8205> 2014-09-18 18:08:25.743436 7f4c9f251700 5 -- op tracker -- , seq: 85698, time: 2014-09-18 18:08:25.743434, event: done, op: pg_notify(2.71(19) epoch 11704) -8204> 2014-09-18 18:08:25.745396 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=39 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).connect error 10.242.42.164:6884/38815, (111) Connection refused -8203> 2014-09-18 18:08:25.745416 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=39 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).fault (111) Connection refused -8202> 2014-09-18 18:08:25.745772 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).connect error 10.242.42.164:6886/38815, (111) Connection refused -8201> 2014-09-18 18:08:25.745774 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).connect error 10.242.42.164:6885/38815, (111) Connection refused -8200> 2014-09-18 18:08:25.745789 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault (111) Connection refused -8199> 2014-09-18 18:08:25.745792 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault (111) Connection refused -8198> 2014-09-18 18:08:25.753301 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 329 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0x10e23f80 con 0x12562260 -8197> 2014-09-18 18:08:25.753317 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11702,11702], i have 11704, src has [1,11702] -8196> 2014-09-18 18:08:25.753342 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 330 ==== pg_notify(2.719(17) epoch 11702) v5 ==== 1601+0+0 (3004000157 0 0) 0xf2af860 con 0x12562260 -8195> 2014-09-18 18:08:25.753364 7f4c9f251700 5 -- op tracker -- , seq: 85699, time: 2014-09-18 18:08:25.753298, event: header_read, op: pg_notify(2.719(17) epoch 11702) -8194> 2014-09-18 18:08:25.753373 7f4c9f251700 5 -- op tracker -- , seq: 85699, time: 2014-09-18 18:08:25.753298, event: throttled, op: pg_notify(2.719(17) epoch 11702) -8193> 2014-09-18 18:08:25.753381 7f4c9f251700 5 -- op tracker -- , seq: 85699, time: 2014-09-18 18:08:25.753331, event: all_read, op: pg_notify(2.719(17) epoch 11702) -8192> 2014-09-18 18:08:25.753389 7f4c9f251700 5 -- op tracker -- , seq: 85699, time: 2014-09-18 18:08:25.753361, event: dispatched, op: pg_notify(2.719(17) epoch 11702) -8191> 2014-09-18 18:08:25.753397 7f4c9f251700 5 -- op tracker -- , seq: 85699, time: 2014-09-18 18:08:25.753395, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11702) -8190> 2014-09-18 18:08:25.753405 7f4c9f251700 5 -- op tracker -- , seq: 85699, time: 2014-09-18 18:08:25.753403, event: started, op: pg_notify(2.719(17) epoch 11702) -8189> 2014-09-18 18:08:25.753421 7f4c9f251700 5 -- op tracker -- , seq: 85699, time: 2014-09-18 18:08:25.753420, event: done, op: pg_notify(2.719(17) epoch 11702) -8188> 2014-09-18 18:08:25.756234 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.45 10.242.42.178:0/7086 105 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:25.755556) v2 ==== 47+0+0 (2722071786 0 0) 0xe0bdde0 con 0x110bb700 -8187> 2014-09-18 18:08:25.756244 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.45 10.242.42.178:0/7086 105 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:25.755556) v2 ==== 47+0+0 (2722071786 0 0) 0x11739e00 con 0x10a5d9c0 -8186> 2014-09-18 18:08:25.756267 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.755556) v2 -- ?+0 0xe76fc00 con 0x110bb700 -8185> 2014-09-18 18:08:25.756297 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.755556) v2 -- ?+0 0xf09c940 con 0x10a5d9c0 -8184> 2014-09-18 18:08:25.785348 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.6 10.242.42.166:0/22481 439 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:25.787232) v2 ==== 47+0+0 (715124098 0 0) 0x21f04d20 con 0x10201700 -8183> 2014-09-18 18:08:25.785386 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/22481 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.787232) v2 -- ?+0 0xe0bdde0 con 0x10201700 -8182> 2014-09-18 18:08:25.785538 7f4c9ca4c700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11703..11704 src has 1..11704) v3 -- ?+0 0x11bb8ac0 con 0x1088aec0 -8181> 2014-09-18 18:08:25.785696 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.6 10.242.42.166:0/22481 439 ==== osd_ping(ping e11701 stamp 2014-09-18 18:08:25.787232) v2 ==== 47+0+0 (715124098 0 0) 0x100ee960 con 0x14393080 -8180> 2014-09-18 18:08:25.785724 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/22481 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.787232) v2 -- ?+0 0x11739e00 con 0x14393080 -8179> 2014-09-18 18:08:25.818515 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3152 ==== pg_notify(2.410(11) epoch 11704) v5 ==== 1325+0+0 (50787774 0 0) 0xf80ed20 con 0x1395ec60 -8178> 2014-09-18 18:08:25.818549 7f4c9f251700 5 -- op tracker -- , seq: 85700, time: 2014-09-18 18:08:25.818424, event: header_read, op: pg_notify(2.410(11) epoch 11704) -8177> 2014-09-18 18:08:25.818570 7f4c9f251700 5 -- op tracker -- , seq: 85700, time: 2014-09-18 18:08:25.818427, event: throttled, op: pg_notify(2.410(11) epoch 11704) -8176> 2014-09-18 18:08:25.818576 7f4c9f251700 5 -- op tracker -- , seq: 85700, time: 2014-09-18 18:08:25.818483, event: all_read, op: pg_notify(2.410(11) epoch 11704) -8175> 2014-09-18 18:08:25.818582 7f4c9f251700 5 -- op tracker -- , seq: 85700, time: 2014-09-18 18:08:25.818535, event: dispatched, op: pg_notify(2.410(11) epoch 11704) -8174> 2014-09-18 18:08:25.818589 7f4c9f251700 5 -- op tracker -- , seq: 85700, time: 2014-09-18 18:08:25.818588, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11704) -8173> 2014-09-18 18:08:25.818599 7f4c9f251700 5 -- op tracker -- , seq: 85700, time: 2014-09-18 18:08:25.818597, event: started, op: pg_notify(2.410(11) epoch 11704) -8172> 2014-09-18 18:08:25.818623 7f4c9f251700 5 -- op tracker -- , seq: 85700, time: 2014-09-18 18:08:25.818622, event: done, op: pg_notify(2.410(11) epoch 11704) -8171> 2014-09-18 18:08:25.819734 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3153 ==== pg_notify(2.186(12) epoch 11704) v5 ==== 1386+0+0 (3042370418 0 0) 0xf810760 con 0x1395ec60 -8170> 2014-09-18 18:08:25.819753 7f4c9f251700 5 -- op tracker -- , seq: 85701, time: 2014-09-18 18:08:25.819627, event: header_read, op: pg_notify(2.186(12) epoch 11704) -8169> 2014-09-18 18:08:25.819764 7f4c9f251700 5 -- op tracker -- , seq: 85701, time: 2014-09-18 18:08:25.819628, event: throttled, op: pg_notify(2.186(12) epoch 11704) -8168> 2014-09-18 18:08:25.819770 7f4c9f251700 5 -- op tracker -- , seq: 85701, time: 2014-09-18 18:08:25.819674, event: all_read, op: pg_notify(2.186(12) epoch 11704) -8167> 2014-09-18 18:08:25.819776 7f4c9f251700 5 -- op tracker -- , seq: 85701, time: 2014-09-18 18:08:25.819749, event: dispatched, op: pg_notify(2.186(12) epoch 11704) -8166> 2014-09-18 18:08:25.819783 7f4c9f251700 5 -- op tracker -- , seq: 85701, time: 2014-09-18 18:08:25.819781, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11704) -8165> 2014-09-18 18:08:25.819791 7f4c9f251700 5 -- op tracker -- , seq: 85701, time: 2014-09-18 18:08:25.819789, event: started, op: pg_notify(2.186(12) epoch 11704) -8164> 2014-09-18 18:08:25.819811 7f4c9f251700 5 -- op tracker -- , seq: 85701, time: 2014-09-18 18:08:25.819810, event: done, op: pg_notify(2.186(12) epoch 11704) -8163> 2014-09-18 18:08:25.820651 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3154 ==== pg_notify(2.74e(11) epoch 11704) v5 ==== 1303+0+0 (1404269234 0 0) 0xf80fc20 con 0x1395ec60 -8162> 2014-09-18 18:08:25.820671 7f4c9f251700 5 -- op tracker -- , seq: 85702, time: 2014-09-18 18:08:25.820559, event: header_read, op: pg_notify(2.74e(11) epoch 11704) -8161> 2014-09-18 18:08:25.820682 7f4c9f251700 5 -- op tracker -- , seq: 85702, time: 2014-09-18 18:08:25.820560, event: throttled, op: pg_notify(2.74e(11) epoch 11704) -8160> 2014-09-18 18:08:25.820690 7f4c9f251700 5 -- op tracker -- , seq: 85702, time: 2014-09-18 18:08:25.820589, event: all_read, op: pg_notify(2.74e(11) epoch 11704) -8159> 2014-09-18 18:08:25.820698 7f4c9f251700 5 -- op tracker -- , seq: 85702, time: 2014-09-18 18:08:25.820667, event: dispatched, op: pg_notify(2.74e(11) epoch 11704) -8158> 2014-09-18 18:08:25.820706 7f4c9f251700 5 -- op tracker -- , seq: 85702, time: 2014-09-18 18:08:25.820705, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11704) -8157> 2014-09-18 18:08:25.820715 7f4c9f251700 5 -- op tracker -- , seq: 85702, time: 2014-09-18 18:08:25.820714, event: started, op: pg_notify(2.74e(11) epoch 11704) -8156> 2014-09-18 18:08:25.820734 7f4c9f251700 5 -- op tracker -- , seq: 85702, time: 2014-09-18 18:08:25.820732, event: done, op: pg_notify(2.74e(11) epoch 11704) -8155> 2014-09-18 18:08:25.837128 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1826 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x1182bb00 con 0x10f60520 -8154> 2014-09-18 18:08:25.837169 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11703,11703], i have 11704, src has [1,11703] -8153> 2014-09-18 18:08:25.837178 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1827 ==== pg_notify(2.4f6(15) epoch 11703) v5 ==== 1581+0+0 (3845601681 0 0) 0x1310d680 con 0x10f60520 -8152> 2014-09-18 18:08:25.837190 7f4c9f251700 5 -- op tracker -- , seq: 85703, time: 2014-09-18 18:08:25.837122, event: header_read, op: pg_notify(2.4f6(15) epoch 11703) -8151> 2014-09-18 18:08:25.837202 7f4c9f251700 5 -- op tracker -- , seq: 85703, time: 2014-09-18 18:08:25.837122, event: throttled, op: pg_notify(2.4f6(15) epoch 11703) -8150> 2014-09-18 18:08:25.837210 7f4c9f251700 5 -- op tracker -- , seq: 85703, time: 2014-09-18 18:08:25.837160, event: all_read, op: pg_notify(2.4f6(15) epoch 11703) -8149> 2014-09-18 18:08:25.837218 7f4c9f251700 5 -- op tracker -- , seq: 85703, time: 2014-09-18 18:08:25.837186, event: dispatched, op: pg_notify(2.4f6(15) epoch 11703) -8148> 2014-09-18 18:08:25.837227 7f4c9f251700 5 -- op tracker -- , seq: 85703, time: 2014-09-18 18:08:25.837225, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11703) -8147> 2014-09-18 18:08:25.837236 7f4c9f251700 5 -- op tracker -- , seq: 85703, time: 2014-09-18 18:08:25.837235, event: started, op: pg_notify(2.4f6(15) epoch 11703) -8146> 2014-09-18 18:08:25.837260 7f4c9f251700 5 -- op tracker -- , seq: 85703, time: 2014-09-18 18:08:25.837258, event: done, op: pg_notify(2.4f6(15) epoch 11703) -8145> 2014-09-18 18:08:25.839702 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1828 ==== pg_notify(2.517(11) epoch 11703) v5 ==== 1335+0+0 (1057151447 0 0) 0x1310c780 con 0x10f60520 -8144> 2014-09-18 18:08:25.839717 7f4c9f251700 5 -- op tracker -- , seq: 85704, time: 2014-09-18 18:08:25.839631, event: header_read, op: pg_notify(2.517(11) epoch 11703) -8143> 2014-09-18 18:08:25.839726 7f4c9f251700 5 -- op tracker -- , seq: 85704, time: 2014-09-18 18:08:25.839632, event: throttled, op: pg_notify(2.517(11) epoch 11703) -8142> 2014-09-18 18:08:25.839732 7f4c9f251700 5 -- op tracker -- , seq: 85704, time: 2014-09-18 18:08:25.839687, event: all_read, op: pg_notify(2.517(11) epoch 11703) -8141> 2014-09-18 18:08:25.839738 7f4c9f251700 5 -- op tracker -- , seq: 85704, time: 2014-09-18 18:08:25.839714, event: dispatched, op: pg_notify(2.517(11) epoch 11703) -8140> 2014-09-18 18:08:25.839744 7f4c9f251700 5 -- op tracker -- , seq: 85704, time: 2014-09-18 18:08:25.839743, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11703) -8139> 2014-09-18 18:08:25.839752 7f4c9f251700 5 -- op tracker -- , seq: 85704, time: 2014-09-18 18:08:25.839750, event: started, op: pg_notify(2.517(11) epoch 11703) -8138> 2014-09-18 18:08:25.839770 7f4c9f251700 5 -- op tracker -- , seq: 85704, time: 2014-09-18 18:08:25.839768, event: done, op: pg_notify(2.517(11) epoch 11703) -8137> 2014-09-18 18:08:25.840302 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1829 ==== pg_notify(2.1d8(9) epoch 11703) v5 ==== 1261+0+0 (4040286175 0 0) 0x1310e760 con 0x10f60520 -8136> 2014-09-18 18:08:25.840319 7f4c9f251700 5 -- op tracker -- , seq: 85705, time: 2014-09-18 18:08:25.840249, event: header_read, op: pg_notify(2.1d8(9) epoch 11703) -8135> 2014-09-18 18:08:25.840329 7f4c9f251700 5 -- op tracker -- , seq: 85705, time: 2014-09-18 18:08:25.840250, event: throttled, op: pg_notify(2.1d8(9) epoch 11703) -8134> 2014-09-18 18:08:25.840335 7f4c9f251700 5 -- op tracker -- , seq: 85705, time: 2014-09-18 18:08:25.840287, event: all_read, op: pg_notify(2.1d8(9) epoch 11703) -8133> 2014-09-18 18:08:25.840341 7f4c9f251700 5 -- op tracker -- , seq: 85705, time: 2014-09-18 18:08:25.840315, event: dispatched, op: pg_notify(2.1d8(9) epoch 11703) -8132> 2014-09-18 18:08:25.840347 7f4c9f251700 5 -- op tracker -- , seq: 85705, time: 2014-09-18 18:08:25.840346, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11703) -8131> 2014-09-18 18:08:25.840355 7f4c9f251700 5 -- op tracker -- , seq: 85705, time: 2014-09-18 18:08:25.840354, event: started, op: pg_notify(2.1d8(9) epoch 11703) -8130> 2014-09-18 18:08:25.840373 7f4c9f251700 5 -- op tracker -- , seq: 85705, time: 2014-09-18 18:08:25.840372, event: done, op: pg_notify(2.1d8(9) epoch 11703) -8129> 2014-09-18 18:08:25.841783 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1830 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x1182ad80 con 0x10f60520 -8128> 2014-09-18 18:08:25.841797 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8127> 2014-09-18 18:08:25.841837 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1831 ==== pg_notify(2.4f6(15) epoch 11704) v5 ==== 1581+0+0 (1898942847 0 0) 0x2b929480 con 0x10f60520 -8126> 2014-09-18 18:08:25.841848 7f4c9f251700 5 -- op tracker -- , seq: 85706, time: 2014-09-18 18:08:25.841787, event: header_read, op: pg_notify(2.4f6(15) epoch 11704) -8125> 2014-09-18 18:08:25.841857 7f4c9f251700 5 -- op tracker -- , seq: 85706, time: 2014-09-18 18:08:25.841788, event: throttled, op: pg_notify(2.4f6(15) epoch 11704) -8124> 2014-09-18 18:08:25.841863 7f4c9f251700 5 -- op tracker -- , seq: 85706, time: 2014-09-18 18:08:25.841828, event: all_read, op: pg_notify(2.4f6(15) epoch 11704) -8123> 2014-09-18 18:08:25.841869 7f4c9f251700 5 -- op tracker -- , seq: 85706, time: 2014-09-18 18:08:25.841845, event: dispatched, op: pg_notify(2.4f6(15) epoch 11704) -8122> 2014-09-18 18:08:25.841875 7f4c9f251700 5 -- op tracker -- , seq: 85706, time: 2014-09-18 18:08:25.841874, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11704) -8121> 2014-09-18 18:08:25.841883 7f4c9f251700 5 -- op tracker -- , seq: 85706, time: 2014-09-18 18:08:25.841881, event: started, op: pg_notify(2.4f6(15) epoch 11704) -8120> 2014-09-18 18:08:25.841900 7f4c9f251700 5 -- op tracker -- , seq: 85706, time: 2014-09-18 18:08:25.841899, event: done, op: pg_notify(2.4f6(15) epoch 11704) -8119> 2014-09-18 18:08:25.846907 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.9 10.242.42.166:0/24959 109 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:25.849204) v2 ==== 47+0+0 (607276268 0 0) 0x12115840 con 0x10a5c520 -8118> 2014-09-18 18:08:25.846931 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.849204) v2 -- ?+0 0x100ee960 con 0x10a5c520 -8117> 2014-09-18 18:08:25.846959 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.9 10.242.42.166:0/24959 109 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:25.849204) v2 ==== 47+0+0 (607276268 0 0) 0x1393d480 con 0x110bb9c0 -8116> 2014-09-18 18:08:25.846986 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11704 stamp 2014-09-18 18:08:25.849204) v2 -- ?+0 0x21f04d20 con 0x110bb9c0 -8115> 2014-09-18 18:08:25.855462 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1832 ==== pg_notify(2.517(11) epoch 11704) v5 ==== 1335+0+0 (813905303 0 0) 0x2b928940 con 0x10f60520 -8114> 2014-09-18 18:08:25.855477 7f4c9f251700 5 -- op tracker -- , seq: 85707, time: 2014-09-18 18:08:25.855416, event: header_read, op: pg_notify(2.517(11) epoch 11704) -8113> 2014-09-18 18:08:25.855484 7f4c9f251700 5 -- op tracker -- , seq: 85707, time: 2014-09-18 18:08:25.855417, event: throttled, op: pg_notify(2.517(11) epoch 11704) -8112> 2014-09-18 18:08:25.855488 7f4c9f251700 5 -- op tracker -- , seq: 85707, time: 2014-09-18 18:08:25.855449, event: all_read, op: pg_notify(2.517(11) epoch 11704) -8111> 2014-09-18 18:08:25.855492 7f4c9f251700 5 -- op tracker -- , seq: 85707, time: 2014-09-18 18:08:25.855474, event: dispatched, op: pg_notify(2.517(11) epoch 11704) -8110> 2014-09-18 18:08:25.855497 7f4c9f251700 5 -- op tracker -- , seq: 85707, time: 2014-09-18 18:08:25.855495, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11704) -8109> 2014-09-18 18:08:25.855502 7f4c9f251700 5 -- op tracker -- , seq: 85707, time: 2014-09-18 18:08:25.855501, event: started, op: pg_notify(2.517(11) epoch 11704) -8108> 2014-09-18 18:08:25.855526 7f4c9f251700 5 -- op tracker -- , seq: 85707, time: 2014-09-18 18:08:25.855524, event: done, op: pg_notify(2.517(11) epoch 11704) -8107> 2014-09-18 18:08:25.855904 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1833 ==== pg_notify(2.1d8(9) epoch 11704) v5 ==== 1261+0+0 (230384592 0 0) 0x2b926000 con 0x10f60520 -8106> 2014-09-18 18:08:25.855924 7f4c9f251700 5 -- op tracker -- , seq: 85708, time: 2014-09-18 18:08:25.855837, event: header_read, op: pg_notify(2.1d8(9) epoch 11704) -8105> 2014-09-18 18:08:25.855932 7f4c9f251700 5 -- op tracker -- , seq: 85708, time: 2014-09-18 18:08:25.855837, event: throttled, op: pg_notify(2.1d8(9) epoch 11704) -8104> 2014-09-18 18:08:25.855937 7f4c9f251700 5 -- op tracker -- , seq: 85708, time: 2014-09-18 18:08:25.855864, event: all_read, op: pg_notify(2.1d8(9) epoch 11704) -8103> 2014-09-18 18:08:25.855941 7f4c9f251700 5 -- op tracker -- , seq: 85708, time: 2014-09-18 18:08:25.855919, event: dispatched, op: pg_notify(2.1d8(9) epoch 11704) -8102> 2014-09-18 18:08:25.855945 7f4c9f251700 5 -- op tracker -- , seq: 85708, time: 2014-09-18 18:08:25.855944, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11704) -8101> 2014-09-18 18:08:25.855951 7f4c9f251700 5 -- op tracker -- , seq: 85708, time: 2014-09-18 18:08:25.855950, event: started, op: pg_notify(2.1d8(9) epoch 11704) -8100> 2014-09-18 18:08:25.855968 7f4c9f251700 5 -- op tracker -- , seq: 85708, time: 2014-09-18 18:08:25.855967, event: done, op: pg_notify(2.1d8(9) epoch 11704) -8099> 2014-09-18 18:08:25.889280 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3155 ==== pg_notify(2.4b1(13),2.4c1(15),2.78f(51),2.4c6(15) epoch 11704) v5 ==== 7874+0+0 (551813040 0 0) 0xf8110c0 con 0x1395ec60 -8098> 2014-09-18 18:08:25.889312 7f4c9f251700 5 -- op tracker -- , seq: 85709, time: 2014-09-18 18:08:25.889109, event: header_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(51),2.4c6(15) epoch 11704) -8097> 2014-09-18 18:08:25.889326 7f4c9f251700 5 -- op tracker -- , seq: 85709, time: 2014-09-18 18:08:25.889112, event: throttled, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(51),2.4c6(15) epoch 11704) -8096> 2014-09-18 18:08:25.889333 7f4c9f251700 5 -- op tracker -- , seq: 85709, time: 2014-09-18 18:08:25.889256, event: all_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(51),2.4c6(15) epoch 11704) -8095> 2014-09-18 18:08:25.889340 7f4c9f251700 5 -- op tracker -- , seq: 85709, time: 2014-09-18 18:08:25.889304, event: dispatched, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(51),2.4c6(15) epoch 11704) -8094> 2014-09-18 18:08:25.889348 7f4c9f251700 5 -- op tracker -- , seq: 85709, time: 2014-09-18 18:08:25.889346, event: waiting_for_osdmap, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(51),2.4c6(15) epoch 11704) -8093> 2014-09-18 18:08:25.889358 7f4c9f251700 5 -- op tracker -- , seq: 85709, time: 2014-09-18 18:08:25.889356, event: started, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(51),2.4c6(15) epoch 11704) -8092> 2014-09-18 18:08:25.889423 7f4c9f251700 5 -- op tracker -- , seq: 85709, time: 2014-09-18 18:08:25.889421, event: done, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(51),2.4c6(15) epoch 11704) -8091> 2014-09-18 18:08:25.889491 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3156 ==== pg_notify(2.799(9),2.4d0(10) epoch 11704) v5 ==== 2509+0+0 (1699278806 0 0) 0xf80fe00 con 0x1395ec60 -8090> 2014-09-18 18:08:25.889504 7f4c9f251700 5 -- op tracker -- , seq: 85710, time: 2014-09-18 18:08:25.889433, event: header_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11704) -8089> 2014-09-18 18:08:25.889511 7f4c9f251700 5 -- op tracker -- , seq: 85710, time: 2014-09-18 18:08:25.889434, event: throttled, op: pg_notify(2.799(9),2.4d0(10) epoch 11704) -8088> 2014-09-18 18:08:25.889518 7f4c9f251700 5 -- op tracker -- , seq: 85710, time: 2014-09-18 18:08:25.889480, event: all_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11704) -8087> 2014-09-18 18:08:25.889524 7f4c9f251700 5 -- op tracker -- , seq: 85710, time: 2014-09-18 18:08:25.889501, event: dispatched, op: pg_notify(2.799(9),2.4d0(10) epoch 11704) -8086> 2014-09-18 18:08:25.889530 7f4c9f251700 5 -- op tracker -- , seq: 85710, time: 2014-09-18 18:08:25.889529, event: waiting_for_osdmap, op: pg_notify(2.799(9),2.4d0(10) epoch 11704) -8085> 2014-09-18 18:08:25.889543 7f4c9f251700 5 -- op tracker -- , seq: 85710, time: 2014-09-18 18:08:25.889541, event: started, op: pg_notify(2.799(9),2.4d0(10) epoch 11704) -8084> 2014-09-18 18:08:25.889748 7f4c9f251700 5 -- op tracker -- , seq: 85710, time: 2014-09-18 18:08:25.889745, event: done, op: pg_notify(2.799(9),2.4d0(10) epoch 11704) -8083> 2014-09-18 18:08:25.937096 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 669 ==== osd_map(11702..11702 src has 1..11702) v3 ==== 2225+0+0 (1197198746 0 0) 0xeee9d40 con 0x1f103e40 -8082> 2014-09-18 18:08:25.937118 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11702,11702], i have 11704, src has [1,11702] -8081> 2014-09-18 18:08:25.937158 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 670 ==== pg_notify(2.4f6(15) epoch 11702) v5 ==== 1543+0+0 (2559540848 0 0) 0x2b93e1e0 con 0x1f103e40 -8080> 2014-09-18 18:08:25.937170 7f4c9f251700 5 -- op tracker -- , seq: 85711, time: 2014-09-18 18:08:25.937097, event: header_read, op: pg_notify(2.4f6(15) epoch 11702) -8079> 2014-09-18 18:08:25.937181 7f4c9f251700 5 -- op tracker -- , seq: 85711, time: 2014-09-18 18:08:25.937097, event: throttled, op: pg_notify(2.4f6(15) epoch 11702) -8078> 2014-09-18 18:08:25.937191 7f4c9f251700 5 -- op tracker -- , seq: 85711, time: 2014-09-18 18:08:25.937148, event: all_read, op: pg_notify(2.4f6(15) epoch 11702) -8077> 2014-09-18 18:08:25.937197 7f4c9f251700 5 -- op tracker -- , seq: 85711, time: 2014-09-18 18:08:25.937166, event: dispatched, op: pg_notify(2.4f6(15) epoch 11702) -8076> 2014-09-18 18:08:25.937204 7f4c9f251700 5 -- op tracker -- , seq: 85711, time: 2014-09-18 18:08:25.937202, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11702) -8075> 2014-09-18 18:08:25.937212 7f4c9f251700 5 -- op tracker -- , seq: 85711, time: 2014-09-18 18:08:25.937211, event: started, op: pg_notify(2.4f6(15) epoch 11702) -8074> 2014-09-18 18:08:25.937233 7f4c9f251700 5 -- op tracker -- , seq: 85711, time: 2014-09-18 18:08:25.937232, event: done, op: pg_notify(2.4f6(15) epoch 11702) -8073> 2014-09-18 18:08:25.939463 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 671 ==== pg_notify(2.799(10),2.4c6(15) epoch 11702) v5 ==== 2779+0+0 (2522693616 0 0) 0x2b940940 con 0x1f103e40 -8072> 2014-09-18 18:08:25.939483 7f4c9f251700 5 -- op tracker -- , seq: 85712, time: 2014-09-18 18:08:25.939393, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11702) -8071> 2014-09-18 18:08:25.939494 7f4c9f251700 5 -- op tracker -- , seq: 85712, time: 2014-09-18 18:08:25.939394, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11702) -8070> 2014-09-18 18:08:25.939501 7f4c9f251700 5 -- op tracker -- , seq: 85712, time: 2014-09-18 18:08:25.939446, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11702) -8069> 2014-09-18 18:08:25.939508 7f4c9f251700 5 -- op tracker -- , seq: 85712, time: 2014-09-18 18:08:25.939479, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11702) -8068> 2014-09-18 18:08:25.939515 7f4c9f251700 5 -- op tracker -- , seq: 85712, time: 2014-09-18 18:08:25.939514, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11702) -8067> 2014-09-18 18:08:25.939525 7f4c9f251700 5 -- op tracker -- , seq: 85712, time: 2014-09-18 18:08:25.939523, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11702) -8066> 2014-09-18 18:08:25.939551 7f4c9f251700 5 -- op tracker -- , seq: 85712, time: 2014-09-18 18:08:25.939549, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11702) -8065> 2014-09-18 18:08:25.940414 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 672 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x19df2d00 con 0x1f103e40 -8064> 2014-09-18 18:08:25.940430 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11703,11703], i have 11704, src has [1,11703] -8063> 2014-09-18 18:08:25.940453 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 673 ==== pg_notify(2.4f6(15) epoch 11703) v5 ==== 1543+0+0 (3236024710 0 0) 0x2b93fc20 con 0x1f103e40 -8062> 2014-09-18 18:08:25.940464 7f4c9f251700 5 -- op tracker -- , seq: 85713, time: 2014-09-18 18:08:25.940411, event: header_read, op: pg_notify(2.4f6(15) epoch 11703) -8061> 2014-09-18 18:08:25.940472 7f4c9f251700 5 -- op tracker -- , seq: 85713, time: 2014-09-18 18:08:25.940411, event: throttled, op: pg_notify(2.4f6(15) epoch 11703) -8060> 2014-09-18 18:08:25.940478 7f4c9f251700 5 -- op tracker -- , seq: 85713, time: 2014-09-18 18:08:25.940443, event: all_read, op: pg_notify(2.4f6(15) epoch 11703) -8059> 2014-09-18 18:08:25.940484 7f4c9f251700 5 -- op tracker -- , seq: 85713, time: 2014-09-18 18:08:25.940461, event: dispatched, op: pg_notify(2.4f6(15) epoch 11703) -8058> 2014-09-18 18:08:25.940490 7f4c9f251700 5 -- op tracker -- , seq: 85713, time: 2014-09-18 18:08:25.940489, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11703) -8057> 2014-09-18 18:08:25.940498 7f4c9f251700 5 -- op tracker -- , seq: 85713, time: 2014-09-18 18:08:25.940496, event: started, op: pg_notify(2.4f6(15) epoch 11703) -8056> 2014-09-18 18:08:25.940514 7f4c9f251700 5 -- op tracker -- , seq: 85713, time: 2014-09-18 18:08:25.940513, event: done, op: pg_notify(2.4f6(15) epoch 11703) -8055> 2014-09-18 18:08:25.945178 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 674 ==== pg_notify(2.799(10),2.4c6(15) epoch 11703) v5 ==== 2779+0+0 (1631217920 0 0) 0x2b941a20 con 0x1f103e40 -8054> 2014-09-18 18:08:25.945198 7f4c9f251700 5 -- op tracker -- , seq: 85714, time: 2014-09-18 18:08:25.945111, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11703) -8053> 2014-09-18 18:08:25.945208 7f4c9f251700 5 -- op tracker -- , seq: 85714, time: 2014-09-18 18:08:25.945112, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11703) -8052> 2014-09-18 18:08:25.945214 7f4c9f251700 5 -- op tracker -- , seq: 85714, time: 2014-09-18 18:08:25.945161, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11703) -8051> 2014-09-18 18:08:25.945220 7f4c9f251700 5 -- op tracker -- , seq: 85714, time: 2014-09-18 18:08:25.945193, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11703) -8050> 2014-09-18 18:08:25.945227 7f4c9f251700 5 -- op tracker -- , seq: 85714, time: 2014-09-18 18:08:25.945225, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11703) -8049> 2014-09-18 18:08:25.945235 7f4c9f251700 5 -- op tracker -- , seq: 85714, time: 2014-09-18 18:08:25.945234, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11703) -8048> 2014-09-18 18:08:25.945262 7f4c9f251700 5 -- op tracker -- , seq: 85714, time: 2014-09-18 18:08:25.945260, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11703) -8047> 2014-09-18 18:08:26.267049 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 331 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0x1440a400 con 0x12562260 -8046> 2014-09-18 18:08:26.267089 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11703,11703], i have 11704, src has [1,11703] -8045> 2014-09-18 18:08:26.267128 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 332 ==== pg_notify(2.719(17) epoch 11703) v5 ==== 1601+0+0 (1562750970 0 0) 0xf2aed20 con 0x12562260 -8044> 2014-09-18 18:08:26.267142 7f4c9f251700 5 -- op tracker -- , seq: 85715, time: 2014-09-18 18:08:26.267050, event: header_read, op: pg_notify(2.719(17) epoch 11703) -8043> 2014-09-18 18:08:26.267154 7f4c9f251700 5 -- op tracker -- , seq: 85715, time: 2014-09-18 18:08:26.267051, event: throttled, op: pg_notify(2.719(17) epoch 11703) -8042> 2014-09-18 18:08:26.267162 7f4c9f251700 5 -- op tracker -- , seq: 85715, time: 2014-09-18 18:08:26.267114, event: all_read, op: pg_notify(2.719(17) epoch 11703) -8041> 2014-09-18 18:08:26.267170 7f4c9f251700 5 -- op tracker -- , seq: 85715, time: 2014-09-18 18:08:26.267137, event: dispatched, op: pg_notify(2.719(17) epoch 11703) -8040> 2014-09-18 18:08:26.267179 7f4c9f251700 5 -- op tracker -- , seq: 85715, time: 2014-09-18 18:08:26.267178, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11703) -8039> 2014-09-18 18:08:26.267188 7f4c9f251700 5 -- op tracker -- , seq: 85715, time: 2014-09-18 18:08:26.267187, event: started, op: pg_notify(2.719(17) epoch 11703) -8038> 2014-09-18 18:08:26.267211 7f4c9f251700 5 -- op tracker -- , seq: 85715, time: 2014-09-18 18:08:26.267210, event: done, op: pg_notify(2.719(17) epoch 11703) -8037> 2014-09-18 18:08:26.269695 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 333 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x14409d40 con 0x12562260 -8036> 2014-09-18 18:08:26.269714 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -8035> 2014-09-18 18:08:26.269720 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 334 ==== pg_notify(2.719(17) epoch 11704) v5 ==== 1601+0+0 (3034855618 0 0) 0xf2b10c0 con 0x12562260 -8034> 2014-09-18 18:08:26.269741 7f4c9f251700 5 -- op tracker -- , seq: 85716, time: 2014-09-18 18:08:26.269617, event: header_read, op: pg_notify(2.719(17) epoch 11704) -8033> 2014-09-18 18:08:26.269750 7f4c9f251700 5 -- op tracker -- , seq: 85716, time: 2014-09-18 18:08:26.269617, event: throttled, op: pg_notify(2.719(17) epoch 11704) -8032> 2014-09-18 18:08:26.269754 7f4c9f251700 5 -- op tracker -- , seq: 85716, time: 2014-09-18 18:08:26.269645, event: all_read, op: pg_notify(2.719(17) epoch 11704) -8031> 2014-09-18 18:08:26.269758 7f4c9f251700 5 -- op tracker -- , seq: 85716, time: 2014-09-18 18:08:26.269725, event: dispatched, op: pg_notify(2.719(17) epoch 11704) -8030> 2014-09-18 18:08:26.269767 7f4c9f251700 5 -- op tracker -- , seq: 85716, time: 2014-09-18 18:08:26.269765, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11704) -8029> 2014-09-18 18:08:26.269775 7f4c9f251700 5 -- op tracker -- , seq: 85716, time: 2014-09-18 18:08:26.269773, event: started, op: pg_notify(2.719(17) epoch 11704) -8028> 2014-09-18 18:08:26.269793 7f4c9f251700 5 -- op tracker -- , seq: 85716, time: 2014-09-18 18:08:26.269792, event: done, op: pg_notify(2.719(17) epoch 11704) -8027> 2014-09-18 18:08:26.272874 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5055 ==== pg_notify(2.120(11) epoch 11701) v5 ==== 1315+0+0 (595764173 0 0) 0xfa26f00 con 0x1088aec0 -8026> 2014-09-18 18:08:26.272892 7f4c9f251700 5 -- op tracker -- , seq: 85717, time: 2014-09-18 18:08:26.272808, event: header_read, op: pg_notify(2.120(11) epoch 11701) -8025> 2014-09-18 18:08:26.272902 7f4c9f251700 5 -- op tracker -- , seq: 85717, time: 2014-09-18 18:08:26.272809, event: throttled, op: pg_notify(2.120(11) epoch 11701) -8024> 2014-09-18 18:08:26.272909 7f4c9f251700 5 -- op tracker -- , seq: 85717, time: 2014-09-18 18:08:26.272858, event: all_read, op: pg_notify(2.120(11) epoch 11701) -8023> 2014-09-18 18:08:26.272925 7f4c9f251700 5 -- op tracker -- , seq: 85717, time: 2014-09-18 18:08:26.272888, event: dispatched, op: pg_notify(2.120(11) epoch 11701) -8022> 2014-09-18 18:08:26.272931 7f4c9f251700 5 -- op tracker -- , seq: 85717, time: 2014-09-18 18:08:26.272930, event: waiting_for_osdmap, op: pg_notify(2.120(11) epoch 11701) -8021> 2014-09-18 18:08:26.272939 7f4c9f251700 5 -- op tracker -- , seq: 85717, time: 2014-09-18 18:08:26.272938, event: started, op: pg_notify(2.120(11) epoch 11701) -8020> 2014-09-18 18:08:26.272958 7f4c9f251700 5 -- op tracker -- , seq: 85717, time: 2014-09-18 18:08:26.272957, event: done, op: pg_notify(2.120(11) epoch 11701) -8019> 2014-09-18 18:08:26.274277 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5056 ==== pg_notify(2.40a(162) epoch 11701) v5 ==== 9736+0+0 (1519969816 0 0) 0x2b986f00 con 0x1088aec0 -8018> 2014-09-18 18:08:26.274292 7f4c9f251700 5 -- op tracker -- , seq: 85718, time: 2014-09-18 18:08:26.274067, event: header_read, op: pg_notify(2.40a(162) epoch 11701) -8017> 2014-09-18 18:08:26.274312 7f4c9f251700 5 -- op tracker -- , seq: 85718, time: 2014-09-18 18:08:26.274068, event: throttled, op: pg_notify(2.40a(162) epoch 11701) -8016> 2014-09-18 18:08:26.274319 7f4c9f251700 5 -- op tracker -- , seq: 85718, time: 2014-09-18 18:08:26.274263, event: all_read, op: pg_notify(2.40a(162) epoch 11701) -8015> 2014-09-18 18:08:26.274325 7f4c9f251700 5 -- op tracker -- , seq: 85718, time: 2014-09-18 18:08:26.274288, event: dispatched, op: pg_notify(2.40a(162) epoch 11701) -8014> 2014-09-18 18:08:26.274332 7f4c9f251700 5 -- op tracker -- , seq: 85718, time: 2014-09-18 18:08:26.274330, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11701) -8013> 2014-09-18 18:08:26.274340 7f4c9f251700 5 -- op tracker -- , seq: 85718, time: 2014-09-18 18:08:26.274338, event: started, op: pg_notify(2.40a(162) epoch 11701) -8012> 2014-09-18 18:08:26.274366 7f4c9f251700 5 -- op tracker -- , seq: 85718, time: 2014-09-18 18:08:26.274365, event: done, op: pg_notify(2.40a(162) epoch 11701) -8011> 2014-09-18 18:08:26.277626 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5057 ==== pg_notify(2.1a2(20) epoch 11701) v5 ==== 1798+0+0 (3001822016 0 0) 0xf96b680 con 0x1088aec0 -8010> 2014-09-18 18:08:26.277641 7f4c9f251700 5 -- op tracker -- , seq: 85719, time: 2014-09-18 18:08:26.277568, event: header_read, op: pg_notify(2.1a2(20) epoch 11701) -8009> 2014-09-18 18:08:26.277649 7f4c9f251700 5 -- op tracker -- , seq: 85719, time: 2014-09-18 18:08:26.277568, event: throttled, op: pg_notify(2.1a2(20) epoch 11701) -8008> 2014-09-18 18:08:26.277655 7f4c9f251700 5 -- op tracker -- , seq: 85719, time: 2014-09-18 18:08:26.277614, event: all_read, op: pg_notify(2.1a2(20) epoch 11701) -8007> 2014-09-18 18:08:26.277662 7f4c9f251700 5 -- op tracker -- , seq: 85719, time: 2014-09-18 18:08:26.277637, event: dispatched, op: pg_notify(2.1a2(20) epoch 11701) -8006> 2014-09-18 18:08:26.277668 7f4c9f251700 5 -- op tracker -- , seq: 85719, time: 2014-09-18 18:08:26.277667, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11701) -8005> 2014-09-18 18:08:26.277676 7f4c9f251700 5 -- op tracker -- , seq: 85719, time: 2014-09-18 18:08:26.277675, event: started, op: pg_notify(2.1a2(20) epoch 11701) -8004> 2014-09-18 18:08:26.277693 7f4c9f251700 5 -- op tracker -- , seq: 85719, time: 2014-09-18 18:08:26.277691, event: done, op: pg_notify(2.1a2(20) epoch 11701) -8003> 2014-09-18 18:08:26.330951 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5058 ==== pg_notify(2.4c1(15) epoch 11701) v5 ==== 1507+0+0 (1692337956 0 0) 0xf96cee0 con 0x1088aec0 -8002> 2014-09-18 18:08:26.330974 7f4c9f251700 5 -- op tracker -- , seq: 85720, time: 2014-09-18 18:08:26.330882, event: header_read, op: pg_notify(2.4c1(15) epoch 11701) -8001> 2014-09-18 18:08:26.330986 7f4c9f251700 5 -- op tracker -- , seq: 85720, time: 2014-09-18 18:08:26.330884, event: throttled, op: pg_notify(2.4c1(15) epoch 11701) -8000> 2014-09-18 18:08:26.330992 7f4c9f251700 5 -- op tracker -- , seq: 85720, time: 2014-09-18 18:08:26.330933, event: all_read, op: pg_notify(2.4c1(15) epoch 11701) -7999> 2014-09-18 18:08:26.330998 7f4c9f251700 5 -- op tracker -- , seq: 85720, time: 2014-09-18 18:08:26.330969, event: dispatched, op: pg_notify(2.4c1(15) epoch 11701) -7998> 2014-09-18 18:08:26.331013 7f4c9f251700 5 -- op tracker -- , seq: 85720, time: 2014-09-18 18:08:26.331011, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11701) -7997> 2014-09-18 18:08:26.331021 7f4c9f251700 5 -- op tracker -- , seq: 85720, time: 2014-09-18 18:08:26.331020, event: started, op: pg_notify(2.4c1(15) epoch 11701) -7996> 2014-09-18 18:08:26.331042 7f4c9f251700 5 -- op tracker -- , seq: 85720, time: 2014-09-18 18:08:26.331040, event: done, op: pg_notify(2.4c1(15) epoch 11701) -7995> 2014-09-18 18:08:26.334838 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 947 ==== pg_notify(2.517(11),2.4c6(15) epoch 11702) v5 ==== 2834+0+0 (2258577451 0 0) 0x1624ffe0 con 0x10a878c0 -7994> 2014-09-18 18:08:26.334858 7f4c9f251700 5 -- op tracker -- , seq: 85721, time: 2014-09-18 18:08:26.334726, event: header_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11702) -7993> 2014-09-18 18:08:26.334869 7f4c9f251700 5 -- op tracker -- , seq: 85721, time: 2014-09-18 18:08:26.334727, event: throttled, op: pg_notify(2.517(11),2.4c6(15) epoch 11702) -7992> 2014-09-18 18:08:26.334876 7f4c9f251700 5 -- op tracker -- , seq: 85721, time: 2014-09-18 18:08:26.334799, event: all_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11702) -7991> 2014-09-18 18:08:26.334882 7f4c9f251700 5 -- op tracker -- , seq: 85721, time: 2014-09-18 18:08:26.334854, event: dispatched, op: pg_notify(2.517(11),2.4c6(15) epoch 11702) -7990> 2014-09-18 18:08:26.334889 7f4c9f251700 5 -- op tracker -- , seq: 85721, time: 2014-09-18 18:08:26.334887, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.4c6(15) epoch 11702) -7989> 2014-09-18 18:08:26.334898 7f4c9f251700 5 -- op tracker -- , seq: 85721, time: 2014-09-18 18:08:26.334896, event: started, op: pg_notify(2.517(11),2.4c6(15) epoch 11702) -7988> 2014-09-18 18:08:26.334928 7f4c9f251700 5 -- op tracker -- , seq: 85721, time: 2014-09-18 18:08:26.334927, event: done, op: pg_notify(2.517(11),2.4c6(15) epoch 11702) -7987> 2014-09-18 18:08:26.388326 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 948 ==== osd_map(11703..11703 src has 1..11703) v3 ==== 1004+0+0 (3639482165 0 0) 0xf85c880 con 0x10a878c0 -7986> 2014-09-18 18:08:26.388366 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11703,11703], i have 11704, src has [1,11703] -7985> 2014-09-18 18:08:26.388375 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 949 ==== pg_notify(2.517(11) epoch 11703) v5 ==== 1335+0+0 (1967579380 0 0) 0x1624eb40 con 0x10a878c0 -7984> 2014-09-18 18:08:26.388386 7f4c9f251700 5 -- op tracker -- , seq: 85722, time: 2014-09-18 18:08:26.388303, event: header_read, op: pg_notify(2.517(11) epoch 11703) -7983> 2014-09-18 18:08:26.388398 7f4c9f251700 5 -- op tracker -- , seq: 85722, time: 2014-09-18 18:08:26.388304, event: throttled, op: pg_notify(2.517(11) epoch 11703) -7982> 2014-09-18 18:08:26.388405 7f4c9f251700 5 -- op tracker -- , seq: 85722, time: 2014-09-18 18:08:26.388356, event: all_read, op: pg_notify(2.517(11) epoch 11703) -7981> 2014-09-18 18:08:26.388421 7f4c9f251700 5 -- op tracker -- , seq: 85722, time: 2014-09-18 18:08:26.388382, event: dispatched, op: pg_notify(2.517(11) epoch 11703) -7980> 2014-09-18 18:08:26.388427 7f4c9f251700 5 -- op tracker -- , seq: 85722, time: 2014-09-18 18:08:26.388426, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11703) -7979> 2014-09-18 18:08:26.388436 7f4c9f251700 5 -- op tracker -- , seq: 85722, time: 2014-09-18 18:08:26.388435, event: started, op: pg_notify(2.517(11) epoch 11703) -7978> 2014-09-18 18:08:26.388459 7f4c9f251700 5 -- op tracker -- , seq: 85722, time: 2014-09-18 18:08:26.388458, event: done, op: pg_notify(2.517(11) epoch 11703) -7977> 2014-09-18 18:08:26.388951 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 950 ==== pg_notify(2.4c6(15) epoch 11703) v5 ==== 1511+0+0 (852857322 0 0) 0x1624e1e0 con 0x10a878c0 -7976> 2014-09-18 18:08:26.388970 7f4c9f251700 5 -- op tracker -- , seq: 85723, time: 2014-09-18 18:08:26.388858, event: header_read, op: pg_notify(2.4c6(15) epoch 11703) -7975> 2014-09-18 18:08:26.388981 7f4c9f251700 5 -- op tracker -- , seq: 85723, time: 2014-09-18 18:08:26.388859, event: throttled, op: pg_notify(2.4c6(15) epoch 11703) -7974> 2014-09-18 18:08:26.388987 7f4c9f251700 5 -- op tracker -- , seq: 85723, time: 2014-09-18 18:08:26.388914, event: all_read, op: pg_notify(2.4c6(15) epoch 11703) -7973> 2014-09-18 18:08:26.388993 7f4c9f251700 5 -- op tracker -- , seq: 85723, time: 2014-09-18 18:08:26.388966, event: dispatched, op: pg_notify(2.4c6(15) epoch 11703) -7972> 2014-09-18 18:08:26.389000 7f4c9f251700 5 -- op tracker -- , seq: 85723, time: 2014-09-18 18:08:26.388998, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11703) -7971> 2014-09-18 18:08:26.389014 7f4c9f251700 5 -- op tracker -- , seq: 85723, time: 2014-09-18 18:08:26.389012, event: started, op: pg_notify(2.4c6(15) epoch 11703) -7970> 2014-09-18 18:08:26.389034 7f4c9f251700 5 -- op tracker -- , seq: 85723, time: 2014-09-18 18:08:26.389032, event: done, op: pg_notify(2.4c6(15) epoch 11703) -7969> 2014-09-18 18:08:26.396234 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 951 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x1182cac0 con 0x10a878c0 -7968> 2014-09-18 18:08:26.396253 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -7967> 2014-09-18 18:08:26.396259 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 952 ==== pg_notify(2.517(11) epoch 11704) v5 ==== 1335+0+0 (2059865268 0 0) 0x16250580 con 0x10a878c0 -7966> 2014-09-18 18:08:26.396270 7f4c9f251700 5 -- op tracker -- , seq: 85724, time: 2014-09-18 18:08:26.396206, event: header_read, op: pg_notify(2.517(11) epoch 11704) -7965> 2014-09-18 18:08:26.396280 7f4c9f251700 5 -- op tracker -- , seq: 85724, time: 2014-09-18 18:08:26.396206, event: throttled, op: pg_notify(2.517(11) epoch 11704) -7964> 2014-09-18 18:08:26.396287 7f4c9f251700 5 -- op tracker -- , seq: 85724, time: 2014-09-18 18:08:26.396245, event: all_read, op: pg_notify(2.517(11) epoch 11704) -7963> 2014-09-18 18:08:26.396294 7f4c9f251700 5 -- op tracker -- , seq: 85724, time: 2014-09-18 18:08:26.396267, event: dispatched, op: pg_notify(2.517(11) epoch 11704) -7962> 2014-09-18 18:08:26.396300 7f4c9f251700 5 -- op tracker -- , seq: 85724, time: 2014-09-18 18:08:26.396299, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11704) -7961> 2014-09-18 18:08:26.396308 7f4c9f251700 5 -- op tracker -- , seq: 85724, time: 2014-09-18 18:08:26.396307, event: started, op: pg_notify(2.517(11) epoch 11704) -7960> 2014-09-18 18:08:26.396327 7f4c9f251700 5 -- op tracker -- , seq: 85724, time: 2014-09-18 18:08:26.396325, event: done, op: pg_notify(2.517(11) epoch 11704) -7959> 2014-09-18 18:08:26.396903 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 953 ==== pg_notify(2.4c6(15) epoch 11704) v5 ==== 1511+0+0 (1845599199 0 0) 0x1624fe00 con 0x10a878c0 -7958> 2014-09-18 18:08:26.396923 7f4c9f251700 5 -- op tracker -- , seq: 85725, time: 2014-09-18 18:08:26.396811, event: header_read, op: pg_notify(2.4c6(15) epoch 11704) -7957> 2014-09-18 18:08:26.396933 7f4c9f251700 5 -- op tracker -- , seq: 85725, time: 2014-09-18 18:08:26.396812, event: throttled, op: pg_notify(2.4c6(15) epoch 11704) -7956> 2014-09-18 18:08:26.396939 7f4c9f251700 5 -- op tracker -- , seq: 85725, time: 2014-09-18 18:08:26.396868, event: all_read, op: pg_notify(2.4c6(15) epoch 11704) -7955> 2014-09-18 18:08:26.396945 7f4c9f251700 5 -- op tracker -- , seq: 85725, time: 2014-09-18 18:08:26.396918, event: dispatched, op: pg_notify(2.4c6(15) epoch 11704) -7954> 2014-09-18 18:08:26.396952 7f4c9f251700 5 -- op tracker -- , seq: 85725, time: 2014-09-18 18:08:26.396950, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11704) -7953> 2014-09-18 18:08:26.396959 7f4c9f251700 5 -- op tracker -- , seq: 85725, time: 2014-09-18 18:08:26.396958, event: started, op: pg_notify(2.4c6(15) epoch 11704) -7952> 2014-09-18 18:08:26.396977 7f4c9f251700 5 -- op tracker -- , seq: 85725, time: 2014-09-18 18:08:26.396976, event: done, op: pg_notify(2.4c6(15) epoch 11704) -7951> 2014-09-18 18:08:26.401530 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 675 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0x19df3600 con 0x1f103e40 -7950> 2014-09-18 18:08:26.401550 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11704,11704], i have 11704, src has [1,11704] -7949> 2014-09-18 18:08:26.401560 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 676 ==== pg_notify(2.4f6(15) epoch 11704) v5 ==== 1543+0+0 (2225699559 0 0) 0xec68000 con 0x1f103e40 -7948> 2014-09-18 18:08:26.401571 7f4c9f251700 5 -- op tracker -- , seq: 85726, time: 2014-09-18 18:08:26.401503, event: header_read, op: pg_notify(2.4f6(15) epoch 11704) -7947> 2014-09-18 18:08:26.401581 7f4c9f251700 5 -- op tracker -- , seq: 85726, time: 2014-09-18 18:08:26.401503, event: throttled, op: pg_notify(2.4f6(15) epoch 11704) -7946> 2014-09-18 18:08:26.401588 7f4c9f251700 5 -- op tracker -- , seq: 85726, time: 2014-09-18 18:08:26.401535, event: all_read, op: pg_notify(2.4f6(15) epoch 11704) -7945> 2014-09-18 18:08:26.401593 7f4c9f251700 5 -- op tracker -- , seq: 85726, time: 2014-09-18 18:08:26.401567, event: dispatched, op: pg_notify(2.4f6(15) epoch 11704) -7944> 2014-09-18 18:08:26.401600 7f4c9f251700 5 -- op tracker -- , seq: 85726, time: 2014-09-18 18:08:26.401598, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11704) -7943> 2014-09-18 18:08:26.401608 7f4c9f251700 5 -- op tracker -- , seq: 85726, time: 2014-09-18 18:08:26.401606, event: started, op: pg_notify(2.4f6(15) epoch 11704) -7942> 2014-09-18 18:08:26.401627 7f4c9f251700 5 -- op tracker -- , seq: 85726, time: 2014-09-18 18:08:26.401626, event: done, op: pg_notify(2.4f6(15) epoch 11704) -7941> 2014-09-18 18:08:26.412350 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 677 ==== pg_notify(2.799(10),2.4c6(15) epoch 11704) v5 ==== 2779+0+0 (208261989 0 0) 0xf1e7fe0 con 0x1f103e40 -7940> 2014-09-18 18:08:26.412371 7f4c9f251700 5 -- op tracker -- , seq: 85727, time: 2014-09-18 18:08:26.412272, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11704) -7939> 2014-09-18 18:08:26.412383 7f4c9f251700 5 -- op tracker -- , seq: 85727, time: 2014-09-18 18:08:26.412273, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11704) -7938> 2014-09-18 18:08:26.412389 7f4c9f251700 5 -- op tracker -- , seq: 85727, time: 2014-09-18 18:08:26.412334, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11704) -7937> 2014-09-18 18:08:26.412395 7f4c9f251700 5 -- op tracker -- , seq: 85727, time: 2014-09-18 18:08:26.412367, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11704) -7936> 2014-09-18 18:08:26.412402 7f4c9f251700 5 -- op tracker -- , seq: 85727, time: 2014-09-18 18:08:26.412400, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11704) -7935> 2014-09-18 18:08:26.412411 7f4c9f251700 5 -- op tracker -- , seq: 85727, time: 2014-09-18 18:08:26.412409, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11704) -7934> 2014-09-18 18:08:26.412444 7f4c9f251700 5 -- op tracker -- , seq: 85727, time: 2014-09-18 18:08:26.412442, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11704) -7933> 2014-09-18 18:08:26.545635 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=39 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).connect error 10.242.42.164:6884/38815, (111) Connection refused -7932> 2014-09-18 18:08:26.545670 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=39 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).fault (111) Connection refused -7931> 2014-09-18 18:08:26.545913 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).connect error 10.242.42.164:6885/38815, (111) Connection refused -7930> 2014-09-18 18:08:26.545913 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).connect error 10.242.42.164:6886/38815, (111) Connection refused -7929> 2014-09-18 18:08:26.545939 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault (111) Connection refused -7928> 2014-09-18 18:08:26.545937 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=127 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault (111) Connection refused -7927> 2014-09-18 18:08:26.618335 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2339 ==== pg_info(1 pgs e11704:2.280) v4 ==== 742+0+0 (2430100778 0 0) 0xf109de0 con 0xdd8c940 -7926> 2014-09-18 18:08:26.618365 7f4c9f251700 5 -- op tracker -- , seq: 85728, time: 2014-09-18 18:08:26.618219, event: header_read, op: pg_info(1 pgs e11704:2.280) -7925> 2014-09-18 18:08:26.618379 7f4c9f251700 5 -- op tracker -- , seq: 85728, time: 2014-09-18 18:08:26.618222, event: throttled, op: pg_info(1 pgs e11704:2.280) -7924> 2014-09-18 18:08:26.618385 7f4c9f251700 5 -- op tracker -- , seq: 85728, time: 2014-09-18 18:08:26.618283, event: all_read, op: pg_info(1 pgs e11704:2.280) -7923> 2014-09-18 18:08:26.618391 7f4c9f251700 5 -- op tracker -- , seq: 85728, time: 2014-09-18 18:08:26.618357, event: dispatched, op: pg_info(1 pgs e11704:2.280) -7922> 2014-09-18 18:08:26.618398 7f4c9f251700 5 -- op tracker -- , seq: 85728, time: 2014-09-18 18:08:26.618396, event: waiting_for_osdmap, op: pg_info(1 pgs e11704:2.280) -7921> 2014-09-18 18:08:26.618409 7f4c9f251700 5 -- op tracker -- , seq: 85728, time: 2014-09-18 18:08:26.618408, event: started, op: pg_info(1 pgs e11704:2.280) -7920> 2014-09-18 18:08:26.618439 7f4c9f251700 5 -- op tracker -- , seq: 85728, time: 2014-09-18 18:08:26.618437, event: done, op: pg_info(1 pgs e11704:2.280) -7919> 2014-09-18 18:08:26.674186 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 778 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x2d7fda80 con 0xe9a27e0 -7918> 2014-09-18 18:08:26.674233 7f4c9f251700 3 osd.42 11704 handle_osd_map epochs [11705,11705], i have 11704, src has [1,11705] -7917> 2014-09-18 18:08:26.676961 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 779 ==== pg_notify(2.2d8(173) epoch 11705) v5 ==== 10757+0+0 (1609794061 0 0) 0x1421fe00 con 0xe9a27e0 -7916> 2014-09-18 18:08:26.676975 7f4c9f251700 5 -- op tracker -- , seq: 85729, time: 2014-09-18 18:08:26.674276, event: header_read, op: pg_notify(2.2d8(173) epoch 11705) -7915> 2014-09-18 18:08:26.676985 7f4c9f251700 5 -- op tracker -- , seq: 85729, time: 2014-09-18 18:08:26.674277, event: throttled, op: pg_notify(2.2d8(173) epoch 11705) -7914> 2014-09-18 18:08:26.677035 7f4c9f251700 5 -- op tracker -- , seq: 85729, time: 2014-09-18 18:08:26.674509, event: all_read, op: pg_notify(2.2d8(173) epoch 11705) -7913> 2014-09-18 18:08:26.677041 7f4c9f251700 5 -- op tracker -- , seq: 85729, time: 2014-09-18 18:08:26.676972, event: dispatched, op: pg_notify(2.2d8(173) epoch 11705) -7912> 2014-09-18 18:08:26.677048 7f4c9f251700 5 -- op tracker -- , seq: 85729, time: 2014-09-18 18:08:26.677046, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11705) -7911> 2014-09-18 18:08:26.677055 7f4c9f251700 5 -- op tracker -- , seq: 85729, time: 2014-09-18 18:08:26.677054, event: started, op: pg_notify(2.2d8(173) epoch 11705) -7910> 2014-09-18 18:08:26.677070 7f4c9f251700 5 -- op tracker -- , seq: 85729, time: 2014-09-18 18:08:26.677069, event: done, op: pg_notify(2.2d8(173) epoch 11705) -7909> 2014-09-18 18:08:26.677226 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11705..11705 src has 1..11705) v3 -- ?+0 0x1e283680 con 0x140f95a0 -7908> 2014-09-18 18:08:26.677243 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11705) v5 -- ?+0 0x24f6d4a0 con 0x140f95a0 -7907> 2014-09-18 18:08:26.677607 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11705..11705 src has 1..11705) v3 -- ?+0 0x2d7fc640 con 0x1395edc0 -7906> 2014-09-18 18:08:26.677652 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11705) v5 -- ?+0 0x21f05fe0 con 0x1395edc0 -7905> 2014-09-18 18:08:26.677837 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11705..11705 src has 1..11705) v3 -- ?+0 0x2d7fad80 con 0x1395f1e0 -7904> 2014-09-18 18:08:26.677849 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11705) v5 -- ?+0 0xddf23c0 con 0x1395f1e0 -7903> 2014-09-18 18:08:26.678210 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11705) v5 -- ?+0 0x146e1de0 con 0x140f95a0 -7902> 2014-09-18 18:08:26.678338 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1030 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0xe937b00 con 0x14358260 -7901> 2014-09-18 18:08:26.678357 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7900> 2014-09-18 18:08:26.678390 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1031 ==== pg_notify(2.719(17) epoch 11705) v5 ==== 1601+0+0 (363334622 0 0) 0xf730d00 con 0x14358260 -7899> 2014-09-18 18:08:26.678402 7f4c9f251700 5 -- op tracker -- , seq: 85730, time: 2014-09-18 18:08:26.678345, event: header_read, op: pg_notify(2.719(17) epoch 11705) -7898> 2014-09-18 18:08:26.678413 7f4c9f251700 5 -- op tracker -- , seq: 85730, time: 2014-09-18 18:08:26.678346, event: throttled, op: pg_notify(2.719(17) epoch 11705) -7897> 2014-09-18 18:08:26.678419 7f4c9f251700 5 -- op tracker -- , seq: 85730, time: 2014-09-18 18:08:26.678376, event: all_read, op: pg_notify(2.719(17) epoch 11705) -7896> 2014-09-18 18:08:26.678425 7f4c9f251700 5 -- op tracker -- , seq: 85730, time: 2014-09-18 18:08:26.678398, event: dispatched, op: pg_notify(2.719(17) epoch 11705) -7895> 2014-09-18 18:08:26.678426 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11705..11705 src has 1..11705) v3 -- ?+0 0x2d7fd840 con 0x1395ef20 -7894> 2014-09-18 18:08:26.678431 7f4c9f251700 5 -- op tracker -- , seq: 85730, time: 2014-09-18 18:08:26.678430, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11705) -7893> 2014-09-18 18:08:26.678445 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(10) epoch 11705) v5 -- ?+0 0xf7310c0 con 0x1395ef20 -7892> 2014-09-18 18:08:26.678448 7f4c9f251700 5 -- op tracker -- , seq: 85730, time: 2014-09-18 18:08:26.678446, event: started, op: pg_notify(2.719(17) epoch 11705) -7891> 2014-09-18 18:08:26.678463 7f4c9f251700 5 -- op tracker -- , seq: 85730, time: 2014-09-18 18:08:26.678462, event: done, op: pg_notify(2.719(17) epoch 11705) -7890> 2014-09-18 18:08:26.679105 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2588 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x19df1680 con 0x4900b00 -7889> 2014-09-18 18:08:26.679111 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11705..11705 src has 1..11705) v3 -- ?+0 0x139a6000 con 0x23a74f20 -7888> 2014-09-18 18:08:26.679121 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7887> 2014-09-18 18:08:26.679124 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11705) v5 -- ?+0 0x24fd3840 con 0x23a74f20 -7886> 2014-09-18 18:08:26.679218 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2589 ==== pg_notify(2.71c(12) epoch 11705) v5 ==== 1424+0+0 (551004330 0 0) 0xe2874a0 con 0x4900b00 -7885> 2014-09-18 18:08:26.679237 7f4c9f251700 5 -- op tracker -- , seq: 85731, time: 2014-09-18 18:08:26.679118, event: header_read, op: pg_notify(2.71c(12) epoch 11705) -7884> 2014-09-18 18:08:26.679252 7f4c9f251700 5 -- op tracker -- , seq: 85731, time: 2014-09-18 18:08:26.679118, event: throttled, op: pg_notify(2.71c(12) epoch 11705) -7883> 2014-09-18 18:08:26.679263 7f4c9f251700 5 -- op tracker -- , seq: 85731, time: 2014-09-18 18:08:26.679162, event: all_read, op: pg_notify(2.71c(12) epoch 11705) -7882> 2014-09-18 18:08:26.679274 7f4c9f251700 5 -- op tracker -- , seq: 85731, time: 2014-09-18 18:08:26.679232, event: dispatched, op: pg_notify(2.71c(12) epoch 11705) -7881> 2014-09-18 18:08:26.679284 7f4c9f251700 5 -- op tracker -- , seq: 85731, time: 2014-09-18 18:08:26.679281, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11705) -7880> 2014-09-18 18:08:26.679296 7f4c9f251700 5 -- op tracker -- , seq: 85731, time: 2014-09-18 18:08:26.679294, event: started, op: pg_notify(2.71c(12) epoch 11705) -7879> 2014-09-18 18:08:26.679334 7f4c9f251700 5 -- op tracker -- , seq: 85731, time: 2014-09-18 18:08:26.679332, event: done, op: pg_notify(2.71c(12) epoch 11705) -7878> 2014-09-18 18:08:26.680699 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 521 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x2d7fd840 con 0x1395ef20 -7877> 2014-09-18 18:08:26.680745 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7876> 2014-09-18 18:08:26.680757 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 522 ==== pg_notify(2.4d0(10) epoch 11705) v5 ==== 1292+0+0 (1623847228 0 0) 0xf7310c0 con 0x1395ef20 -7875> 2014-09-18 18:08:26.680770 7f4c9f251700 5 -- op tracker -- , seq: 85732, time: 2014-09-18 18:08:26.680645, event: header_read, op: pg_notify(2.4d0(10) epoch 11705) -7874> 2014-09-18 18:08:26.680783 7f4c9f251700 5 -- op tracker -- , seq: 85732, time: 2014-09-18 18:08:26.680646, event: throttled, op: pg_notify(2.4d0(10) epoch 11705) -7873> 2014-09-18 18:08:26.680788 7f4c9f251700 5 -- op tracker -- , seq: 85732, time: 2014-09-18 18:08:26.680670, event: all_read, op: pg_notify(2.4d0(10) epoch 11705) -7872> 2014-09-18 18:08:26.680791 7f4c9f251700 5 -- op tracker -- , seq: 85732, time: 2014-09-18 18:08:26.680765, event: dispatched, op: pg_notify(2.4d0(10) epoch 11705) -7871> 2014-09-18 18:08:26.680809 7f4c9f251700 5 -- op tracker -- , seq: 85732, time: 2014-09-18 18:08:26.680808, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11705) -7870> 2014-09-18 18:08:26.680818 7f4c9f251700 5 -- op tracker -- , seq: 85732, time: 2014-09-18 18:08:26.680816, event: started, op: pg_notify(2.4d0(10) epoch 11705) -7869> 2014-09-18 18:08:26.680842 7f4c9f251700 5 -- op tracker -- , seq: 85732, time: 2014-09-18 18:08:26.680840, event: done, op: pg_notify(2.4d0(10) epoch 11705) -7868> 2014-09-18 18:08:26.681570 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1256 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0xeee9b00 con 0xe8db760 -7867> 2014-09-18 18:08:26.681589 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7866> 2014-09-18 18:08:26.681628 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1257 ==== pg_notify(2.17e(17) epoch 11705) v5 ==== 1593+0+0 (3381414688 0 0) 0xf1092a0 con 0xe8db760 -7865> 2014-09-18 18:08:26.681643 7f4c9f251700 5 -- op tracker -- , seq: 85733, time: 2014-09-18 18:08:26.681534, event: header_read, op: pg_notify(2.17e(17) epoch 11705) -7864> 2014-09-18 18:08:26.681654 7f4c9f251700 5 -- op tracker -- , seq: 85733, time: 2014-09-18 18:08:26.681534, event: throttled, op: pg_notify(2.17e(17) epoch 11705) -7863> 2014-09-18 18:08:26.681664 7f4c9f251700 5 -- op tracker -- , seq: 85733, time: 2014-09-18 18:08:26.681597, event: all_read, op: pg_notify(2.17e(17) epoch 11705) -7862> 2014-09-18 18:08:26.681673 7f4c9f251700 5 -- op tracker -- , seq: 85733, time: 2014-09-18 18:08:26.681639, event: dispatched, op: pg_notify(2.17e(17) epoch 11705) -7861> 2014-09-18 18:08:26.681686 7f4c9f251700 5 -- op tracker -- , seq: 85733, time: 2014-09-18 18:08:26.681685, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11705) -7860> 2014-09-18 18:08:26.681695 7f4c9f251700 5 -- op tracker -- , seq: 85733, time: 2014-09-18 18:08:26.681694, event: started, op: pg_notify(2.17e(17) epoch 11705) -7859> 2014-09-18 18:08:26.681717 7f4c9f251700 5 -- op tracker -- , seq: 85733, time: 2014-09-18 18:08:26.681715, event: done, op: pg_notify(2.17e(17) epoch 11705) -7858> 2014-09-18 18:08:26.682271 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1715 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x10e22240 con 0x4900000 -7857> 2014-09-18 18:08:26.682297 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7856> 2014-09-18 18:08:26.682322 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1716 ==== pg_notify(2.e0(79) epoch 11705) v5 ==== 5119+0+0 (1786434908 0 0) 0xf1edc00 con 0x4900000 -7855> 2014-09-18 18:08:26.682338 7f4c9f251700 5 -- op tracker -- , seq: 85734, time: 2014-09-18 18:08:26.682125, event: header_read, op: pg_notify(2.e0(79) epoch 11705) -7854> 2014-09-18 18:08:26.682349 7f4c9f251700 5 -- op tracker -- , seq: 85734, time: 2014-09-18 18:08:26.682126, event: throttled, op: pg_notify(2.e0(79) epoch 11705) -7853> 2014-09-18 18:08:26.682356 7f4c9f251700 5 -- op tracker -- , seq: 85734, time: 2014-09-18 18:08:26.682249, event: all_read, op: pg_notify(2.e0(79) epoch 11705) -7852> 2014-09-18 18:08:26.682363 7f4c9f251700 5 -- op tracker -- , seq: 85734, time: 2014-09-18 18:08:26.682332, event: dispatched, op: pg_notify(2.e0(79) epoch 11705) -7851> 2014-09-18 18:08:26.682370 7f4c9f251700 5 -- op tracker -- , seq: 85734, time: 2014-09-18 18:08:26.682368, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11705) -7850> 2014-09-18 18:08:26.682380 7f4c9f251700 5 -- op tracker -- , seq: 85734, time: 2014-09-18 18:08:26.682377, event: started, op: pg_notify(2.e0(79) epoch 11705) -7849> 2014-09-18 18:08:26.682405 7f4c9f251700 5 -- op tracker -- , seq: 85734, time: 2014-09-18 18:08:26.682402, event: done, op: pg_notify(2.e0(79) epoch 11705) -7848> 2014-09-18 18:08:26.683080 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2003 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x1182c880 con 0xe2d9a20 -7847> 2014-09-18 18:08:26.683100 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7846> 2014-09-18 18:08:26.683149 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2004 ==== pg_notify(2.26d(9) epoch 11705) v5 ==== 1257+0+0 (1508794142 0 0) 0xe1a2780 con 0xe2d9a20 -7845> 2014-09-18 18:08:26.683173 7f4c9f251700 5 -- op tracker -- , seq: 85735, time: 2014-09-18 18:08:26.683086, event: header_read, op: pg_notify(2.26d(9) epoch 11705) -7844> 2014-09-18 18:08:26.683203 7f4c9f251700 5 -- op tracker -- , seq: 85735, time: 2014-09-18 18:08:26.683087, event: throttled, op: pg_notify(2.26d(9) epoch 11705) -7843> 2014-09-18 18:08:26.683218 7f4c9f251700 5 -- op tracker -- , seq: 85735, time: 2014-09-18 18:08:26.683134, event: all_read, op: pg_notify(2.26d(9) epoch 11705) -7842> 2014-09-18 18:08:26.683237 7f4c9f251700 5 -- op tracker -- , seq: 85735, time: 2014-09-18 18:08:26.683166, event: dispatched, op: pg_notify(2.26d(9) epoch 11705) -7841> 2014-09-18 18:08:26.683245 7f4c9f251700 5 -- op tracker -- , seq: 85735, time: 2014-09-18 18:08:26.683243, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11705) -7840> 2014-09-18 18:08:26.683256 7f4c9f251700 5 -- op tracker -- , seq: 85735, time: 2014-09-18 18:08:26.683253, event: started, op: pg_notify(2.26d(9) epoch 11705) -7839> 2014-09-18 18:08:26.683461 7f4c9f251700 5 -- op tracker -- , seq: 85735, time: 2014-09-18 18:08:26.683457, event: done, op: pg_notify(2.26d(9) epoch 11705) -7838> 2014-09-18 18:08:26.685469 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3604 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x12bf1680 con 0xcbecec0 -7837> 2014-09-18 18:08:26.685492 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7836> 2014-09-18 18:08:26.685506 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 915 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x2db46f40 con 0x1395f8c0 -7835> 2014-09-18 18:08:26.685520 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7834> 2014-09-18 18:08:26.685529 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3605 ==== pg_notify(2.4f6(12) epoch 11705) v5 ==== 1382+0+0 (3549621777 0 0) 0x2b94c3a0 con 0xcbecec0 -7833> 2014-09-18 18:08:26.685542 7f4c9f251700 5 -- op tracker -- , seq: 85736, time: 2014-09-18 18:08:26.685465, event: header_read, op: pg_notify(2.4f6(12) epoch 11705) -7832> 2014-09-18 18:08:26.685578 7f4c9f251700 5 -- op tracker -- , seq: 85736, time: 2014-09-18 18:08:26.685466, event: throttled, op: pg_notify(2.4f6(12) epoch 11705) -7831> 2014-09-18 18:08:26.685591 7f4c9f251700 5 -- op tracker -- , seq: 85736, time: 2014-09-18 18:08:26.685494, event: all_read, op: pg_notify(2.4f6(12) epoch 11705) -7830> 2014-09-18 18:08:26.685598 7f4c9f251700 5 -- op tracker -- , seq: 85736, time: 2014-09-18 18:08:26.685538, event: dispatched, op: pg_notify(2.4f6(12) epoch 11705) -7829> 2014-09-18 18:08:26.685604 7f4c9f251700 5 -- op tracker -- , seq: 85736, time: 2014-09-18 18:08:26.685603, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11705) -7828> 2014-09-18 18:08:26.685626 7f4c9f251700 5 -- op tracker -- , seq: 85736, time: 2014-09-18 18:08:26.685623, event: started, op: pg_notify(2.4f6(12) epoch 11705) -7827> 2014-09-18 18:08:26.685662 7f4c9f251700 5 -- op tracker -- , seq: 85736, time: 2014-09-18 18:08:26.685659, event: done, op: pg_notify(2.4f6(12) epoch 11705) -7826> 2014-09-18 18:08:26.685677 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 916 ==== pg_notify(2.410(11) epoch 11705) v5 ==== 1287+0+0 (2379751464 0 0) 0xe9565a0 con 0x1395f8c0 -7825> 2014-09-18 18:08:26.685690 7f4c9f251700 5 -- op tracker -- , seq: 85737, time: 2014-09-18 18:08:26.685505, event: header_read, op: pg_notify(2.410(11) epoch 11705) -7824> 2014-09-18 18:08:26.685701 7f4c9f251700 5 -- op tracker -- , seq: 85737, time: 2014-09-18 18:08:26.685505, event: throttled, op: pg_notify(2.410(11) epoch 11705) -7823> 2014-09-18 18:08:26.685710 7f4c9f251700 5 -- op tracker -- , seq: 85737, time: 2014-09-18 18:08:26.685543, event: all_read, op: pg_notify(2.410(11) epoch 11705) -7822> 2014-09-18 18:08:26.685720 7f4c9f251700 5 -- op tracker -- , seq: 85737, time: 2014-09-18 18:08:26.685686, event: dispatched, op: pg_notify(2.410(11) epoch 11705) -7821> 2014-09-18 18:08:26.685730 7f4c9f251700 5 -- op tracker -- , seq: 85737, time: 2014-09-18 18:08:26.685727, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11705) -7820> 2014-09-18 18:08:26.685741 7f4c9f251700 5 -- op tracker -- , seq: 85737, time: 2014-09-18 18:08:26.685738, event: started, op: pg_notify(2.410(11) epoch 11705) -7819> 2014-09-18 18:08:26.685790 7f4c9f251700 5 -- op tracker -- , seq: 85737, time: 2014-09-18 18:08:26.685787, event: done, op: pg_notify(2.410(11) epoch 11705) -7818> 2014-09-18 18:08:26.685807 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2340 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0xe267440 con 0xdd8c940 -7817> 2014-09-18 18:08:26.685872 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7816> 2014-09-18 18:08:26.685880 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2341 ==== pg_notify(2.457(21) epoch 11705) v5 ==== 1875+0+0 (2184951058 0 0) 0xf2b9660 con 0xdd8c940 -7815> 2014-09-18 18:08:26.685899 7f4c9f251700 5 -- op tracker -- , seq: 85738, time: 2014-09-18 18:08:26.685562, event: header_read, op: pg_notify(2.457(21) epoch 11705) -7814> 2014-09-18 18:08:26.685907 7f4c9f251700 5 -- op tracker -- , seq: 85738, time: 2014-09-18 18:08:26.685563, event: throttled, op: pg_notify(2.457(21) epoch 11705) -7813> 2014-09-18 18:08:26.685914 7f4c9f251700 5 -- op tracker -- , seq: 85738, time: 2014-09-18 18:08:26.685620, event: all_read, op: pg_notify(2.457(21) epoch 11705) -7812> 2014-09-18 18:08:26.685918 7f4c9f251700 5 -- op tracker -- , seq: 85738, time: 2014-09-18 18:08:26.685887, event: dispatched, op: pg_notify(2.457(21) epoch 11705) -7811> 2014-09-18 18:08:26.685924 7f4c9f251700 5 -- op tracker -- , seq: 85738, time: 2014-09-18 18:08:26.685923, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11705) -7810> 2014-09-18 18:08:26.685930 7f4c9f251700 5 -- op tracker -- , seq: 85738, time: 2014-09-18 18:08:26.685928, event: started, op: pg_notify(2.457(21) epoch 11705) -7809> 2014-09-18 18:08:26.685947 7f4c9f251700 5 -- op tracker -- , seq: 85738, time: 2014-09-18 18:08:26.685945, event: done, op: pg_notify(2.457(21) epoch 11705) -7808> 2014-09-18 18:08:26.686702 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2342 ==== pg_notify(2.2c7(139) epoch 11705) v5 ==== 8651+0+0 (1538902679 0 0) 0xf2b72c0 con 0xdd8c940 -7807> 2014-09-18 18:08:26.686719 7f4c9f251700 5 -- op tracker -- , seq: 85739, time: 2014-09-18 18:08:26.686436, event: header_read, op: pg_notify(2.2c7(139) epoch 11705) -7806> 2014-09-18 18:08:26.686729 7f4c9f251700 5 -- op tracker -- , seq: 85739, time: 2014-09-18 18:08:26.686437, event: throttled, op: pg_notify(2.2c7(139) epoch 11705) -7805> 2014-09-18 18:08:26.686735 7f4c9f251700 5 -- op tracker -- , seq: 85739, time: 2014-09-18 18:08:26.686684, event: all_read, op: pg_notify(2.2c7(139) epoch 11705) -7804> 2014-09-18 18:08:26.686741 7f4c9f251700 5 -- op tracker -- , seq: 85739, time: 2014-09-18 18:08:26.686715, event: dispatched, op: pg_notify(2.2c7(139) epoch 11705) -7803> 2014-09-18 18:08:26.686747 7f4c9f251700 5 -- op tracker -- , seq: 85739, time: 2014-09-18 18:08:26.686746, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11705) -7802> 2014-09-18 18:08:26.686755 7f4c9f251700 5 -- op tracker -- , seq: 85739, time: 2014-09-18 18:08:26.686754, event: started, op: pg_notify(2.2c7(139) epoch 11705) -7801> 2014-09-18 18:08:26.686775 7f4c9f251700 5 -- op tracker -- , seq: 85739, time: 2014-09-18 18:08:26.686773, event: done, op: pg_notify(2.2c7(139) epoch 11705) -7800> 2014-09-18 18:08:26.686991 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1188 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x1182afc0 con 0x1088b440 -7799> 2014-09-18 18:08:26.687007 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7798> 2014-09-18 18:08:26.687332 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1189 ==== pg_notify(2.2d8(173) epoch 11705) v5 ==== 10757+0+0 (1609794061 0 0) 0x100f1660 con 0x1088b440 -7797> 2014-09-18 18:08:26.687355 7f4c9f251700 5 -- op tracker -- , seq: 85740, time: 2014-09-18 18:08:26.687156, event: header_read, op: pg_notify(2.2d8(173) epoch 11705) -7796> 2014-09-18 18:08:26.687364 7f4c9f251700 5 -- op tracker -- , seq: 85740, time: 2014-09-18 18:08:26.687157, event: throttled, op: pg_notify(2.2d8(173) epoch 11705) -7795> 2014-09-18 18:08:26.687370 7f4c9f251700 5 -- op tracker -- , seq: 85740, time: 2014-09-18 18:08:26.687319, event: all_read, op: pg_notify(2.2d8(173) epoch 11705) -7794> 2014-09-18 18:08:26.687376 7f4c9f251700 5 -- op tracker -- , seq: 85740, time: 2014-09-18 18:08:26.687351, event: dispatched, op: pg_notify(2.2d8(173) epoch 11705) -7793> 2014-09-18 18:08:26.687382 7f4c9f251700 5 -- op tracker -- , seq: 85740, time: 2014-09-18 18:08:26.687381, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11705) -7792> 2014-09-18 18:08:26.687390 7f4c9f251700 5 -- op tracker -- , seq: 85740, time: 2014-09-18 18:08:26.687388, event: started, op: pg_notify(2.2d8(173) epoch 11705) -7791> 2014-09-18 18:08:26.687408 7f4c9f251700 5 -- op tracker -- , seq: 85740, time: 2014-09-18 18:08:26.687406, event: done, op: pg_notify(2.2d8(173) epoch 11705) -7790> 2014-09-18 18:08:26.689219 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3606 ==== pg_notify(2.3b6(10) epoch 11705) v5 ==== 1260+0+0 (885019659 0 0) 0x1390dc00 con 0xcbecec0 -7789> 2014-09-18 18:08:26.689234 7f4c9f251700 5 -- op tracker -- , seq: 85741, time: 2014-09-18 18:08:26.689171, event: header_read, op: pg_notify(2.3b6(10) epoch 11705) -7788> 2014-09-18 18:08:26.689242 7f4c9f251700 5 -- op tracker -- , seq: 85741, time: 2014-09-18 18:08:26.689172, event: throttled, op: pg_notify(2.3b6(10) epoch 11705) -7787> 2014-09-18 18:08:26.689250 7f4c9f251700 5 -- op tracker -- , seq: 85741, time: 2014-09-18 18:08:26.689206, event: all_read, op: pg_notify(2.3b6(10) epoch 11705) -7786> 2014-09-18 18:08:26.689259 7f4c9f251700 5 -- op tracker -- , seq: 85741, time: 2014-09-18 18:08:26.689230, event: dispatched, op: pg_notify(2.3b6(10) epoch 11705) -7785> 2014-09-18 18:08:26.689270 7f4c9f251700 5 -- op tracker -- , seq: 85741, time: 2014-09-18 18:08:26.689267, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11705) -7784> 2014-09-18 18:08:26.689281 7f4c9f251700 5 -- op tracker -- , seq: 85741, time: 2014-09-18 18:08:26.689279, event: started, op: pg_notify(2.3b6(10) epoch 11705) -7783> 2014-09-18 18:08:26.689307 7f4c9f251700 5 -- op tracker -- , seq: 85741, time: 2014-09-18 18:08:26.689305, event: done, op: pg_notify(2.3b6(10) epoch 11705) -7782> 2014-09-18 18:08:26.689395 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3278 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x2fba0640 con 0x4900580 -7781> 2014-09-18 18:08:26.689408 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7780> 2014-09-18 18:08:26.689778 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3279 ==== pg_notify(2.4f6(184) epoch 11705) v5 ==== 11296+0+0 (1700894540 0 0) 0x19f24b20 con 0x4900580 -7779> 2014-09-18 18:08:26.689795 7f4c9f251700 5 -- op tracker -- , seq: 85742, time: 2014-09-18 18:08:26.689564, event: header_read, op: pg_notify(2.4f6(184) epoch 11705) -7778> 2014-09-18 18:08:26.689807 7f4c9f251700 5 -- op tracker -- , seq: 85742, time: 2014-09-18 18:08:26.689565, event: throttled, op: pg_notify(2.4f6(184) epoch 11705) -7777> 2014-09-18 18:08:26.689817 7f4c9f251700 5 -- op tracker -- , seq: 85742, time: 2014-09-18 18:08:26.689753, event: all_read, op: pg_notify(2.4f6(184) epoch 11705) -7776> 2014-09-18 18:08:26.689825 7f4c9f251700 5 -- op tracker -- , seq: 85742, time: 2014-09-18 18:08:26.689791, event: dispatched, op: pg_notify(2.4f6(184) epoch 11705) -7775> 2014-09-18 18:08:26.689835 7f4c9f251700 5 -- op tracker -- , seq: 85742, time: 2014-09-18 18:08:26.689833, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11705) -7774> 2014-09-18 18:08:26.689846 7f4c9f251700 5 -- op tracker -- , seq: 85742, time: 2014-09-18 18:08:26.689844, event: started, op: pg_notify(2.4f6(184) epoch 11705) -7773> 2014-09-18 18:08:26.689882 7f4c9f251700 5 -- op tracker -- , seq: 85742, time: 2014-09-18 18:08:26.689879, event: done, op: pg_notify(2.4f6(184) epoch 11705) -7772> 2014-09-18 18:08:26.689924 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3280 ==== pg_notify(2.17e(17) epoch 11705) v5 ==== 1593+0+0 (3381414688 0 0) 0x19f23e00 con 0x4900580 -7771> 2014-09-18 18:08:26.689941 7f4c9f251700 5 -- op tracker -- , seq: 85743, time: 2014-09-18 18:08:26.689780, event: header_read, op: pg_notify(2.17e(17) epoch 11705) -7770> 2014-09-18 18:08:26.689953 7f4c9f251700 5 -- op tracker -- , seq: 85743, time: 2014-09-18 18:08:26.689780, event: throttled, op: pg_notify(2.17e(17) epoch 11705) -7769> 2014-09-18 18:08:26.689961 7f4c9f251700 5 -- op tracker -- , seq: 85743, time: 2014-09-18 18:08:26.689836, event: all_read, op: pg_notify(2.17e(17) epoch 11705) -7768> 2014-09-18 18:08:26.689969 7f4c9f251700 5 -- op tracker -- , seq: 85743, time: 2014-09-18 18:08:26.689937, event: dispatched, op: pg_notify(2.17e(17) epoch 11705) -7767> 2014-09-18 18:08:26.689979 7f4c9f251700 5 -- op tracker -- , seq: 85743, time: 2014-09-18 18:08:26.689977, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11705) -7766> 2014-09-18 18:08:26.689990 7f4c9f251700 5 -- op tracker -- , seq: 85743, time: 2014-09-18 18:08:26.689987, event: started, op: pg_notify(2.17e(17) epoch 11705) -7765> 2014-09-18 18:08:26.690013 7f4c9f251700 5 -- op tracker -- , seq: 85743, time: 2014-09-18 18:08:26.690011, event: done, op: pg_notify(2.17e(17) epoch 11705) -7764> 2014-09-18 18:08:26.690424 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3281 ==== pg_notify(2.1bb(9) epoch 11705) v5 ==== 1189+0+0 (2751460153 0 0) 0x19f22d20 con 0x4900580 -7763> 2014-09-18 18:08:26.690439 7f4c9f251700 5 -- op tracker -- , seq: 85744, time: 2014-09-18 18:08:26.690360, event: header_read, op: pg_notify(2.1bb(9) epoch 11705) -7762> 2014-09-18 18:08:26.690447 7f4c9f251700 5 -- op tracker -- , seq: 85744, time: 2014-09-18 18:08:26.690361, event: throttled, op: pg_notify(2.1bb(9) epoch 11705) -7761> 2014-09-18 18:08:26.690453 7f4c9f251700 5 -- op tracker -- , seq: 85744, time: 2014-09-18 18:08:26.690410, event: all_read, op: pg_notify(2.1bb(9) epoch 11705) -7760> 2014-09-18 18:08:26.690459 7f4c9f251700 5 -- op tracker -- , seq: 85744, time: 2014-09-18 18:08:26.690435, event: dispatched, op: pg_notify(2.1bb(9) epoch 11705) -7759> 2014-09-18 18:08:26.690466 7f4c9f251700 5 -- op tracker -- , seq: 85744, time: 2014-09-18 18:08:26.690464, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11705) -7758> 2014-09-18 18:08:26.690473 7f4c9f251700 5 -- op tracker -- , seq: 85744, time: 2014-09-18 18:08:26.690472, event: started, op: pg_notify(2.1bb(9) epoch 11705) -7757> 2014-09-18 18:08:26.690491 7f4c9f251700 5 -- op tracker -- , seq: 85744, time: 2014-09-18 18:08:26.690489, event: done, op: pg_notify(2.1bb(9) epoch 11705) -7756> 2014-09-18 18:08:26.691168 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3607 ==== pg_notify(2.517(11) epoch 11705) v5 ==== 1343+0+0 (1355948759 0 0) 0x1390be00 con 0xcbecec0 -7755> 2014-09-18 18:08:26.691181 7f4c9f251700 5 -- op tracker -- , seq: 85745, time: 2014-09-18 18:08:26.691116, event: header_read, op: pg_notify(2.517(11) epoch 11705) -7754> 2014-09-18 18:08:26.691189 7f4c9f251700 5 -- op tracker -- , seq: 85745, time: 2014-09-18 18:08:26.691117, event: throttled, op: pg_notify(2.517(11) epoch 11705) -7753> 2014-09-18 18:08:26.691196 7f4c9f251700 5 -- op tracker -- , seq: 85745, time: 2014-09-18 18:08:26.691155, event: all_read, op: pg_notify(2.517(11) epoch 11705) -7752> 2014-09-18 18:08:26.691202 7f4c9f251700 5 -- op tracker -- , seq: 85745, time: 2014-09-18 18:08:26.691178, event: dispatched, op: pg_notify(2.517(11) epoch 11705) -7751> 2014-09-18 18:08:26.691207 7f4c9f251700 5 -- op tracker -- , seq: 85745, time: 2014-09-18 18:08:26.691206, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11705) -7750> 2014-09-18 18:08:26.691215 7f4c9f251700 5 -- op tracker -- , seq: 85745, time: 2014-09-18 18:08:26.691213, event: started, op: pg_notify(2.517(11) epoch 11705) -7749> 2014-09-18 18:08:26.691251 7f4c9f251700 5 -- op tracker -- , seq: 85745, time: 2014-09-18 18:08:26.691249, event: done, op: pg_notify(2.517(11) epoch 11705) -7748> 2014-09-18 18:08:26.692391 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3608 ==== pg_notify(2.2c7(139) epoch 11705) v5 ==== 8651+0+0 (2584622460 0 0) 0x12ba3840 con 0xcbecec0 -7747> 2014-09-18 18:08:26.692404 7f4c9f251700 5 -- op tracker -- , seq: 85746, time: 2014-09-18 18:08:26.692222, event: header_read, op: pg_notify(2.2c7(139) epoch 11705) -7746> 2014-09-18 18:08:26.692412 7f4c9f251700 5 -- op tracker -- , seq: 85746, time: 2014-09-18 18:08:26.692223, event: throttled, op: pg_notify(2.2c7(139) epoch 11705) -7745> 2014-09-18 18:08:26.692419 7f4c9f251700 5 -- op tracker -- , seq: 85746, time: 2014-09-18 18:08:26.692380, event: all_read, op: pg_notify(2.2c7(139) epoch 11705) -7744> 2014-09-18 18:08:26.692423 7f4c9f251700 5 -- op tracker -- , seq: 85746, time: 2014-09-18 18:08:26.692401, event: dispatched, op: pg_notify(2.2c7(139) epoch 11705) -7743> 2014-09-18 18:08:26.692429 7f4c9f251700 5 -- op tracker -- , seq: 85746, time: 2014-09-18 18:08:26.692428, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11705) -7742> 2014-09-18 18:08:26.692437 7f4c9f251700 5 -- op tracker -- , seq: 85746, time: 2014-09-18 18:08:26.692435, event: started, op: pg_notify(2.2c7(139) epoch 11705) -7741> 2014-09-18 18:08:26.692453 7f4c9f251700 5 -- op tracker -- , seq: 85746, time: 2014-09-18 18:08:26.692452, event: done, op: pg_notify(2.2c7(139) epoch 11705) -7740> 2014-09-18 18:08:26.694521 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3609 ==== pg_notify(2.40a(162) epoch 11705) v5 ==== 9736+0+0 (1146056640 0 0) 0xf8734a0 con 0xcbecec0 -7739> 2014-09-18 18:08:26.694553 7f4c9f251700 5 -- op tracker -- , seq: 85747, time: 2014-09-18 18:08:26.694393, event: header_read, op: pg_notify(2.40a(162) epoch 11705) -7738> 2014-09-18 18:08:26.694566 7f4c9f251700 5 -- op tracker -- , seq: 85747, time: 2014-09-18 18:08:26.694394, event: throttled, op: pg_notify(2.40a(162) epoch 11705) -7737> 2014-09-18 18:08:26.694573 7f4c9f251700 5 -- op tracker -- , seq: 85747, time: 2014-09-18 18:08:26.694503, event: all_read, op: pg_notify(2.40a(162) epoch 11705) -7736> 2014-09-18 18:08:26.694578 7f4c9f251700 5 -- op tracker -- , seq: 85747, time: 2014-09-18 18:08:26.694537, event: dispatched, op: pg_notify(2.40a(162) epoch 11705) -7735> 2014-09-18 18:08:26.694585 7f4c9f251700 5 -- op tracker -- , seq: 85747, time: 2014-09-18 18:08:26.694583, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11705) -7734> 2014-09-18 18:08:26.694593 7f4c9f251700 5 -- op tracker -- , seq: 85747, time: 2014-09-18 18:08:26.694591, event: started, op: pg_notify(2.40a(162) epoch 11705) -7733> 2014-09-18 18:08:26.694610 7f4c9f251700 5 -- op tracker -- , seq: 85747, time: 2014-09-18 18:08:26.694609, event: done, op: pg_notify(2.40a(162) epoch 11705) -7732> 2014-09-18 18:08:26.695133 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 34 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x12bf2400 con 0x1395f340 -7731> 2014-09-18 18:08:26.695152 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7730> 2014-09-18 18:08:26.695178 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 35 ==== pg_notify(2.1d8(9) epoch 11705) v5 ==== 1261+0+0 (3551314781 0 0) 0x100f1480 con 0x1395f340 -7729> 2014-09-18 18:08:26.695192 7f4c9f251700 5 -- op tracker -- , seq: 85748, time: 2014-09-18 18:08:26.695131, event: header_read, op: pg_notify(2.1d8(9) epoch 11705) -7728> 2014-09-18 18:08:26.695203 7f4c9f251700 5 -- op tracker -- , seq: 85748, time: 2014-09-18 18:08:26.695131, event: throttled, op: pg_notify(2.1d8(9) epoch 11705) -7727> 2014-09-18 18:08:26.695211 7f4c9f251700 5 -- op tracker -- , seq: 85748, time: 2014-09-18 18:08:26.695165, event: all_read, op: pg_notify(2.1d8(9) epoch 11705) -7726> 2014-09-18 18:08:26.695219 7f4c9f251700 5 -- op tracker -- , seq: 85748, time: 2014-09-18 18:08:26.695188, event: dispatched, op: pg_notify(2.1d8(9) epoch 11705) -7725> 2014-09-18 18:08:26.695229 7f4c9f251700 5 -- op tracker -- , seq: 85748, time: 2014-09-18 18:08:26.695226, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11705) -7724> 2014-09-18 18:08:26.695241 7f4c9f251700 5 -- op tracker -- , seq: 85748, time: 2014-09-18 18:08:26.695239, event: started, op: pg_notify(2.1d8(9) epoch 11705) -7723> 2014-09-18 18:08:26.695267 7f4c9f251700 5 -- op tracker -- , seq: 85748, time: 2014-09-18 18:08:26.695265, event: done, op: pg_notify(2.1d8(9) epoch 11705) -7722> 2014-09-18 18:08:26.695678 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3610 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11705) v5 ==== 6174+0+0 (842021316 0 0) 0xf874760 con 0xcbecec0 -7721> 2014-09-18 18:08:26.695695 7f4c9f251700 5 -- op tracker -- , seq: 85749, time: 2014-09-18 18:08:26.695593, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11705) -7720> 2014-09-18 18:08:26.695704 7f4c9f251700 5 -- op tracker -- , seq: 85749, time: 2014-09-18 18:08:26.695594, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11705) -7719> 2014-09-18 18:08:26.695711 7f4c9f251700 5 -- op tracker -- , seq: 85749, time: 2014-09-18 18:08:26.695664, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11705) -7718> 2014-09-18 18:08:26.695718 7f4c9f251700 5 -- op tracker -- , seq: 85749, time: 2014-09-18 18:08:26.695691, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11705) -7717> 2014-09-18 18:08:26.695725 7f4c9f251700 5 -- op tracker -- , seq: 85749, time: 2014-09-18 18:08:26.695723, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11705) -7716> 2014-09-18 18:08:26.695734 7f4c9f251700 5 -- op tracker -- , seq: 85749, time: 2014-09-18 18:08:26.695732, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11705) -7715> 2014-09-18 18:08:26.695771 7f4c9f251700 5 -- op tracker -- , seq: 85749, time: 2014-09-18 18:08:26.695769, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11705) -7714> 2014-09-18 18:08:26.696694 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3611 ==== pg_notify(2.1bb(9) epoch 11705) v5 ==== 1189+0+0 (259408170 0 0) 0x2b940760 con 0xcbecec0 -7713> 2014-09-18 18:08:26.696713 7f4c9f251700 5 -- op tracker -- , seq: 85750, time: 2014-09-18 18:08:26.696640, event: header_read, op: pg_notify(2.1bb(9) epoch 11705) -7712> 2014-09-18 18:08:26.696725 7f4c9f251700 5 -- op tracker -- , seq: 85750, time: 2014-09-18 18:08:26.696640, event: throttled, op: pg_notify(2.1bb(9) epoch 11705) -7711> 2014-09-18 18:08:26.696735 7f4c9f251700 5 -- op tracker -- , seq: 85750, time: 2014-09-18 18:08:26.696666, event: all_read, op: pg_notify(2.1bb(9) epoch 11705) -7710> 2014-09-18 18:08:26.696745 7f4c9f251700 5 -- op tracker -- , seq: 85750, time: 2014-09-18 18:08:26.696708, event: dispatched, op: pg_notify(2.1bb(9) epoch 11705) -7709> 2014-09-18 18:08:26.696756 7f4c9f251700 5 -- op tracker -- , seq: 85750, time: 2014-09-18 18:08:26.696753, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11705) -7708> 2014-09-18 18:08:26.696768 7f4c9f251700 5 -- op tracker -- , seq: 85750, time: 2014-09-18 18:08:26.696765, event: started, op: pg_notify(2.1bb(9) epoch 11705) -7707> 2014-09-18 18:08:26.696794 7f4c9f251700 5 -- op tracker -- , seq: 85750, time: 2014-09-18 18:08:26.696791, event: done, op: pg_notify(2.1bb(9) epoch 11705) -7706> 2014-09-18 18:08:26.702091 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.51 10.242.42.168:0/15056 148 ==== osd_ping(ping e11705 stamp 2014-09-18 18:08:26.569161) v2 ==== 47+0+0 (703610207 0 0) 0x10507680 con 0xcc76f20 -7705> 2014-09-18 18:08:26.702092 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.51 10.242.42.168:0/15056 148 ==== osd_ping(ping e11705 stamp 2014-09-18 18:08:26.569161) v2 ==== 47+0+0 (703610207 0 0) 0xf789c00 con 0x17de92e0 -7704> 2014-09-18 18:08:26.702122 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/15056 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:26.569161) v2 -- ?+0 0x12115840 con 0xcc76f20 -7703> 2014-09-18 18:08:26.702157 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/15056 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:26.569161) v2 -- ?+0 0x1393d480 con 0x17de92e0 -7702> 2014-09-18 18:08:26.716382 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1270 ==== osd_map(11704..11704 src has 1..11704) v3 ==== 352+0+0 (1848137236 0 0) 0xeee8240 con 0x1f102840 -7701> 2014-09-18 18:08:26.716402 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11704,11704], i have 11705, src has [1,11704] -7700> 2014-09-18 18:08:26.716408 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1271 ==== pg_notify(2.4f6(15) epoch 11704) v5 ==== 1543+0+0 (1891807793 0 0) 0xf488ee0 con 0x1f102840 -7699> 2014-09-18 18:08:26.716416 7f4c9f251700 5 -- op tracker -- , seq: 85751, time: 2014-09-18 18:08:26.716307, event: header_read, op: pg_notify(2.4f6(15) epoch 11704) -7698> 2014-09-18 18:08:26.716427 7f4c9f251700 5 -- op tracker -- , seq: 85751, time: 2014-09-18 18:08:26.716307, event: throttled, op: pg_notify(2.4f6(15) epoch 11704) -7697> 2014-09-18 18:08:26.716438 7f4c9f251700 5 -- op tracker -- , seq: 85751, time: 2014-09-18 18:08:26.716336, event: all_read, op: pg_notify(2.4f6(15) epoch 11704) -7696> 2014-09-18 18:08:26.716446 7f4c9f251700 5 -- op tracker -- , seq: 85751, time: 2014-09-18 18:08:26.716413, event: dispatched, op: pg_notify(2.4f6(15) epoch 11704) -7695> 2014-09-18 18:08:26.716454 7f4c9f251700 5 -- op tracker -- , seq: 85751, time: 2014-09-18 18:08:26.716453, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11704) -7694> 2014-09-18 18:08:26.716463 7f4c9f251700 5 -- op tracker -- , seq: 85751, time: 2014-09-18 18:08:26.716461, event: started, op: pg_notify(2.4f6(15) epoch 11704) -7693> 2014-09-18 18:08:26.716481 7f4c9f251700 5 -- op tracker -- , seq: 85751, time: 2014-09-18 18:08:26.716479, event: done, op: pg_notify(2.4f6(15) epoch 11704) -7692> 2014-09-18 18:08:26.734004 7f4caae41700 5 osd.42 11705 tick -7691> 2014-09-18 18:08:26.742328 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1272 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0xeee8000 con 0x1f102840 -7690> 2014-09-18 18:08:26.742347 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7689> 2014-09-18 18:08:26.742385 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1273 ==== pg_notify(2.4f6(15) epoch 11705) v5 ==== 1543+0+0 (682368455 0 0) 0xf0974a0 con 0x1f102840 -7688> 2014-09-18 18:08:26.742399 7f4c9f251700 5 -- op tracker -- , seq: 85752, time: 2014-09-18 18:08:26.742310, event: header_read, op: pg_notify(2.4f6(15) epoch 11705) -7687> 2014-09-18 18:08:26.742413 7f4c9f251700 5 -- op tracker -- , seq: 85752, time: 2014-09-18 18:08:26.742311, event: throttled, op: pg_notify(2.4f6(15) epoch 11705) -7686> 2014-09-18 18:08:26.742422 7f4c9f251700 5 -- op tracker -- , seq: 85752, time: 2014-09-18 18:08:26.742374, event: all_read, op: pg_notify(2.4f6(15) epoch 11705) -7685> 2014-09-18 18:08:26.742430 7f4c9f251700 5 -- op tracker -- , seq: 85752, time: 2014-09-18 18:08:26.742395, event: dispatched, op: pg_notify(2.4f6(15) epoch 11705) -7684> 2014-09-18 18:08:26.742445 7f4c9f251700 5 -- op tracker -- , seq: 85752, time: 2014-09-18 18:08:26.742443, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11705) -7683> 2014-09-18 18:08:26.742453 7f4c9f251700 5 -- op tracker -- , seq: 85752, time: 2014-09-18 18:08:26.742451, event: started, op: pg_notify(2.4f6(15) epoch 11705) -7682> 2014-09-18 18:08:26.742470 7f4c9f251700 5 -- op tracker -- , seq: 85752, time: 2014-09-18 18:08:26.742468, event: done, op: pg_notify(2.4f6(15) epoch 11705) -7681> 2014-09-18 18:08:26.746730 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.39 10.242.42.172:0/2660 1587 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:15.851121) v2 ==== 47+0+0 (2925056237 0 0) 0x11b0ef00 con 0xec39340 -7680> 2014-09-18 18:08:26.746753 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:15.851121) v2 -- ?+0 0xf789c00 con 0xec39340 -7679> 2014-09-18 18:08:26.747334 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.39 10.242.42.172:0/2660 1587 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:15.851121) v2 ==== 47+0+0 (2925056237 0 0) 0x21e12b20 con 0x4750840 -7678> 2014-09-18 18:08:26.747361 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:15.851121) v2 -- ?+0 0x10507680 con 0x4750840 -7677> 2014-09-18 18:08:26.785386 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.48 10.242.42.164:6836/27583 12 ==== osd_map(11703..11705 src has 1..11705) v3 ==== 1629+0+0 (3739120107 0 0) 0xe937200 con 0x10aebe40 -7676> 2014-09-18 18:08:26.785420 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11703,11705], i have 11705, src has [1,11705] -7675> 2014-09-18 18:08:26.785434 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.48 10.242.42.164:6836/27583 13 ==== pg_info(1 pgs e11702:2.4c1) v4 ==== 718+0+0 (3528625165 0 0) 0xf15fc20 con 0x10aebe40 -7674> 2014-09-18 18:08:26.785445 7f4c9f251700 5 -- op tracker -- , seq: 85753, time: 2014-09-18 18:08:26.785368, event: header_read, op: pg_info(1 pgs e11702:2.4c1) -7673> 2014-09-18 18:08:26.785456 7f4c9f251700 5 -- op tracker -- , seq: 85753, time: 2014-09-18 18:08:26.785368, event: throttled, op: pg_info(1 pgs e11702:2.4c1) -7672> 2014-09-18 18:08:26.785464 7f4c9f251700 5 -- op tracker -- , seq: 85753, time: 2014-09-18 18:08:26.785407, event: all_read, op: pg_info(1 pgs e11702:2.4c1) -7671> 2014-09-18 18:08:26.785471 7f4c9f251700 5 -- op tracker -- , seq: 85753, time: 2014-09-18 18:08:26.785440, event: dispatched, op: pg_info(1 pgs e11702:2.4c1) -7670> 2014-09-18 18:08:26.785479 7f4c9f251700 5 -- op tracker -- , seq: 85753, time: 2014-09-18 18:08:26.785478, event: waiting_for_osdmap, op: pg_info(1 pgs e11702:2.4c1) -7669> 2014-09-18 18:08:26.785489 7f4c9f251700 5 -- op tracker -- , seq: 85753, time: 2014-09-18 18:08:26.785488, event: started, op: pg_info(1 pgs e11702:2.4c1) -7668> 2014-09-18 18:08:26.785513 7f4c9f251700 5 -- op tracker -- , seq: 85753, time: 2014-09-18 18:08:26.785512, event: done, op: pg_info(1 pgs e11702:2.4c1) -7667> 2014-09-18 18:08:26.785637 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11704..11705 src has 1..11705) v3 -- ?+0 0xe937680 con 0x1395f8c0 -7666> 2014-09-18 18:08:26.785654 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11705:2.4c1) v4 -- ?+0 0xf09c580 con 0x1395f8c0 -7665> 2014-09-18 18:08:26.785719 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6836/27583 -- osd_map(11703..11705 src has 1..11705) v3 -- ?+0 0xe938880 con 0x10aebe40 -7664> 2014-09-18 18:08:26.785734 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6836/27583 -- pg_info(1 pgs e11705:2.4c1) v4 -- ?+0 0xf09b4a0 con 0x10aebe40 -7663> 2014-09-18 18:08:26.785850 7f4c97241700 5 osd.42 pg_epoch: 11705 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11702 n=535 ec=188 les/c 11702/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=27,48 from 0//0//-1 -7662> 2014-09-18 18:08:26.786109 7f4c97241700 5 osd.42 pg_epoch: 11705 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11702 n=535 ec=188 les/c 11702/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 27 from 0//0//-1 -7661> 2014-09-18 18:08:26.786120 7f4c97241700 5 osd.42 pg_epoch: 11705 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11702 n=535 ec=188 les/c 11702/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 48 from 0//0//-1 -7660> 2014-09-18 18:08:26.786140 7f4c97241700 5 osd.42 pg_epoch: 11705 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11702 n=535 ec=188 les/c 11702/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 2.861563 20 0.002009 -7659> 2014-09-18 18:08:26.786166 7f4c97241700 5 osd.42 pg_epoch: 11705 pg[2.4c1( v 6761'26972 (4138'23485,6761'26972] local-les=11702 n=535 ec=188 les/c 11702/9946 11701/11701/10779) [48,42,27]/[42] r=0 lpr=11701 pi=9903-11700/15 bft=27,48 crt=6761'26972 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -7658> 2014-09-18 18:08:26.814784 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3157 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x1182c400 con 0x1395ec60 -7657> 2014-09-18 18:08:26.814813 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7656> 2014-09-18 18:08:26.814824 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3158 ==== pg_notify(2.71(19) epoch 11705) v5 ==== 1795+0+0 (3782450711 0 0) 0xf80e1e0 con 0x1395ec60 -7655> 2014-09-18 18:08:26.814846 7f4c9f251700 5 -- op tracker -- , seq: 85754, time: 2014-09-18 18:08:26.814717, event: header_read, op: pg_notify(2.71(19) epoch 11705) -7654> 2014-09-18 18:08:26.814862 7f4c9f251700 5 -- op tracker -- , seq: 85754, time: 2014-09-18 18:08:26.814717, event: throttled, op: pg_notify(2.71(19) epoch 11705) -7653> 2014-09-18 18:08:26.814872 7f4c9f251700 5 -- op tracker -- , seq: 85754, time: 2014-09-18 18:08:26.814770, event: all_read, op: pg_notify(2.71(19) epoch 11705) -7652> 2014-09-18 18:08:26.814879 7f4c9f251700 5 -- op tracker -- , seq: 85754, time: 2014-09-18 18:08:26.814840, event: dispatched, op: pg_notify(2.71(19) epoch 11705) -7651> 2014-09-18 18:08:26.814887 7f4c9f251700 5 -- op tracker -- , seq: 85754, time: 2014-09-18 18:08:26.814886, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11705) -7650> 2014-09-18 18:08:26.814905 7f4c9f251700 5 -- op tracker -- , seq: 85754, time: 2014-09-18 18:08:26.814903, event: started, op: pg_notify(2.71(19) epoch 11705) -7649> 2014-09-18 18:08:26.814928 7f4c9f251700 5 -- op tracker -- , seq: 85754, time: 2014-09-18 18:08:26.814926, event: done, op: pg_notify(2.71(19) epoch 11705) -7648> 2014-09-18 18:08:26.844363 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 335 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0xe936fc0 con 0x12562260 -7647> 2014-09-18 18:08:26.844402 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11705,11705], i have 11705, src has [1,11705] -7646> 2014-09-18 18:08:26.844417 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 336 ==== pg_notify(2.719(17) epoch 11705) v5 ==== 1601+0+0 (1523380389 0 0) 0xf2ae1e0 con 0x12562260 -7645> 2014-09-18 18:08:26.844437 7f4c9f251700 5 -- op tracker -- , seq: 85755, time: 2014-09-18 18:08:26.844356, event: header_read, op: pg_notify(2.719(17) epoch 11705) -7644> 2014-09-18 18:08:26.844456 7f4c9f251700 5 -- op tracker -- , seq: 85755, time: 2014-09-18 18:08:26.844356, event: throttled, op: pg_notify(2.719(17) epoch 11705) -7643> 2014-09-18 18:08:26.844467 7f4c9f251700 5 -- op tracker -- , seq: 85755, time: 2014-09-18 18:08:26.844400, event: all_read, op: pg_notify(2.719(17) epoch 11705) -7642> 2014-09-18 18:08:26.844477 7f4c9f251700 5 -- op tracker -- , seq: 85755, time: 2014-09-18 18:08:26.844430, event: dispatched, op: pg_notify(2.719(17) epoch 11705) -7641> 2014-09-18 18:08:26.844489 7f4c9f251700 5 -- op tracker -- , seq: 85755, time: 2014-09-18 18:08:26.844486, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11705) -7640> 2014-09-18 18:08:26.844505 7f4c9f251700 5 -- op tracker -- , seq: 85755, time: 2014-09-18 18:08:26.844502, event: started, op: pg_notify(2.719(17) epoch 11705) -7639> 2014-09-18 18:08:26.844545 7f4c9f251700 5 -- op tracker -- , seq: 85755, time: 2014-09-18 18:08:26.844542, event: done, op: pg_notify(2.719(17) epoch 11705) -7638> 2014-09-18 18:08:26.893011 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3159 ==== pg_notify(2.410(11) epoch 11705) v5 ==== 1325+0+0 (580951719 0 0) 0xf810d00 con 0x1395ec60 -7637> 2014-09-18 18:08:26.893038 7f4c9f251700 5 -- op tracker -- , seq: 85756, time: 2014-09-18 18:08:26.892902, event: header_read, op: pg_notify(2.410(11) epoch 11705) -7636> 2014-09-18 18:08:26.893049 7f4c9f251700 5 -- op tracker -- , seq: 85756, time: 2014-09-18 18:08:26.892915, event: throttled, op: pg_notify(2.410(11) epoch 11705) -7635> 2014-09-18 18:08:26.893055 7f4c9f251700 5 -- op tracker -- , seq: 85756, time: 2014-09-18 18:08:26.892988, event: all_read, op: pg_notify(2.410(11) epoch 11705) -7634> 2014-09-18 18:08:26.893061 7f4c9f251700 5 -- op tracker -- , seq: 85756, time: 2014-09-18 18:08:26.893032, event: dispatched, op: pg_notify(2.410(11) epoch 11705) -7633> 2014-09-18 18:08:26.893068 7f4c9f251700 5 -- op tracker -- , seq: 85756, time: 2014-09-18 18:08:26.893066, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11705) -7632> 2014-09-18 18:08:26.893077 7f4c9f251700 5 -- op tracker -- , seq: 85756, time: 2014-09-18 18:08:26.893076, event: started, op: pg_notify(2.410(11) epoch 11705) -7631> 2014-09-18 18:08:26.893213 7f4c9f251700 5 -- op tracker -- , seq: 85756, time: 2014-09-18 18:08:26.893211, event: done, op: pg_notify(2.410(11) epoch 11705) -7630> 2014-09-18 18:08:26.893711 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3160 ==== pg_notify(2.186(12) epoch 11705) v5 ==== 1386+0+0 (275336983 0 0) 0xf80f860 con 0x1395ec60 -7629> 2014-09-18 18:08:26.893722 7f4c9f251700 5 -- op tracker -- , seq: 85757, time: 2014-09-18 18:08:26.893663, event: header_read, op: pg_notify(2.186(12) epoch 11705) -7628> 2014-09-18 18:08:26.893730 7f4c9f251700 5 -- op tracker -- , seq: 85757, time: 2014-09-18 18:08:26.893664, event: throttled, op: pg_notify(2.186(12) epoch 11705) -7627> 2014-09-18 18:08:26.893736 7f4c9f251700 5 -- op tracker -- , seq: 85757, time: 2014-09-18 18:08:26.893699, event: all_read, op: pg_notify(2.186(12) epoch 11705) -7626> 2014-09-18 18:08:26.893742 7f4c9f251700 5 -- op tracker -- , seq: 85757, time: 2014-09-18 18:08:26.893719, event: dispatched, op: pg_notify(2.186(12) epoch 11705) -7625> 2014-09-18 18:08:26.893748 7f4c9f251700 5 -- op tracker -- , seq: 85757, time: 2014-09-18 18:08:26.893747, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11705) -7624> 2014-09-18 18:08:26.893756 7f4c9f251700 5 -- op tracker -- , seq: 85757, time: 2014-09-18 18:08:26.893754, event: started, op: pg_notify(2.186(12) epoch 11705) -7623> 2014-09-18 18:08:26.893773 7f4c9f251700 5 -- op tracker -- , seq: 85757, time: 2014-09-18 18:08:26.893772, event: done, op: pg_notify(2.186(12) epoch 11705) -7622> 2014-09-18 18:08:26.895546 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.46 10.242.42.178:0/4507 117 ==== osd_ping(ping e11705 stamp 2014-09-18 18:08:26.894888) v2 ==== 47+0+0 (3240536955 0 0) 0xf4ee780 con 0x10a5c3c0 -7621> 2014-09-18 18:08:26.895572 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/4507 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:26.894888) v2 -- ?+0 0x21e12b20 con 0x10a5c3c0 -7620> 2014-09-18 18:08:26.895572 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.46 10.242.42.178:0/4507 117 ==== osd_ping(ping e11705 stamp 2014-09-18 18:08:26.894888) v2 ==== 47+0+0 (3240536955 0 0) 0xf2b34a0 con 0x110bb5a0 -7619> 2014-09-18 18:08:26.895612 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/4507 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:26.894888) v2 -- ?+0 0x11b0ef00 con 0x110bb5a0 -7618> 2014-09-18 18:08:26.915194 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5059 ==== pg_notify(2.62(44) epoch 11703) v5 ==== 3220+0+0 (1270411782 0 0) 0xf96d660 con 0x1088aec0 -7617> 2014-09-18 18:08:26.915225 7f4c9f251700 5 -- op tracker -- , seq: 85758, time: 2014-09-18 18:08:26.915096, event: header_read, op: pg_notify(2.62(44) epoch 11703) -7616> 2014-09-18 18:08:26.915237 7f4c9f251700 5 -- op tracker -- , seq: 85758, time: 2014-09-18 18:08:26.915099, event: throttled, op: pg_notify(2.62(44) epoch 11703) -7615> 2014-09-18 18:08:26.915246 7f4c9f251700 5 -- op tracker -- , seq: 85758, time: 2014-09-18 18:08:26.915171, event: all_read, op: pg_notify(2.62(44) epoch 11703) -7614> 2014-09-18 18:08:26.915254 7f4c9f251700 5 -- op tracker -- , seq: 85758, time: 2014-09-18 18:08:26.915218, event: dispatched, op: pg_notify(2.62(44) epoch 11703) -7613> 2014-09-18 18:08:26.915263 7f4c9f251700 5 -- op tracker -- , seq: 85758, time: 2014-09-18 18:08:26.915262, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11703) -7612> 2014-09-18 18:08:26.915273 7f4c9f251700 5 -- op tracker -- , seq: 85758, time: 2014-09-18 18:08:26.915271, event: started, op: pg_notify(2.62(44) epoch 11703) -7611> 2014-09-18 18:08:26.915297 7f4c9f251700 5 -- op tracker -- , seq: 85758, time: 2014-09-18 18:08:26.915295, event: done, op: pg_notify(2.62(44) epoch 11703) -7610> 2014-09-18 18:08:26.918545 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5060 ==== pg_notify(2.3b6(10) epoch 11703) v5 ==== 1260+0+0 (3249039002 0 0) 0xf96ad20 con 0x1088aec0 -7609> 2014-09-18 18:08:26.918575 7f4c9f251700 5 -- op tracker -- , seq: 85759, time: 2014-09-18 18:08:26.918483, event: header_read, op: pg_notify(2.3b6(10) epoch 11703) -7608> 2014-09-18 18:08:26.918587 7f4c9f251700 5 -- op tracker -- , seq: 85759, time: 2014-09-18 18:08:26.918484, event: throttled, op: pg_notify(2.3b6(10) epoch 11703) -7607> 2014-09-18 18:08:26.918594 7f4c9f251700 5 -- op tracker -- , seq: 85759, time: 2014-09-18 18:08:26.918530, event: all_read, op: pg_notify(2.3b6(10) epoch 11703) -7606> 2014-09-18 18:08:26.918600 7f4c9f251700 5 -- op tracker -- , seq: 85759, time: 2014-09-18 18:08:26.918570, event: dispatched, op: pg_notify(2.3b6(10) epoch 11703) -7605> 2014-09-18 18:08:26.918606 7f4c9f251700 5 -- op tracker -- , seq: 85759, time: 2014-09-18 18:08:26.918605, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11703) -7604> 2014-09-18 18:08:26.918615 7f4c9f251700 5 -- op tracker -- , seq: 85759, time: 2014-09-18 18:08:26.918613, event: started, op: pg_notify(2.3b6(10) epoch 11703) -7603> 2014-09-18 18:08:26.918642 7f4c9f251700 5 -- op tracker -- , seq: 85759, time: 2014-09-18 18:08:26.918641, event: done, op: pg_notify(2.3b6(10) epoch 11703) -7602> 2014-09-18 18:08:26.922158 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5061 ==== pg_notify(2.517(11) epoch 11704) v5 ==== 1335+0+0 (971084982 0 0) 0xf96a5a0 con 0x1088aec0 -7601> 2014-09-18 18:08:26.922181 7f4c9f251700 5 -- op tracker -- , seq: 85760, time: 2014-09-18 18:08:26.922092, event: header_read, op: pg_notify(2.517(11) epoch 11704) -7600> 2014-09-18 18:08:26.922191 7f4c9f251700 5 -- op tracker -- , seq: 85760, time: 2014-09-18 18:08:26.922094, event: throttled, op: pg_notify(2.517(11) epoch 11704) -7599> 2014-09-18 18:08:26.922197 7f4c9f251700 5 -- op tracker -- , seq: 85760, time: 2014-09-18 18:08:26.922141, event: all_read, op: pg_notify(2.517(11) epoch 11704) -7598> 2014-09-18 18:08:26.922203 7f4c9f251700 5 -- op tracker -- , seq: 85760, time: 2014-09-18 18:08:26.922175, event: dispatched, op: pg_notify(2.517(11) epoch 11704) -7597> 2014-09-18 18:08:26.922209 7f4c9f251700 5 -- op tracker -- , seq: 85760, time: 2014-09-18 18:08:26.922208, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11704) -7596> 2014-09-18 18:08:26.922218 7f4c9f251700 5 -- op tracker -- , seq: 85760, time: 2014-09-18 18:08:26.922216, event: started, op: pg_notify(2.517(11) epoch 11704) -7595> 2014-09-18 18:08:26.922238 7f4c9f251700 5 -- op tracker -- , seq: 85760, time: 2014-09-18 18:08:26.922236, event: done, op: pg_notify(2.517(11) epoch 11704) -7594> 2014-09-18 18:08:26.926598 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5062 ==== pg_notify(2.120(11) epoch 11704) v5 ==== 1315+0+0 (3809255600 0 0) 0xf96d480 con 0x1088aec0 -7593> 2014-09-18 18:08:26.926630 7f4c9f251700 5 -- op tracker -- , seq: 85761, time: 2014-09-18 18:08:26.926512, event: header_read, op: pg_notify(2.120(11) epoch 11704) -7592> 2014-09-18 18:08:26.926648 7f4c9f251700 5 -- op tracker -- , seq: 85761, time: 2014-09-18 18:08:26.926513, event: throttled, op: pg_notify(2.120(11) epoch 11704) -7591> 2014-09-18 18:08:26.926660 7f4c9f251700 5 -- op tracker -- , seq: 85761, time: 2014-09-18 18:08:26.926569, event: all_read, op: pg_notify(2.120(11) epoch 11704) -7590> 2014-09-18 18:08:26.926672 7f4c9f251700 5 -- op tracker -- , seq: 85761, time: 2014-09-18 18:08:26.926624, event: dispatched, op: pg_notify(2.120(11) epoch 11704) -7589> 2014-09-18 18:08:26.926684 7f4c9f251700 5 -- op tracker -- , seq: 85761, time: 2014-09-18 18:08:26.926681, event: waiting_for_osdmap, op: pg_notify(2.120(11) epoch 11704) -7588> 2014-09-18 18:08:26.926700 7f4c9f251700 5 -- op tracker -- , seq: 85761, time: 2014-09-18 18:08:26.926697, event: started, op: pg_notify(2.120(11) epoch 11704) -7587> 2014-09-18 18:08:26.926735 7f4c9f251700 5 -- op tracker -- , seq: 85761, time: 2014-09-18 18:08:26.926732, event: done, op: pg_notify(2.120(11) epoch 11704) -7586> 2014-09-18 18:08:26.926850 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5063 ==== pg_notify(2.40a(162) epoch 11704) v5 ==== 9736+0+0 (382027185 0 0) 0xf96c580 con 0x1088aec0 -7585> 2014-09-18 18:08:26.926872 7f4c9f251700 5 -- op tracker -- , seq: 85762, time: 2014-09-18 18:08:26.926635, event: header_read, op: pg_notify(2.40a(162) epoch 11704) -7584> 2014-09-18 18:08:26.926885 7f4c9f251700 5 -- op tracker -- , seq: 85762, time: 2014-09-18 18:08:26.926636, event: throttled, op: pg_notify(2.40a(162) epoch 11704) -7583> 2014-09-18 18:08:26.926897 7f4c9f251700 5 -- op tracker -- , seq: 85762, time: 2014-09-18 18:08:26.926824, event: all_read, op: pg_notify(2.40a(162) epoch 11704) -7582> 2014-09-18 18:08:26.926908 7f4c9f251700 5 -- op tracker -- , seq: 85762, time: 2014-09-18 18:08:26.926866, event: dispatched, op: pg_notify(2.40a(162) epoch 11704) -7581> 2014-09-18 18:08:26.926919 7f4c9f251700 5 -- op tracker -- , seq: 85762, time: 2014-09-18 18:08:26.926916, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11704) -7580> 2014-09-18 18:08:26.926933 7f4c9f251700 5 -- op tracker -- , seq: 85762, time: 2014-09-18 18:08:26.926930, event: started, op: pg_notify(2.40a(162) epoch 11704) -7579> 2014-09-18 18:08:26.926973 7f4c9f251700 5 -- op tracker -- , seq: 85762, time: 2014-09-18 18:08:26.926970, event: done, op: pg_notify(2.40a(162) epoch 11704) -7578> 2014-09-18 18:08:26.929389 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5064 ==== pg_notify(2.1a2(20) epoch 11704) v5 ==== 1798+0+0 (3838160904 0 0) 0xf96ab40 con 0x1088aec0 -7577> 2014-09-18 18:08:26.929412 7f4c9f251700 5 -- op tracker -- , seq: 85763, time: 2014-09-18 18:08:26.929332, event: header_read, op: pg_notify(2.1a2(20) epoch 11704) -7576> 2014-09-18 18:08:26.929426 7f4c9f251700 5 -- op tracker -- , seq: 85763, time: 2014-09-18 18:08:26.929333, event: throttled, op: pg_notify(2.1a2(20) epoch 11704) -7575> 2014-09-18 18:08:26.929437 7f4c9f251700 5 -- op tracker -- , seq: 85763, time: 2014-09-18 18:08:26.929373, event: all_read, op: pg_notify(2.1a2(20) epoch 11704) -7574> 2014-09-18 18:08:26.929447 7f4c9f251700 5 -- op tracker -- , seq: 85763, time: 2014-09-18 18:08:26.929406, event: dispatched, op: pg_notify(2.1a2(20) epoch 11704) -7573> 2014-09-18 18:08:26.929458 7f4c9f251700 5 -- op tracker -- , seq: 85763, time: 2014-09-18 18:08:26.929455, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11704) -7572> 2014-09-18 18:08:26.929472 7f4c9f251700 5 -- op tracker -- , seq: 85763, time: 2014-09-18 18:08:26.929469, event: started, op: pg_notify(2.1a2(20) epoch 11704) -7571> 2014-09-18 18:08:26.929500 7f4c9f251700 5 -- op tracker -- , seq: 85763, time: 2014-09-18 18:08:26.929497, event: done, op: pg_notify(2.1a2(20) epoch 11704) -7570> 2014-09-18 18:08:26.932355 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5065 ==== pg_notify(2.4c1(15) epoch 11704) v5 ==== 1507+0+0 (369753307 0 0) 0xf96dc00 con 0x1088aec0 -7569> 2014-09-18 18:08:26.932378 7f4c9f251700 5 -- op tracker -- , seq: 85764, time: 2014-09-18 18:08:26.932251, event: header_read, op: pg_notify(2.4c1(15) epoch 11704) -7568> 2014-09-18 18:08:26.932392 7f4c9f251700 5 -- op tracker -- , seq: 85764, time: 2014-09-18 18:08:26.932252, event: throttled, op: pg_notify(2.4c1(15) epoch 11704) -7567> 2014-09-18 18:08:26.932402 7f4c9f251700 5 -- op tracker -- , seq: 85764, time: 2014-09-18 18:08:26.932337, event: all_read, op: pg_notify(2.4c1(15) epoch 11704) -7566> 2014-09-18 18:08:26.932412 7f4c9f251700 5 -- op tracker -- , seq: 85764, time: 2014-09-18 18:08:26.932372, event: dispatched, op: pg_notify(2.4c1(15) epoch 11704) -7565> 2014-09-18 18:08:26.932423 7f4c9f251700 5 -- op tracker -- , seq: 85764, time: 2014-09-18 18:08:26.932421, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11704) -7564> 2014-09-18 18:08:26.932434 7f4c9f251700 5 -- op tracker -- , seq: 85764, time: 2014-09-18 18:08:26.932432, event: started, op: pg_notify(2.4c1(15) epoch 11704) -7563> 2014-09-18 18:08:26.932454 7f4c9f251700 5 -- op tracker -- , seq: 85764, time: 2014-09-18 18:08:26.932453, event: done, op: pg_notify(2.4c1(15) epoch 11704) -7562> 2014-09-18 18:08:26.934512 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5066 ==== pg_notify(2.62(44) epoch 11704) v5 ==== 3220+0+0 (2282213186 0 0) 0x2b94a000 con 0x1088aec0 -7561> 2014-09-18 18:08:26.934554 7f4c9f251700 5 -- op tracker -- , seq: 85765, time: 2014-09-18 18:08:26.934431, event: header_read, op: pg_notify(2.62(44) epoch 11704) -7560> 2014-09-18 18:08:26.934565 7f4c9f251700 5 -- op tracker -- , seq: 85765, time: 2014-09-18 18:08:26.934432, event: throttled, op: pg_notify(2.62(44) epoch 11704) -7559> 2014-09-18 18:08:26.934570 7f4c9f251700 5 -- op tracker -- , seq: 85765, time: 2014-09-18 18:08:26.934496, event: all_read, op: pg_notify(2.62(44) epoch 11704) -7558> 2014-09-18 18:08:26.934574 7f4c9f251700 5 -- op tracker -- , seq: 85765, time: 2014-09-18 18:08:26.934549, event: dispatched, op: pg_notify(2.62(44) epoch 11704) -7557> 2014-09-18 18:08:26.934589 7f4c9f251700 5 -- op tracker -- , seq: 85765, time: 2014-09-18 18:08:26.934588, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11704) -7556> 2014-09-18 18:08:26.934595 7f4c9f251700 5 -- op tracker -- , seq: 85765, time: 2014-09-18 18:08:26.934594, event: started, op: pg_notify(2.62(44) epoch 11704) -7555> 2014-09-18 18:08:26.934740 7f4c9f251700 5 -- op tracker -- , seq: 85765, time: 2014-09-18 18:08:26.934738, event: done, op: pg_notify(2.62(44) epoch 11704) -7554> 2014-09-18 18:08:26.936191 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5067 ==== pg_notify(2.3b6(10) epoch 11704) v5 ==== 1260+0+0 (1111088832 0 0) 0x2b94c1c0 con 0x1088aec0 -7553> 2014-09-18 18:08:26.936210 7f4c9f251700 5 -- op tracker -- , seq: 85766, time: 2014-09-18 18:08:26.936109, event: header_read, op: pg_notify(2.3b6(10) epoch 11704) -7552> 2014-09-18 18:08:26.936219 7f4c9f251700 5 -- op tracker -- , seq: 85766, time: 2014-09-18 18:08:26.936110, event: throttled, op: pg_notify(2.3b6(10) epoch 11704) -7551> 2014-09-18 18:08:26.936225 7f4c9f251700 5 -- op tracker -- , seq: 85766, time: 2014-09-18 18:08:26.936167, event: all_read, op: pg_notify(2.3b6(10) epoch 11704) -7550> 2014-09-18 18:08:26.936231 7f4c9f251700 5 -- op tracker -- , seq: 85766, time: 2014-09-18 18:08:26.936206, event: dispatched, op: pg_notify(2.3b6(10) epoch 11704) -7549> 2014-09-18 18:08:26.936238 7f4c9f251700 5 -- op tracker -- , seq: 85766, time: 2014-09-18 18:08:26.936236, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11704) -7548> 2014-09-18 18:08:26.936245 7f4c9f251700 5 -- op tracker -- , seq: 85766, time: 2014-09-18 18:08:26.936244, event: started, op: pg_notify(2.3b6(10) epoch 11704) -7547> 2014-09-18 18:08:26.936263 7f4c9f251700 5 -- op tracker -- , seq: 85766, time: 2014-09-18 18:08:26.936261, event: done, op: pg_notify(2.3b6(10) epoch 11704) -7546> 2014-09-18 18:08:26.937294 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.24 10.242.42.172:0/5612 111 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:16.041391) v2 ==== 47+0+0 (4291315436 0 0) 0xf72e5a0 con 0x10a5bb80 -7545> 2014-09-18 18:08:26.937320 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:16.041391) v2 -- ?+0 0xf4ee780 con 0x10a5bb80 -7544> 2014-09-18 18:08:26.938184 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.24 10.242.42.172:0/5612 111 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:16.041391) v2 ==== 47+0+0 (4291315436 0 0) 0x12c12580 con 0x110bb020 -7543> 2014-09-18 18:08:26.938206 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:16.041391) v2 -- ?+0 0xf2b34a0 con 0x110bb020 -7542> 2014-09-18 18:08:26.964531 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3161 ==== pg_notify(2.74e(11) epoch 11705) v5 ==== 1303+0+0 (969641544 0 0) 0xf80e960 con 0x1395ec60 -7541> 2014-09-18 18:08:26.964562 7f4c9f251700 5 -- op tracker -- , seq: 85767, time: 2014-09-18 18:08:26.964411, event: header_read, op: pg_notify(2.74e(11) epoch 11705) -7540> 2014-09-18 18:08:26.964575 7f4c9f251700 5 -- op tracker -- , seq: 85767, time: 2014-09-18 18:08:26.964416, event: throttled, op: pg_notify(2.74e(11) epoch 11705) -7539> 2014-09-18 18:08:26.964583 7f4c9f251700 5 -- op tracker -- , seq: 85767, time: 2014-09-18 18:08:26.964472, event: all_read, op: pg_notify(2.74e(11) epoch 11705) -7538> 2014-09-18 18:08:26.964591 7f4c9f251700 5 -- op tracker -- , seq: 85767, time: 2014-09-18 18:08:26.964555, event: dispatched, op: pg_notify(2.74e(11) epoch 11705) -7537> 2014-09-18 18:08:26.964600 7f4c9f251700 5 -- op tracker -- , seq: 85767, time: 2014-09-18 18:08:26.964599, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11705) -7536> 2014-09-18 18:08:26.964611 7f4c9f251700 5 -- op tracker -- , seq: 85767, time: 2014-09-18 18:08:26.964609, event: started, op: pg_notify(2.74e(11) epoch 11705) -7535> 2014-09-18 18:08:26.964635 7f4c9f251700 5 -- op tracker -- , seq: 85767, time: 2014-09-18 18:08:26.964634, event: done, op: pg_notify(2.74e(11) epoch 11705) -7534> 2014-09-18 18:08:27.004908 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.43 10.242.42.178:0/4511 145 ==== osd_ping(ping e11705 stamp 2014-09-18 18:08:27.004259) v2 ==== 47+0+0 (75342426 0 0) 0x21cfd840 con 0x17de6000 -7533> 2014-09-18 18:08:27.004978 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.43 10.242.42.178:0/4511 145 ==== osd_ping(ping e11705 stamp 2014-09-18 18:08:27.004259) v2 ==== 47+0+0 (75342426 0 0) 0xf09ef00 con 0xcc76dc0 -7532> 2014-09-18 18:08:27.004991 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/4511 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:27.004259) v2 -- ?+0 0x12c12580 con 0x17de6000 -7531> 2014-09-18 18:08:27.005027 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/4511 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:27.004259) v2 -- ?+0 0xf72e5a0 con 0xcc76dc0 -7530> 2014-09-18 18:08:27.039764 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3162 ==== pg_notify(2.4b1(13) epoch 11705) v5 ==== 1441+0+0 (3459346837 0 0) 0xf80ef00 con 0x1395ec60 -7529> 2014-09-18 18:08:27.039793 7f4c9f251700 5 -- op tracker -- , seq: 85768, time: 2014-09-18 18:08:27.039652, event: header_read, op: pg_notify(2.4b1(13) epoch 11705) -7528> 2014-09-18 18:08:27.039806 7f4c9f251700 5 -- op tracker -- , seq: 85768, time: 2014-09-18 18:08:27.039655, event: throttled, op: pg_notify(2.4b1(13) epoch 11705) -7527> 2014-09-18 18:08:27.039823 7f4c9f251700 5 -- op tracker -- , seq: 85768, time: 2014-09-18 18:08:27.039728, event: all_read, op: pg_notify(2.4b1(13) epoch 11705) -7526> 2014-09-18 18:08:27.039830 7f4c9f251700 5 -- op tracker -- , seq: 85768, time: 2014-09-18 18:08:27.039786, event: dispatched, op: pg_notify(2.4b1(13) epoch 11705) -7525> 2014-09-18 18:08:27.039838 7f4c9f251700 5 -- op tracker -- , seq: 85768, time: 2014-09-18 18:08:27.039836, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11705) -7524> 2014-09-18 18:08:27.039849 7f4c9f251700 5 -- op tracker -- , seq: 85768, time: 2014-09-18 18:08:27.039848, event: started, op: pg_notify(2.4b1(13) epoch 11705) -7523> 2014-09-18 18:08:27.039875 7f4c9f251700 5 -- op tracker -- , seq: 85768, time: 2014-09-18 18:08:27.039873, event: done, op: pg_notify(2.4b1(13) epoch 11705) -7522> 2014-09-18 18:08:27.040422 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3163 ==== pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11705) v5 ==== 8942+0+0 (2731739575 0 0) 0xf80f2c0 con 0x1395ec60 -7521> 2014-09-18 18:08:27.040437 7f4c9f251700 5 -- op tracker -- , seq: 85769, time: 2014-09-18 18:08:27.040279, event: header_read, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11705) -7520> 2014-09-18 18:08:27.040445 7f4c9f251700 5 -- op tracker -- , seq: 85769, time: 2014-09-18 18:08:27.040281, event: throttled, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11705) -7519> 2014-09-18 18:08:27.040450 7f4c9f251700 5 -- op tracker -- , seq: 85769, time: 2014-09-18 18:08:27.040397, event: all_read, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11705) -7518> 2014-09-18 18:08:27.040462 7f4c9f251700 5 -- op tracker -- , seq: 85769, time: 2014-09-18 18:08:27.040433, event: dispatched, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11705) -7517> 2014-09-18 18:08:27.040469 7f4c9f251700 5 -- op tracker -- , seq: 85769, time: 2014-09-18 18:08:27.040467, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11705) -7516> 2014-09-18 18:08:27.040478 7f4c9f251700 5 -- op tracker -- , seq: 85769, time: 2014-09-18 18:08:27.040476, event: started, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11705) -7515> 2014-09-18 18:08:27.040523 7f4c9f251700 5 -- op tracker -- , seq: 85769, time: 2014-09-18 18:08:27.040520, event: done, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15),2.799(9),2.4d0(10) epoch 11705) -7514> 2014-09-18 18:08:27.205623 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.27 10.242.42.174:0/21548 148 ==== osd_ping(ping e11705 stamp 2014-09-18 18:06:33.036460) v2 ==== 47+0+0 (1874585817 0 0) 0x130405a0 con 0x107358c0 -7513> 2014-09-18 18:08:27.205665 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.174:0/21548 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:06:33.036460) v2 -- ?+0 0x21cfd840 con 0x107358c0 -7512> 2014-09-18 18:08:27.205701 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.27 10.242.42.174:0/21548 148 ==== osd_ping(ping e11705 stamp 2014-09-18 18:06:33.036460) v2 ==== 47+0+0 (1874585817 0 0) 0xf2752a0 con 0x101cb4a0 -7511> 2014-09-18 18:08:27.205726 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.174:0/21548 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:06:33.036460) v2 -- ?+0 0xf09ef00 con 0x101cb4a0 -7510> 2014-09-18 18:08:27.249784 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.39 10.242.42.172:0/2660 1588 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:16.354152) v2 ==== 47+0+0 (3677694191 0 0) 0x11b0fc20 con 0xec39340 -7509> 2014-09-18 18:08:27.249780 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.39 10.242.42.172:0/2660 1588 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:16.354152) v2 ==== 47+0+0 (3677694191 0 0) 0x11b11c00 con 0x4750840 -7508> 2014-09-18 18:08:27.249824 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:16.354152) v2 -- ?+0 0x130405a0 con 0xec39340 -7507> 2014-09-18 18:08:27.249870 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:16.354152) v2 -- ?+0 0xf2752a0 con 0x4750840 -7506> 2014-09-18 18:08:27.381613 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.56 10.242.42.172:0/36458 652 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:16.486135) v2 ==== 47+0+0 (1422257536 0 0) 0xf0e4ee0 con 0x14392c60 -7505> 2014-09-18 18:08:27.381651 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/36458 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:16.486135) v2 -- ?+0 0x11b11c00 con 0x14392c60 -7504> 2014-09-18 18:08:27.381656 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.56 10.242.42.172:0/36458 652 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:16.486135) v2 ==== 47+0+0 (1422257536 0 0) 0x21d14580 con 0x10a81700 -7503> 2014-09-18 18:08:27.381686 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/36458 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:16.486135) v2 -- ?+0 0x11b0fc20 con 0x10a81700 -7502> 2014-09-18 18:08:27.517988 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.52 10.242.42.168:0/16848 128 ==== osd_ping(ping e11705 stamp 2014-09-18 18:08:27.384726) v2 ==== 47+0+0 (3631552578 0 0) 0xedfbe00 con 0xcc49b80 -7501> 2014-09-18 18:08:27.517986 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.52 10.242.42.168:0/16848 128 ==== osd_ping(ping e11705 stamp 2014-09-18 18:08:27.384726) v2 ==== 47+0+0 (3631552578 0 0) 0xf109c00 con 0x101cc520 -7500> 2014-09-18 18:08:27.518032 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16848 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:27.384726) v2 -- ?+0 0x21d14580 con 0xcc49b80 -7499> 2014-09-18 18:08:27.518065 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16848 -- osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:27.384726) v2 -- ?+0 0xf0e4ee0 con 0x101cc520 -7498> 2014-09-18 18:08:27.555213 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.55 10.242.42.172:0/19404 1103 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:16.659586) v2 ==== 47+0+0 (668465187 0 0) 0xe7745a0 con 0xeddec60 -7497> 2014-09-18 18:08:27.555255 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/19404 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:16.659586) v2 -- ?+0 0xf109c00 con 0xeddec60 -7496> 2014-09-18 18:08:27.555319 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.55 10.242.42.172:0/19404 1103 ==== osd_ping(ping e11705 stamp 2014-09-18 23:37:16.659586) v2 ==== 47+0+0 (668465187 0 0) 0xe288580 con 0x4b7ab00 -7495> 2014-09-18 18:08:27.555342 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/19404 -- osd_ping(ping_reply e11705 stamp 2014-09-18 23:37:16.659586) v2 -- ?+0 0xedfbe00 con 0x4b7ab00 -7494> 2014-09-18 18:08:27.680041 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2590 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x19df3840 con 0x4900b00 -7493> 2014-09-18 18:08:27.680093 7f4c9f251700 3 osd.42 11705 handle_osd_map epochs [11706,11706], i have 11705, src has [1,11706] -7492> 2014-09-18 18:08:27.682623 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11696/8547) [24,27,17]/[24,27,42] r=2 lpr=11696 pi=11505-11695/5 luod=0'0 crt=6767'20194 active+remapped] exit Started/ReplicaActive/RepNotRecovering 9.831016 28 0.000718 -7491> 2014-09-18 18:08:27.682655 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11696/8547) [24,27,17]/[24,27,42] r=2 lpr=11696 pi=11505-11695/5 luod=0'0 crt=6767'20194 active+remapped] exit Started/ReplicaActive 9.831086 0 0.000000 -7490> 2014-09-18 18:08:27.682679 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11696/8547) [24,27,17]/[24,27,42] r=2 lpr=11696 pi=11505-11695/5 luod=0'0 crt=6767'20194 active+remapped] exit Started 10.841524 0 0.000000 -7489> 2014-09-18 18:08:27.682697 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11696/8547) [24,27,17]/[24,27,42] r=2 lpr=11696 pi=11505-11695/5 luod=0'0 crt=6767'20194 active+remapped] enter Reset -7488> 2014-09-18 18:08:27.682770 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11706/8547) [24,27,17] r=-1 lpr=11706 pi=11505-11705/6 crt=6767'20194 inactive NOTIFY] exit Reset 0.000074 1 0.000132 -7487> 2014-09-18 18:08:27.682876 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11706/8547) [24,27,17] r=-1 lpr=11706 pi=11505-11705/6 crt=6767'20194 inactive NOTIFY] enter Started -7486> 2014-09-18 18:08:27.682891 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11706/8547) [24,27,17] r=-1 lpr=11706 pi=11505-11705/6 crt=6767'20194 inactive NOTIFY] enter Start -7485> 2014-09-18 18:08:27.682903 7f4c97241700 1 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11706/8547) [24,27,17] r=-1 lpr=11706 pi=11505-11705/6 crt=6767'20194 inactive NOTIFY] state: transitioning to Stray -7484> 2014-09-18 18:08:27.682919 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11706/8547) [24,27,17] r=-1 lpr=11706 pi=11505-11705/6 crt=6767'20194 inactive NOTIFY] exit Start 0.000028 0 0.000000 -7483> 2014-09-18 18:08:27.682932 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.280( v 6767'20194 (4068'16425,6767'20194] local-les=11697 n=499 ec=188 les/c 11697/11704 11695/11706/8547) [24,27,17] r=-1 lpr=11706 pi=11505-11705/6 crt=6767'20194 inactive NOTIFY] enter Started/Stray -7482> 2014-09-18 18:08:27.682943 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2591 ==== pg_notify(2.71c(12) epoch 11706) v5 ==== 1424+0+0 (3970789069 0 0) 0xf243680 con 0x4900b00 -7481> 2014-09-18 18:08:27.682972 7f4c9f251700 5 -- op tracker -- , seq: 85770, time: 2014-09-18 18:08:27.680036, event: header_read, op: pg_notify(2.71c(12) epoch 11706) -7480> 2014-09-18 18:08:27.682988 7f4c9f251700 5 -- op tracker -- , seq: 85770, time: 2014-09-18 18:08:27.680037, event: throttled, op: pg_notify(2.71c(12) epoch 11706) -7479> 2014-09-18 18:08:27.682998 7f4c9f251700 5 -- op tracker -- , seq: 85770, time: 2014-09-18 18:08:27.680117, event: all_read, op: pg_notify(2.71c(12) epoch 11706) -7478> 2014-09-18 18:08:27.682999 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11706..11706 src has 1..11706) v3 -- ?+0 0x10d19600 con 0x140f95a0 -7477> 2014-09-18 18:08:27.683009 7f4c9f251700 5 -- op tracker -- , seq: 85770, time: 2014-09-18 18:08:27.682966, event: dispatched, op: pg_notify(2.71c(12) epoch 11706) -7476> 2014-09-18 18:08:27.683016 7f4c9f251700 5 -- op tracker -- , seq: 85770, time: 2014-09-18 18:08:27.683014, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11706) -7475> 2014-09-18 18:08:27.683027 7f4c9f251700 5 -- op tracker -- , seq: 85770, time: 2014-09-18 18:08:27.683025, event: started, op: pg_notify(2.71c(12) epoch 11706) -7474> 2014-09-18 18:08:27.683047 7f4c9f251700 5 -- op tracker -- , seq: 85770, time: 2014-09-18 18:08:27.683045, event: done, op: pg_notify(2.71c(12) epoch 11706) -7473> 2014-09-18 18:08:27.683063 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1258 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0xeeea1c0 con 0xe8db760 -7472> 2014-09-18 18:08:27.683077 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7471> 2014-09-18 18:08:27.683094 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1032 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0xe937f80 con 0x14358260 -7470> 2014-09-18 18:08:27.683104 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7469> 2014-09-18 18:08:27.683112 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1259 ==== pg_notify(2.17e(17) epoch 11706) v5 ==== 1593+0+0 (3989242314 0 0) 0xf107e00 con 0xe8db760 -7468> 2014-09-18 18:08:27.683127 7f4c9f251700 5 -- op tracker -- , seq: 85771, time: 2014-09-18 18:08:27.682766, event: header_read, op: pg_notify(2.17e(17) epoch 11706) -7467> 2014-09-18 18:08:27.683126 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11706) v5 -- ?+0 0x146de780 con 0x140f95a0 -7466> 2014-09-18 18:08:27.683142 7f4c9f251700 5 -- op tracker -- , seq: 85771, time: 2014-09-18 18:08:27.682767, event: throttled, op: pg_notify(2.17e(17) epoch 11706) -7465> 2014-09-18 18:08:27.683154 7f4c9f251700 5 -- op tracker -- , seq: 85771, time: 2014-09-18 18:08:27.682802, event: all_read, op: pg_notify(2.17e(17) epoch 11706) -7464> 2014-09-18 18:08:27.683162 7f4c9f251700 5 -- op tracker -- , seq: 85771, time: 2014-09-18 18:08:27.683123, event: dispatched, op: pg_notify(2.17e(17) epoch 11706) -7463> 2014-09-18 18:08:27.683172 7f4c9f251700 5 -- op tracker -- , seq: 85771, time: 2014-09-18 18:08:27.683169, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11706) -7462> 2014-09-18 18:08:27.683191 7f4c9f251700 5 -- op tracker -- , seq: 85771, time: 2014-09-18 18:08:27.683180, event: started, op: pg_notify(2.17e(17) epoch 11706) -7461> 2014-09-18 18:08:27.683211 7f4c9f251700 5 -- op tracker -- , seq: 85771, time: 2014-09-18 18:08:27.683209, event: done, op: pg_notify(2.17e(17) epoch 11706) -7460> 2014-09-18 18:08:27.683226 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11706..11706 src has 1..11706) v3 -- ?+0 0x10d17f80 con 0x1395edc0 -7459> 2014-09-18 18:08:27.683226 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1033 ==== pg_notify(2.719(17) epoch 11706) v5 ==== 1601+0+0 (574432646 0 0) 0xf7312a0 con 0x14358260 -7458> 2014-09-18 18:08:27.683260 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11706) v5 -- ?+0 0x146de5a0 con 0x1395edc0 -7457> 2014-09-18 18:08:27.683262 7f4c9f251700 5 -- op tracker -- , seq: 85772, time: 2014-09-18 18:08:27.682376, event: header_read, op: pg_notify(2.719(17) epoch 11706) -7456> 2014-09-18 18:08:27.683272 7f4c9f251700 5 -- op tracker -- , seq: 85772, time: 2014-09-18 18:08:27.682377, event: throttled, op: pg_notify(2.719(17) epoch 11706) -7455> 2014-09-18 18:08:27.683281 7f4c9f251700 5 -- op tracker -- , seq: 85772, time: 2014-09-18 18:08:27.682429, event: all_read, op: pg_notify(2.719(17) epoch 11706) -7454> 2014-09-18 18:08:27.683291 7f4c9f251700 5 -- op tracker -- , seq: 85772, time: 2014-09-18 18:08:27.683253, event: dispatched, op: pg_notify(2.719(17) epoch 11706) -7453> 2014-09-18 18:08:27.683301 7f4c9f251700 5 -- op tracker -- , seq: 85772, time: 2014-09-18 18:08:27.683298, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11706) -7452> 2014-09-18 18:08:27.683312 7f4c9f251700 5 -- op tracker -- , seq: 85772, time: 2014-09-18 18:08:27.683309, event: started, op: pg_notify(2.719(17) epoch 11706) -7451> 2014-09-18 18:08:27.683334 7f4c9f251700 5 -- op tracker -- , seq: 85772, time: 2014-09-18 18:08:27.683331, event: done, op: pg_notify(2.719(17) epoch 11706) -7450> 2014-09-18 18:08:27.683543 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 917 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0xe937680 con 0x1395f8c0 -7449> 2014-09-18 18:08:27.683563 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7448> 2014-09-18 18:08:27.683635 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 918 ==== pg_notify(2.410(11) epoch 11706) v5 ==== 1287+0+0 (670918777 0 0) 0xf09c580 con 0x1395f8c0 -7447> 2014-09-18 18:08:27.683653 7f4c9f251700 5 -- op tracker -- , seq: 85773, time: 2014-09-18 18:08:27.683576, event: header_read, op: pg_notify(2.410(11) epoch 11706) -7446> 2014-09-18 18:08:27.683665 7f4c9f251700 5 -- op tracker -- , seq: 85773, time: 2014-09-18 18:08:27.683577, event: throttled, op: pg_notify(2.410(11) epoch 11706) -7445> 2014-09-18 18:08:27.683674 7f4c9f251700 5 -- op tracker -- , seq: 85773, time: 2014-09-18 18:08:27.683619, event: all_read, op: pg_notify(2.410(11) epoch 11706) -7444> 2014-09-18 18:08:27.683683 7f4c9f251700 5 -- op tracker -- , seq: 85773, time: 2014-09-18 18:08:27.683648, event: dispatched, op: pg_notify(2.410(11) epoch 11706) -7443> 2014-09-18 18:08:27.683692 7f4c9f251700 5 -- op tracker -- , seq: 85773, time: 2014-09-18 18:08:27.683690, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11706) -7442> 2014-09-18 18:08:27.683704 7f4c9f251700 5 -- op tracker -- , seq: 85773, time: 2014-09-18 18:08:27.683701, event: started, op: pg_notify(2.410(11) epoch 11706) -7441> 2014-09-18 18:08:27.683733 7f4c9f251700 5 -- op tracker -- , seq: 85773, time: 2014-09-18 18:08:27.683731, event: done, op: pg_notify(2.410(11) epoch 11706) -7440> 2014-09-18 18:08:27.683880 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- osd_map(11706..11706 src has 1..11706) v3 -- ?+0 0xeeea640 con 0xdd8c940 -7439> 2014-09-18 18:08:27.683927 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- pg_notify(2.280(6) epoch 11706) v5 -- ?+0 0xf2b6780 con 0xdd8c940 -7438> 2014-09-18 18:08:27.684004 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11706..11706 src has 1..11706) v3 -- ?+0 0x14409d40 con 0x1395f1e0 -7437> 2014-09-18 18:08:27.684048 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11706) v5 -- ?+0 0x19f22d20 con 0x1395f1e0 -7436> 2014-09-18 18:08:27.684172 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11706) v5 -- ?+0 0xe796d00 con 0x140f95a0 -7435> 2014-09-18 18:08:27.684672 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3612 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x1533a000 con 0xcbecec0 -7434> 2014-09-18 18:08:27.684692 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7433> 2014-09-18 18:08:27.684719 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3613 ==== pg_notify(2.4f6(12) epoch 11706) v5 ==== 1382+0+0 (799839422 0 0) 0xf2432c0 con 0xcbecec0 -7432> 2014-09-18 18:08:27.684756 7f4c9f251700 5 -- op tracker -- , seq: 85774, time: 2014-09-18 18:08:27.684673, event: header_read, op: pg_notify(2.4f6(12) epoch 11706) -7431> 2014-09-18 18:08:27.684766 7f4c9f251700 5 -- op tracker -- , seq: 85774, time: 2014-09-18 18:08:27.684673, event: throttled, op: pg_notify(2.4f6(12) epoch 11706) -7430> 2014-09-18 18:08:27.684773 7f4c9f251700 5 -- op tracker -- , seq: 85774, time: 2014-09-18 18:08:27.684704, event: all_read, op: pg_notify(2.4f6(12) epoch 11706) -7429> 2014-09-18 18:08:27.684783 7f4c9f251700 5 -- op tracker -- , seq: 85774, time: 2014-09-18 18:08:27.684751, event: dispatched, op: pg_notify(2.4f6(12) epoch 11706) -7428> 2014-09-18 18:08:27.684793 7f4c9f251700 5 -- op tracker -- , seq: 85774, time: 2014-09-18 18:08:27.684791, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11706) -7427> 2014-09-18 18:08:27.684805 7f4c9f251700 5 -- op tracker -- , seq: 85774, time: 2014-09-18 18:08:27.684803, event: started, op: pg_notify(2.4f6(12) epoch 11706) -7426> 2014-09-18 18:08:27.684822 7f4c9f251700 5 -- op tracker -- , seq: 85774, time: 2014-09-18 18:08:27.684821, event: done, op: pg_notify(2.4f6(12) epoch 11706) -7425> 2014-09-18 18:08:27.685200 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11706..11706 src has 1..11706) v3 -- ?+0 0x2fb9ed80 con 0x1395ef20 -7424> 2014-09-18 18:08:27.685216 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(10) epoch 11706) v5 -- ?+0 0xf344b20 con 0x1395ef20 -7423> 2014-09-18 18:08:27.686078 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11706..11706 src has 1..11706) v3 -- ?+0 0x2db44b40 con 0x23a74f20 -7422> 2014-09-18 18:08:27.686092 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11706) v5 -- ?+0 0x1390dc00 con 0x23a74f20 -7421> 2014-09-18 18:08:27.687502 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 523 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x2fb9ed80 con 0x1395ef20 -7420> 2014-09-18 18:08:27.687529 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7419> 2014-09-18 18:08:27.687551 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 524 ==== pg_notify(2.4d0(10) epoch 11706) v5 ==== 1292+0+0 (1636364873 0 0) 0xf344b20 con 0x1395ef20 -7418> 2014-09-18 18:08:27.687566 7f4c9f251700 5 -- op tracker -- , seq: 85775, time: 2014-09-18 18:08:27.687504, event: header_read, op: pg_notify(2.4d0(10) epoch 11706) -7417> 2014-09-18 18:08:27.687576 7f4c9f251700 5 -- op tracker -- , seq: 85775, time: 2014-09-18 18:08:27.687505, event: throttled, op: pg_notify(2.4d0(10) epoch 11706) -7416> 2014-09-18 18:08:27.687582 7f4c9f251700 5 -- op tracker -- , seq: 85775, time: 2014-09-18 18:08:27.687540, event: all_read, op: pg_notify(2.4d0(10) epoch 11706) -7415> 2014-09-18 18:08:27.687589 7f4c9f251700 5 -- op tracker -- , seq: 85775, time: 2014-09-18 18:08:27.687562, event: dispatched, op: pg_notify(2.4d0(10) epoch 11706) -7414> 2014-09-18 18:08:27.687596 7f4c9f251700 5 -- op tracker -- , seq: 85775, time: 2014-09-18 18:08:27.687594, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11706) -7413> 2014-09-18 18:08:27.687603 7f4c9f251700 5 -- op tracker -- , seq: 85775, time: 2014-09-18 18:08:27.687602, event: started, op: pg_notify(2.4d0(10) epoch 11706) -7412> 2014-09-18 18:08:27.687624 7f4c9f251700 5 -- op tracker -- , seq: 85775, time: 2014-09-18 18:08:27.687622, event: done, op: pg_notify(2.4d0(10) epoch 11706) -7411> 2014-09-18 18:08:27.687818 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2343 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0xeeea640 con 0xdd8c940 -7410> 2014-09-18 18:08:27.687843 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7409> 2014-09-18 18:08:27.687852 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2344 ==== pg_query(2.280 epoch 11706) v3 ==== 144+0+0 (2842749046 0 0) 0xf2b6780 con 0xdd8c940 -7408> 2014-09-18 18:08:27.687863 7f4c9f251700 5 -- op tracker -- , seq: 85776, time: 2014-09-18 18:08:27.687818, event: header_read, op: pg_query(2.280 epoch 11706) -7407> 2014-09-18 18:08:27.687872 7f4c9f251700 5 -- op tracker -- , seq: 85776, time: 2014-09-18 18:08:27.687819, event: throttled, op: pg_query(2.280 epoch 11706) -7406> 2014-09-18 18:08:27.687879 7f4c9f251700 5 -- op tracker -- , seq: 85776, time: 2014-09-18 18:08:27.687842, event: all_read, op: pg_query(2.280 epoch 11706) -7405> 2014-09-18 18:08:27.687885 7f4c9f251700 5 -- op tracker -- , seq: 85776, time: 2014-09-18 18:08:27.687860, event: dispatched, op: pg_query(2.280 epoch 11706) -7404> 2014-09-18 18:08:27.687892 7f4c9f251700 5 -- op tracker -- , seq: 85776, time: 2014-09-18 18:08:27.687890, event: waiting_for_osdmap, op: pg_query(2.280 epoch 11706) -7403> 2014-09-18 18:08:27.687900 7f4c9f251700 5 -- op tracker -- , seq: 85776, time: 2014-09-18 18:08:27.687899, event: started, op: pg_query(2.280 epoch 11706) -7402> 2014-09-18 18:08:27.687950 7f4c9f251700 5 -- op tracker -- , seq: 85776, time: 2014-09-18 18:08:27.687949, event: done, op: pg_query(2.280 epoch 11706) -7401> 2014-09-18 18:08:27.688235 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- pg_notify(2.280(6) epoch 11706) v5 -- ?+0 0x21f052c0 con 0xdd8c940 -7400> 2014-09-18 18:08:27.689149 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3614 ==== pg_notify(2.3b6(10) epoch 11706) v5 ==== 1260+0+0 (515542041 0 0) 0xe1ad2a0 con 0xcbecec0 -7399> 2014-09-18 18:08:27.689169 7f4c9f251700 5 -- op tracker -- , seq: 85777, time: 2014-09-18 18:08:27.689090, event: header_read, op: pg_notify(2.3b6(10) epoch 11706) -7398> 2014-09-18 18:08:27.689182 7f4c9f251700 5 -- op tracker -- , seq: 85777, time: 2014-09-18 18:08:27.689091, event: throttled, op: pg_notify(2.3b6(10) epoch 11706) -7397> 2014-09-18 18:08:27.689193 7f4c9f251700 5 -- op tracker -- , seq: 85777, time: 2014-09-18 18:08:27.689133, event: all_read, op: pg_notify(2.3b6(10) epoch 11706) -7396> 2014-09-18 18:08:27.689201 7f4c9f251700 5 -- op tracker -- , seq: 85777, time: 2014-09-18 18:08:27.689164, event: dispatched, op: pg_notify(2.3b6(10) epoch 11706) -7395> 2014-09-18 18:08:27.689206 7f4c9f251700 5 -- op tracker -- , seq: 85777, time: 2014-09-18 18:08:27.689204, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11706) -7394> 2014-09-18 18:08:27.689212 7f4c9f251700 5 -- op tracker -- , seq: 85777, time: 2014-09-18 18:08:27.689211, event: started, op: pg_notify(2.3b6(10) epoch 11706) -7393> 2014-09-18 18:08:27.689234 7f4c9f251700 5 -- op tracker -- , seq: 85777, time: 2014-09-18 18:08:27.689233, event: done, op: pg_notify(2.3b6(10) epoch 11706) -7392> 2014-09-18 18:08:27.689799 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2345 ==== pg_notify(2.2c7(139) epoch 11706) v5 ==== 8651+0+0 (717085715 0 0) 0x21f052c0 con 0xdd8c940 -7391> 2014-09-18 18:08:27.689816 7f4c9f251700 5 -- op tracker -- , seq: 85778, time: 2014-09-18 18:08:27.689676, event: header_read, op: pg_notify(2.2c7(139) epoch 11706) -7390> 2014-09-18 18:08:27.689826 7f4c9f251700 5 -- op tracker -- , seq: 85778, time: 2014-09-18 18:08:27.689676, event: throttled, op: pg_notify(2.2c7(139) epoch 11706) -7389> 2014-09-18 18:08:27.689864 7f4c9f251700 5 -- op tracker -- , seq: 85778, time: 2014-09-18 18:08:27.689783, event: all_read, op: pg_notify(2.2c7(139) epoch 11706) -7388> 2014-09-18 18:08:27.689870 7f4c9f251700 5 -- op tracker -- , seq: 85778, time: 2014-09-18 18:08:27.689812, event: dispatched, op: pg_notify(2.2c7(139) epoch 11706) -7387> 2014-09-18 18:08:27.689877 7f4c9f251700 5 -- op tracker -- , seq: 85778, time: 2014-09-18 18:08:27.689875, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11706) -7386> 2014-09-18 18:08:27.689884 7f4c9f251700 5 -- op tracker -- , seq: 85778, time: 2014-09-18 18:08:27.689883, event: started, op: pg_notify(2.2c7(139) epoch 11706) -7385> 2014-09-18 18:08:27.690110 7f4c9f251700 5 -- op tracker -- , seq: 85778, time: 2014-09-18 18:08:27.690108, event: done, op: pg_notify(2.2c7(139) epoch 11706) -7384> 2014-09-18 18:08:27.690130 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2005 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x11175600 con 0xe2d9a20 -7383> 2014-09-18 18:08:27.690142 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7382> 2014-09-18 18:08:27.690148 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2346 ==== pg_notify(2.457(21) epoch 11706) v5 ==== 1875+0+0 (1691327144 0 0) 0xf2b8580 con 0xdd8c940 -7381> 2014-09-18 18:08:27.690157 7f4c9f251700 5 -- op tracker -- , seq: 85779, time: 2014-09-18 18:08:27.690079, event: header_read, op: pg_notify(2.457(21) epoch 11706) -7380> 2014-09-18 18:08:27.690165 7f4c9f251700 5 -- op tracker -- , seq: 85779, time: 2014-09-18 18:08:27.690079, event: throttled, op: pg_notify(2.457(21) epoch 11706) -7379> 2014-09-18 18:08:27.690171 7f4c9f251700 5 -- op tracker -- , seq: 85779, time: 2014-09-18 18:08:27.690114, event: all_read, op: pg_notify(2.457(21) epoch 11706) -7378> 2014-09-18 18:08:27.690176 7f4c9f251700 5 -- op tracker -- , seq: 85779, time: 2014-09-18 18:08:27.690155, event: dispatched, op: pg_notify(2.457(21) epoch 11706) -7377> 2014-09-18 18:08:27.690182 7f4c9f251700 5 -- op tracker -- , seq: 85779, time: 2014-09-18 18:08:27.690181, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11706) -7376> 2014-09-18 18:08:27.690190 7f4c9f251700 5 -- op tracker -- , seq: 85779, time: 2014-09-18 18:08:27.690188, event: started, op: pg_notify(2.457(21) epoch 11706) -7375> 2014-09-18 18:08:27.690223 7f4c9f251700 5 -- op tracker -- , seq: 85779, time: 2014-09-18 18:08:27.690221, event: done, op: pg_notify(2.457(21) epoch 11706) -7374> 2014-09-18 18:08:27.690235 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2006 ==== pg_notify(2.26d(9) epoch 11706) v5 ==== 1257+0+0 (1380773922 0 0) 0x12c12ee0 con 0xe2d9a20 -7373> 2014-09-18 18:08:27.690244 7f4c9f251700 5 -- op tracker -- , seq: 85780, time: 2014-09-18 18:08:27.689997, event: header_read, op: pg_notify(2.26d(9) epoch 11706) -7372> 2014-09-18 18:08:27.690249 7f4c9f251700 5 -- op tracker -- , seq: 85780, time: 2014-09-18 18:08:27.689997, event: throttled, op: pg_notify(2.26d(9) epoch 11706) -7371> 2014-09-18 18:08:27.690255 7f4c9f251700 5 -- op tracker -- , seq: 85780, time: 2014-09-18 18:08:27.690046, event: all_read, op: pg_notify(2.26d(9) epoch 11706) -7370> 2014-09-18 18:08:27.690261 7f4c9f251700 5 -- op tracker -- , seq: 85780, time: 2014-09-18 18:08:27.690242, event: dispatched, op: pg_notify(2.26d(9) epoch 11706) -7369> 2014-09-18 18:08:27.690267 7f4c9f251700 5 -- op tracker -- , seq: 85780, time: 2014-09-18 18:08:27.690266, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11706) -7368> 2014-09-18 18:08:27.690274 7f4c9f251700 5 -- op tracker -- , seq: 85780, time: 2014-09-18 18:08:27.690273, event: started, op: pg_notify(2.26d(9) epoch 11706) -7367> 2014-09-18 18:08:27.690290 7f4c9f251700 5 -- op tracker -- , seq: 85780, time: 2014-09-18 18:08:27.690288, event: done, op: pg_notify(2.26d(9) epoch 11706) -7366> 2014-09-18 18:08:27.691375 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3615 ==== pg_notify(2.517(11) epoch 11706) v5 ==== 1343+0+0 (699255619 0 0) 0xe1aa3c0 con 0xcbecec0 -7365> 2014-09-18 18:08:27.691399 7f4c9f251700 5 -- op tracker -- , seq: 85781, time: 2014-09-18 18:08:27.691312, event: header_read, op: pg_notify(2.517(11) epoch 11706) -7364> 2014-09-18 18:08:27.691409 7f4c9f251700 5 -- op tracker -- , seq: 85781, time: 2014-09-18 18:08:27.691313, event: throttled, op: pg_notify(2.517(11) epoch 11706) -7363> 2014-09-18 18:08:27.691415 7f4c9f251700 5 -- op tracker -- , seq: 85781, time: 2014-09-18 18:08:27.691361, event: all_read, op: pg_notify(2.517(11) epoch 11706) -7362> 2014-09-18 18:08:27.691421 7f4c9f251700 5 -- op tracker -- , seq: 85781, time: 2014-09-18 18:08:27.691396, event: dispatched, op: pg_notify(2.517(11) epoch 11706) -7361> 2014-09-18 18:08:27.691428 7f4c9f251700 5 -- op tracker -- , seq: 85781, time: 2014-09-18 18:08:27.691427, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11706) -7360> 2014-09-18 18:08:27.691436 7f4c9f251700 5 -- op tracker -- , seq: 85781, time: 2014-09-18 18:08:27.691434, event: started, op: pg_notify(2.517(11) epoch 11706) -7359> 2014-09-18 18:08:27.691555 7f4c9f251700 5 -- op tracker -- , seq: 85781, time: 2014-09-18 18:08:27.691552, event: done, op: pg_notify(2.517(11) epoch 11706) -7358> 2014-09-18 18:08:27.691568 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1717 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0xef56240 con 0x4900000 -7357> 2014-09-18 18:08:27.691578 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7356> 2014-09-18 18:08:27.691584 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 780 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x2d7fc400 con 0xe9a27e0 -7355> 2014-09-18 18:08:27.691592 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7354> 2014-09-18 18:08:27.691625 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1718 ==== pg_notify(2.e0(79) epoch 11706) v5 ==== 5119+0+0 (3335568250 0 0) 0xf1ec3a0 con 0x4900000 -7353> 2014-09-18 18:08:27.691637 7f4c9f251700 5 -- op tracker -- , seq: 85782, time: 2014-09-18 18:08:27.691523, event: header_read, op: pg_notify(2.e0(79) epoch 11706) -7352> 2014-09-18 18:08:27.691644 7f4c9f251700 5 -- op tracker -- , seq: 85782, time: 2014-09-18 18:08:27.691524, event: throttled, op: pg_notify(2.e0(79) epoch 11706) -7351> 2014-09-18 18:08:27.691650 7f4c9f251700 5 -- op tracker -- , seq: 85782, time: 2014-09-18 18:08:27.691614, event: all_read, op: pg_notify(2.e0(79) epoch 11706) -7350> 2014-09-18 18:08:27.691656 7f4c9f251700 5 -- op tracker -- , seq: 85782, time: 2014-09-18 18:08:27.691634, event: dispatched, op: pg_notify(2.e0(79) epoch 11706) -7349> 2014-09-18 18:08:27.691662 7f4c9f251700 5 -- op tracker -- , seq: 85782, time: 2014-09-18 18:08:27.691661, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11706) -7348> 2014-09-18 18:08:27.691670 7f4c9f251700 5 -- op tracker -- , seq: 85782, time: 2014-09-18 18:08:27.691668, event: started, op: pg_notify(2.e0(79) epoch 11706) -7347> 2014-09-18 18:08:27.691685 7f4c9f251700 5 -- op tracker -- , seq: 85782, time: 2014-09-18 18:08:27.691683, event: done, op: pg_notify(2.e0(79) epoch 11706) -7346> 2014-09-18 18:08:27.691867 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 781 ==== pg_notify(2.2d8(173) epoch 11706) v5 ==== 10757+0+0 (2682059974 0 0) 0x14220940 con 0xe9a27e0 -7345> 2014-09-18 18:08:27.691879 7f4c9f251700 5 -- op tracker -- , seq: 85783, time: 2014-09-18 18:08:27.691617, event: header_read, op: pg_notify(2.2d8(173) epoch 11706) -7344> 2014-09-18 18:08:27.691895 7f4c9f251700 5 -- op tracker -- , seq: 85783, time: 2014-09-18 18:08:27.691618, event: throttled, op: pg_notify(2.2d8(173) epoch 11706) -7343> 2014-09-18 18:08:27.691901 7f4c9f251700 5 -- op tracker -- , seq: 85783, time: 2014-09-18 18:08:27.691854, event: all_read, op: pg_notify(2.2d8(173) epoch 11706) -7342> 2014-09-18 18:08:27.691907 7f4c9f251700 5 -- op tracker -- , seq: 85783, time: 2014-09-18 18:08:27.691876, event: dispatched, op: pg_notify(2.2d8(173) epoch 11706) -7341> 2014-09-18 18:08:27.691914 7f4c9f251700 5 -- op tracker -- , seq: 85783, time: 2014-09-18 18:08:27.691913, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11706) -7340> 2014-09-18 18:08:27.691922 7f4c9f251700 5 -- op tracker -- , seq: 85783, time: 2014-09-18 18:08:27.691920, event: started, op: pg_notify(2.2d8(173) epoch 11706) -7339> 2014-09-18 18:08:27.691938 7f4c9f251700 5 -- op tracker -- , seq: 85783, time: 2014-09-18 18:08:27.691936, event: done, op: pg_notify(2.2d8(173) epoch 11706) -7338> 2014-09-18 18:08:27.692338 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3282 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x12bf06c0 con 0x4900580 -7337> 2014-09-18 18:08:27.692349 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7336> 2014-09-18 18:08:27.692682 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3283 ==== pg_notify(2.4f6(184) epoch 11706) v5 ==== 11296+0+0 (2168201376 0 0) 0x19f252a0 con 0x4900580 -7335> 2014-09-18 18:08:27.692699 7f4c9f251700 5 -- op tracker -- , seq: 85784, time: 2014-09-18 18:08:27.692465, event: header_read, op: pg_notify(2.4f6(184) epoch 11706) -7334> 2014-09-18 18:08:27.692710 7f4c9f251700 5 -- op tracker -- , seq: 85784, time: 2014-09-18 18:08:27.692466, event: throttled, op: pg_notify(2.4f6(184) epoch 11706) -7333> 2014-09-18 18:08:27.692728 7f4c9f251700 5 -- op tracker -- , seq: 85784, time: 2014-09-18 18:08:27.692669, event: all_read, op: pg_notify(2.4f6(184) epoch 11706) -7332> 2014-09-18 18:08:27.692739 7f4c9f251700 5 -- op tracker -- , seq: 85784, time: 2014-09-18 18:08:27.692696, event: dispatched, op: pg_notify(2.4f6(184) epoch 11706) -7331> 2014-09-18 18:08:27.692750 7f4c9f251700 5 -- op tracker -- , seq: 85784, time: 2014-09-18 18:08:27.692748, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11706) -7330> 2014-09-18 18:08:27.692763 7f4c9f251700 5 -- op tracker -- , seq: 85784, time: 2014-09-18 18:08:27.692761, event: started, op: pg_notify(2.4f6(184) epoch 11706) -7329> 2014-09-18 18:08:27.692789 7f4c9f251700 5 -- op tracker -- , seq: 85784, time: 2014-09-18 18:08:27.692787, event: done, op: pg_notify(2.4f6(184) epoch 11706) -7328> 2014-09-18 18:08:27.692874 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3616 ==== pg_notify(2.2c7(139) epoch 11706) v5 ==== 8651+0+0 (3943310328 0 0) 0xe1abe00 con 0xcbecec0 -7327> 2014-09-18 18:08:27.692891 7f4c9f251700 5 -- op tracker -- , seq: 85785, time: 2014-09-18 18:08:27.692672, event: header_read, op: pg_notify(2.2c7(139) epoch 11706) -7326> 2014-09-18 18:08:27.692899 7f4c9f251700 5 -- op tracker -- , seq: 85785, time: 2014-09-18 18:08:27.692673, event: throttled, op: pg_notify(2.2c7(139) epoch 11706) -7325> 2014-09-18 18:08:27.692905 7f4c9f251700 5 -- op tracker -- , seq: 85785, time: 2014-09-18 18:08:27.692814, event: all_read, op: pg_notify(2.2c7(139) epoch 11706) -7324> 2014-09-18 18:08:27.692911 7f4c9f251700 5 -- op tracker -- , seq: 85785, time: 2014-09-18 18:08:27.692887, event: dispatched, op: pg_notify(2.2c7(139) epoch 11706) -7323> 2014-09-18 18:08:27.692917 7f4c9f251700 5 -- op tracker -- , seq: 85785, time: 2014-09-18 18:08:27.692916, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11706) -7322> 2014-09-18 18:08:27.692927 7f4c9f251700 5 -- op tracker -- , seq: 85785, time: 2014-09-18 18:08:27.692924, event: started, op: pg_notify(2.2c7(139) epoch 11706) -7321> 2014-09-18 18:08:27.692953 7f4c9f251700 5 -- op tracker -- , seq: 85785, time: 2014-09-18 18:08:27.692950, event: done, op: pg_notify(2.2c7(139) epoch 11706) -7320> 2014-09-18 18:08:27.693636 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3284 ==== pg_notify(2.17e(17) epoch 11706) v5 ==== 1593+0+0 (3989242314 0 0) 0x19f23a40 con 0x4900580 -7319> 2014-09-18 18:08:27.693651 7f4c9f251700 5 -- op tracker -- , seq: 85786, time: 2014-09-18 18:08:27.693591, event: header_read, op: pg_notify(2.17e(17) epoch 11706) -7318> 2014-09-18 18:08:27.693660 7f4c9f251700 5 -- op tracker -- , seq: 85786, time: 2014-09-18 18:08:27.693592, event: throttled, op: pg_notify(2.17e(17) epoch 11706) -7317> 2014-09-18 18:08:27.693666 7f4c9f251700 5 -- op tracker -- , seq: 85786, time: 2014-09-18 18:08:27.693624, event: all_read, op: pg_notify(2.17e(17) epoch 11706) -7316> 2014-09-18 18:08:27.693671 7f4c9f251700 5 -- op tracker -- , seq: 85786, time: 2014-09-18 18:08:27.693647, event: dispatched, op: pg_notify(2.17e(17) epoch 11706) -7315> 2014-09-18 18:08:27.693686 7f4c9f251700 5 -- op tracker -- , seq: 85786, time: 2014-09-18 18:08:27.693684, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11706) -7314> 2014-09-18 18:08:27.693695 7f4c9f251700 5 -- op tracker -- , seq: 85786, time: 2014-09-18 18:08:27.693693, event: started, op: pg_notify(2.17e(17) epoch 11706) -7313> 2014-09-18 18:08:27.693711 7f4c9f251700 5 -- op tracker -- , seq: 85786, time: 2014-09-18 18:08:27.693710, event: done, op: pg_notify(2.17e(17) epoch 11706) -7312> 2014-09-18 18:08:27.694530 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3617 ==== pg_notify(2.40a(162) epoch 11706) v5 ==== 9736+0+0 (3195569035 0 0) 0xe1ab680 con 0xcbecec0 -7311> 2014-09-18 18:08:27.694558 7f4c9f251700 5 -- op tracker -- , seq: 85787, time: 2014-09-18 18:08:27.694370, event: header_read, op: pg_notify(2.40a(162) epoch 11706) -7310> 2014-09-18 18:08:27.694570 7f4c9f251700 5 -- op tracker -- , seq: 85787, time: 2014-09-18 18:08:27.694371, event: throttled, op: pg_notify(2.40a(162) epoch 11706) -7309> 2014-09-18 18:08:27.694580 7f4c9f251700 5 -- op tracker -- , seq: 85787, time: 2014-09-18 18:08:27.694515, event: all_read, op: pg_notify(2.40a(162) epoch 11706) -7308> 2014-09-18 18:08:27.694591 7f4c9f251700 5 -- op tracker -- , seq: 85787, time: 2014-09-18 18:08:27.694553, event: dispatched, op: pg_notify(2.40a(162) epoch 11706) -7307> 2014-09-18 18:08:27.694601 7f4c9f251700 5 -- op tracker -- , seq: 85787, time: 2014-09-18 18:08:27.694599, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11706) -7306> 2014-09-18 18:08:27.694607 7f4c9f251700 5 -- op tracker -- , seq: 85787, time: 2014-09-18 18:08:27.694605, event: started, op: pg_notify(2.40a(162) epoch 11706) -7305> 2014-09-18 18:08:27.694624 7f4c9f251700 5 -- op tracker -- , seq: 85787, time: 2014-09-18 18:08:27.694623, event: done, op: pg_notify(2.40a(162) epoch 11706) -7304> 2014-09-18 18:08:27.695013 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3285 ==== pg_notify(2.1bb(9) epoch 11706) v5 ==== 1189+0+0 (3623597211 0 0) 0x19f25de0 con 0x4900580 -7303> 2014-09-18 18:08:27.695026 7f4c9f251700 5 -- op tracker -- , seq: 85788, time: 2014-09-18 18:08:27.694968, event: header_read, op: pg_notify(2.1bb(9) epoch 11706) -7302> 2014-09-18 18:08:27.695034 7f4c9f251700 5 -- op tracker -- , seq: 85788, time: 2014-09-18 18:08:27.694969, event: throttled, op: pg_notify(2.1bb(9) epoch 11706) -7301> 2014-09-18 18:08:27.695040 7f4c9f251700 5 -- op tracker -- , seq: 85788, time: 2014-09-18 18:08:27.695001, event: all_read, op: pg_notify(2.1bb(9) epoch 11706) -7300> 2014-09-18 18:08:27.695046 7f4c9f251700 5 -- op tracker -- , seq: 85788, time: 2014-09-18 18:08:27.695023, event: dispatched, op: pg_notify(2.1bb(9) epoch 11706) -7299> 2014-09-18 18:08:27.695052 7f4c9f251700 5 -- op tracker -- , seq: 85788, time: 2014-09-18 18:08:27.695051, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11706) -7298> 2014-09-18 18:08:27.695060 7f4c9f251700 5 -- op tracker -- , seq: 85788, time: 2014-09-18 18:08:27.695058, event: started, op: pg_notify(2.1bb(9) epoch 11706) -7297> 2014-09-18 18:08:27.695078 7f4c9f251700 5 -- op tracker -- , seq: 85788, time: 2014-09-18 18:08:27.695076, event: done, op: pg_notify(2.1bb(9) epoch 11706) -7296> 2014-09-18 18:08:27.695482 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3618 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11706) v5 ==== 6174+0+0 (3425406985 0 0) 0xe1aad20 con 0xcbecec0 -7295> 2014-09-18 18:08:27.695495 7f4c9f251700 5 -- op tracker -- , seq: 85789, time: 2014-09-18 18:08:27.695409, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11706) -7294> 2014-09-18 18:08:27.695503 7f4c9f251700 5 -- op tracker -- , seq: 85789, time: 2014-09-18 18:08:27.695410, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11706) -7293> 2014-09-18 18:08:27.695509 7f4c9f251700 5 -- op tracker -- , seq: 85789, time: 2014-09-18 18:08:27.695470, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11706) -7292> 2014-09-18 18:08:27.695516 7f4c9f251700 5 -- op tracker -- , seq: 85789, time: 2014-09-18 18:08:27.695491, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11706) -7291> 2014-09-18 18:08:27.695523 7f4c9f251700 5 -- op tracker -- , seq: 85789, time: 2014-09-18 18:08:27.695521, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11706) -7290> 2014-09-18 18:08:27.695535 7f4c9f251700 5 -- op tracker -- , seq: 85789, time: 2014-09-18 18:08:27.695531, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11706) -7289> 2014-09-18 18:08:27.695598 7f4c9f251700 5 -- op tracker -- , seq: 85789, time: 2014-09-18 18:08:27.695595, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11706) -7288> 2014-09-18 18:08:27.696691 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3619 ==== pg_notify(2.1bb(9) epoch 11706) v5 ==== 1189+0+0 (2071070344 0 0) 0xe1acee0 con 0xcbecec0 -7287> 2014-09-18 18:08:27.696705 7f4c9f251700 5 -- op tracker -- , seq: 85790, time: 2014-09-18 18:08:27.696653, event: header_read, op: pg_notify(2.1bb(9) epoch 11706) -7286> 2014-09-18 18:08:27.696713 7f4c9f251700 5 -- op tracker -- , seq: 85790, time: 2014-09-18 18:08:27.696654, event: throttled, op: pg_notify(2.1bb(9) epoch 11706) -7285> 2014-09-18 18:08:27.696720 7f4c9f251700 5 -- op tracker -- , seq: 85790, time: 2014-09-18 18:08:27.696681, event: all_read, op: pg_notify(2.1bb(9) epoch 11706) -7284> 2014-09-18 18:08:27.696726 7f4c9f251700 5 -- op tracker -- , seq: 85790, time: 2014-09-18 18:08:27.696702, event: dispatched, op: pg_notify(2.1bb(9) epoch 11706) -7283> 2014-09-18 18:08:27.696732 7f4c9f251700 5 -- op tracker -- , seq: 85790, time: 2014-09-18 18:08:27.696731, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11706) -7282> 2014-09-18 18:08:27.696740 7f4c9f251700 5 -- op tracker -- , seq: 85790, time: 2014-09-18 18:08:27.696738, event: started, op: pg_notify(2.1bb(9) epoch 11706) -7281> 2014-09-18 18:08:27.696765 7f4c9f251700 5 -- op tracker -- , seq: 85790, time: 2014-09-18 18:08:27.696763, event: done, op: pg_notify(2.1bb(9) epoch 11706) -7280> 2014-09-18 18:08:27.714851 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.0 10.242.42.164:0/15878 159 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:27.714168) v2 ==== 47+0+0 (1640515602 0 0) 0xde60760 con 0x142e8260 -7279> 2014-09-18 18:08:27.714857 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.0 10.242.42.164:0/15878 159 ==== osd_ping(ping e11704 stamp 2014-09-18 18:08:27.714168) v2 ==== 47+0+0 (1640515602 0 0) 0x12139680 con 0x1d5bc7e0 -7278> 2014-09-18 18:08:27.714883 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:27.714168) v2 -- ?+0 0xe288580 con 0x142e8260 -7277> 2014-09-18 18:08:27.714908 7f4c9ca4c700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- osd_map(11705..11706 src has 1..11706) v3 -- ?+0 0x11bb81c0 con 0x10a878c0 -7276> 2014-09-18 18:08:27.714934 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:27.714168) v2 -- ?+0 0xe7745a0 con 0x1d5bc7e0 -7275> 2014-09-18 18:08:27.717388 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.32 10.242.42.176:0/1099 111 ==== osd_ping(ping e11706 stamp 2014-09-18 23:37:32.465292) v2 ==== 47+0+0 (159760691 0 0) 0xe1a21e0 con 0x110bb2e0 -7274> 2014-09-18 18:08:27.717412 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/1099 -- osd_ping(ping_reply e11706 stamp 2014-09-18 23:37:32.465292) v2 -- ?+0 0xde60760 con 0x110bb2e0 -7273> 2014-09-18 18:08:27.717452 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.32 10.242.42.176:0/1099 111 ==== osd_ping(ping e11706 stamp 2014-09-18 23:37:32.465292) v2 ==== 47+0+0 (159760691 0 0) 0x2b92fe00 con 0x12abd080 -7272> 2014-09-18 18:08:27.717484 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/1099 -- osd_ping(ping_reply e11706 stamp 2014-09-18 23:37:32.465292) v2 -- ?+0 0x12139680 con 0x12abd080 -7271> 2014-09-18 18:08:27.734152 7f4caae41700 5 osd.42 11706 tick -7270> 2014-09-18 18:08:27.754765 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1190 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x1182a900 con 0x1088b440 -7269> 2014-09-18 18:08:27.754795 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7268> 2014-09-18 18:08:27.754950 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1191 ==== pg_notify(2.2d8(173) epoch 11706) v5 ==== 10757+0+0 (2682059974 0 0) 0x1390b680 con 0x1088b440 -7267> 2014-09-18 18:08:27.754968 7f4c9f251700 5 -- op tracker -- , seq: 85791, time: 2014-09-18 18:08:27.754811, event: header_read, op: pg_notify(2.2d8(173) epoch 11706) -7266> 2014-09-18 18:08:27.754982 7f4c9f251700 5 -- op tracker -- , seq: 85791, time: 2014-09-18 18:08:27.754812, event: throttled, op: pg_notify(2.2d8(173) epoch 11706) -7265> 2014-09-18 18:08:27.754992 7f4c9f251700 5 -- op tracker -- , seq: 85791, time: 2014-09-18 18:08:27.754937, event: all_read, op: pg_notify(2.2d8(173) epoch 11706) -7264> 2014-09-18 18:08:27.755001 7f4c9f251700 5 -- op tracker -- , seq: 85791, time: 2014-09-18 18:08:27.754962, event: dispatched, op: pg_notify(2.2d8(173) epoch 11706) -7263> 2014-09-18 18:08:27.755010 7f4c9f251700 5 -- op tracker -- , seq: 85791, time: 2014-09-18 18:08:27.755008, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11706) -7262> 2014-09-18 18:08:27.755023 7f4c9f251700 5 -- op tracker -- , seq: 85791, time: 2014-09-18 18:08:27.755020, event: started, op: pg_notify(2.2d8(173) epoch 11706) -7261> 2014-09-18 18:08:27.755052 7f4c9f251700 5 -- op tracker -- , seq: 85791, time: 2014-09-18 18:08:27.755049, event: done, op: pg_notify(2.2d8(173) epoch 11706) -7260> 2014-09-18 18:08:27.764141 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 36 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0xf954640 con 0x1395f340 -7259> 2014-09-18 18:08:27.764163 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7258> 2014-09-18 18:08:27.764185 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 37 ==== pg_notify(2.1d8(9) epoch 11706) v5 ==== 1261+0+0 (3027784763 0 0) 0xf344b20 con 0x1395f340 -7257> 2014-09-18 18:08:27.764205 7f4c9f251700 5 -- op tracker -- , seq: 85792, time: 2014-09-18 18:08:27.764136, event: header_read, op: pg_notify(2.1d8(9) epoch 11706) -7256> 2014-09-18 18:08:27.764220 7f4c9f251700 5 -- op tracker -- , seq: 85792, time: 2014-09-18 18:08:27.764137, event: throttled, op: pg_notify(2.1d8(9) epoch 11706) -7255> 2014-09-18 18:08:27.764232 7f4c9f251700 5 -- op tracker -- , seq: 85792, time: 2014-09-18 18:08:27.764173, event: all_read, op: pg_notify(2.1d8(9) epoch 11706) -7254> 2014-09-18 18:08:27.764243 7f4c9f251700 5 -- op tracker -- , seq: 85792, time: 2014-09-18 18:08:27.764199, event: dispatched, op: pg_notify(2.1d8(9) epoch 11706) -7253> 2014-09-18 18:08:27.764254 7f4c9f251700 5 -- op tracker -- , seq: 85792, time: 2014-09-18 18:08:27.764251, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11706) -7252> 2014-09-18 18:08:27.764267 7f4c9f251700 5 -- op tracker -- , seq: 85792, time: 2014-09-18 18:08:27.764264, event: started, op: pg_notify(2.1d8(9) epoch 11706) -7251> 2014-09-18 18:08:27.764295 7f4c9f251700 5 -- op tracker -- , seq: 85792, time: 2014-09-18 18:08:27.764292, event: done, op: pg_notify(2.1d8(9) epoch 11706) -7250> 2014-09-18 18:08:27.822801 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3164 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x1182b8c0 con 0x1395ec60 -7249> 2014-09-18 18:08:27.822831 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7248> 2014-09-18 18:08:27.822845 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3165 ==== pg_notify(2.71(19) epoch 11706) v5 ==== 1795+0+0 (4027614831 0 0) 0xf8112a0 con 0x1395ec60 -7247> 2014-09-18 18:08:27.822855 7f4c9f251700 5 -- op tracker -- , seq: 85793, time: 2014-09-18 18:08:27.822794, event: header_read, op: pg_notify(2.71(19) epoch 11706) -7246> 2014-09-18 18:08:27.822877 7f4c9f251700 5 -- op tracker -- , seq: 85793, time: 2014-09-18 18:08:27.822794, event: throttled, op: pg_notify(2.71(19) epoch 11706) -7245> 2014-09-18 18:08:27.822886 7f4c9f251700 5 -- op tracker -- , seq: 85793, time: 2014-09-18 18:08:27.822834, event: all_read, op: pg_notify(2.71(19) epoch 11706) -7244> 2014-09-18 18:08:27.822890 7f4c9f251700 5 -- op tracker -- , seq: 85793, time: 2014-09-18 18:08:27.822850, event: dispatched, op: pg_notify(2.71(19) epoch 11706) -7243> 2014-09-18 18:08:27.822895 7f4c9f251700 5 -- op tracker -- , seq: 85793, time: 2014-09-18 18:08:27.822894, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11706) -7242> 2014-09-18 18:08:27.822902 7f4c9f251700 5 -- op tracker -- , seq: 85793, time: 2014-09-18 18:08:27.822901, event: started, op: pg_notify(2.71(19) epoch 11706) -7241> 2014-09-18 18:08:27.822924 7f4c9f251700 5 -- op tracker -- , seq: 85793, time: 2014-09-18 18:08:27.822923, event: done, op: pg_notify(2.71(19) epoch 11706) -7240> 2014-09-18 18:08:27.829840 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3166 ==== pg_notify(2.410(11) epoch 11706) v5 ==== 1325+0+0 (1078600588 0 0) 0xf8101c0 con 0x1395ec60 -7239> 2014-09-18 18:08:27.829860 7f4c9f251700 5 -- op tracker -- , seq: 85794, time: 2014-09-18 18:08:27.829702, event: header_read, op: pg_notify(2.410(11) epoch 11706) -7238> 2014-09-18 18:08:27.829871 7f4c9f251700 5 -- op tracker -- , seq: 85794, time: 2014-09-18 18:08:27.829703, event: throttled, op: pg_notify(2.410(11) epoch 11706) -7237> 2014-09-18 18:08:27.829877 7f4c9f251700 5 -- op tracker -- , seq: 85794, time: 2014-09-18 18:08:27.829738, event: all_read, op: pg_notify(2.410(11) epoch 11706) -7236> 2014-09-18 18:08:27.829883 7f4c9f251700 5 -- op tracker -- , seq: 85794, time: 2014-09-18 18:08:27.829856, event: dispatched, op: pg_notify(2.410(11) epoch 11706) -7235> 2014-09-18 18:08:27.829889 7f4c9f251700 5 -- op tracker -- , seq: 85794, time: 2014-09-18 18:08:27.829888, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11706) -7234> 2014-09-18 18:08:27.829897 7f4c9f251700 5 -- op tracker -- , seq: 85794, time: 2014-09-18 18:08:27.829896, event: started, op: pg_notify(2.410(11) epoch 11706) -7233> 2014-09-18 18:08:27.829923 7f4c9f251700 5 -- op tracker -- , seq: 85794, time: 2014-09-18 18:08:27.829921, event: done, op: pg_notify(2.410(11) epoch 11706) -7232> 2014-09-18 18:08:27.830771 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3167 ==== pg_notify(2.186(12) epoch 11706) v5 ==== 1386+0+0 (4207267145 0 0) 0xf811840 con 0x1395ec60 -7231> 2014-09-18 18:08:27.830791 7f4c9f251700 5 -- op tracker -- , seq: 85795, time: 2014-09-18 18:08:27.830615, event: header_read, op: pg_notify(2.186(12) epoch 11706) -7230> 2014-09-18 18:08:27.830801 7f4c9f251700 5 -- op tracker -- , seq: 85795, time: 2014-09-18 18:08:27.830616, event: throttled, op: pg_notify(2.186(12) epoch 11706) -7229> 2014-09-18 18:08:27.830810 7f4c9f251700 5 -- op tracker -- , seq: 85795, time: 2014-09-18 18:08:27.830667, event: all_read, op: pg_notify(2.186(12) epoch 11706) -7228> 2014-09-18 18:08:27.830816 7f4c9f251700 5 -- op tracker -- , seq: 85795, time: 2014-09-18 18:08:27.830786, event: dispatched, op: pg_notify(2.186(12) epoch 11706) -7227> 2014-09-18 18:08:27.830823 7f4c9f251700 5 -- op tracker -- , seq: 85795, time: 2014-09-18 18:08:27.830821, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11706) -7226> 2014-09-18 18:08:27.830831 7f4c9f251700 5 -- op tracker -- , seq: 85795, time: 2014-09-18 18:08:27.830829, event: started, op: pg_notify(2.186(12) epoch 11706) -7225> 2014-09-18 18:08:27.830850 7f4c9f251700 5 -- op tracker -- , seq: 85795, time: 2014-09-18 18:08:27.830849, event: done, op: pg_notify(2.186(12) epoch 11706) -7224> 2014-09-18 18:08:27.898641 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3168 ==== pg_notify(2.74e(11) epoch 11706) v5 ==== 1303+0+0 (2269285190 0 0) 0xf811de0 con 0x1395ec60 -7223> 2014-09-18 18:08:27.898671 7f4c9f251700 5 -- op tracker -- , seq: 85796, time: 2014-09-18 18:08:27.898563, event: header_read, op: pg_notify(2.74e(11) epoch 11706) -7222> 2014-09-18 18:08:27.898683 7f4c9f251700 5 -- op tracker -- , seq: 85796, time: 2014-09-18 18:08:27.898566, event: throttled, op: pg_notify(2.74e(11) epoch 11706) -7221> 2014-09-18 18:08:27.898692 7f4c9f251700 5 -- op tracker -- , seq: 85796, time: 2014-09-18 18:08:27.898621, event: all_read, op: pg_notify(2.74e(11) epoch 11706) -7220> 2014-09-18 18:08:27.898699 7f4c9f251700 5 -- op tracker -- , seq: 85796, time: 2014-09-18 18:08:27.898664, event: dispatched, op: pg_notify(2.74e(11) epoch 11706) -7219> 2014-09-18 18:08:27.898708 7f4c9f251700 5 -- op tracker -- , seq: 85796, time: 2014-09-18 18:08:27.898707, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11706) -7218> 2014-09-18 18:08:27.898718 7f4c9f251700 5 -- op tracker -- , seq: 85796, time: 2014-09-18 18:08:27.898717, event: started, op: pg_notify(2.74e(11) epoch 11706) -7217> 2014-09-18 18:08:27.898741 7f4c9f251700 5 -- op tracker -- , seq: 85796, time: 2014-09-18 18:08:27.898740, event: done, op: pg_notify(2.74e(11) epoch 11706) -7216> 2014-09-18 18:08:27.923343 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 38 ==== pg_info(1 pgs e11706:2.29c) v4 ==== 738+0+0 (4182781027 0 0) 0xf09c580 con 0x1395f340 -7215> 2014-09-18 18:08:27.923384 7f4c9f251700 5 -- op tracker -- , seq: 85797, time: 2014-09-18 18:08:27.923237, event: header_read, op: pg_info(1 pgs e11706:2.29c) -7214> 2014-09-18 18:08:27.923400 7f4c9f251700 5 -- op tracker -- , seq: 85797, time: 2014-09-18 18:08:27.923240, event: throttled, op: pg_info(1 pgs e11706:2.29c) -7213> 2014-09-18 18:08:27.923411 7f4c9f251700 5 -- op tracker -- , seq: 85797, time: 2014-09-18 18:08:27.923295, event: all_read, op: pg_info(1 pgs e11706:2.29c) -7212> 2014-09-18 18:08:27.923418 7f4c9f251700 5 -- op tracker -- , seq: 85797, time: 2014-09-18 18:08:27.923376, event: dispatched, op: pg_info(1 pgs e11706:2.29c) -7211> 2014-09-18 18:08:27.923426 7f4c9f251700 5 -- op tracker -- , seq: 85797, time: 2014-09-18 18:08:27.923424, event: waiting_for_osdmap, op: pg_info(1 pgs e11706:2.29c) -7210> 2014-09-18 18:08:27.923437 7f4c9f251700 5 -- op tracker -- , seq: 85797, time: 2014-09-18 18:08:27.923436, event: started, op: pg_info(1 pgs e11706:2.29c) -7209> 2014-09-18 18:08:27.923465 7f4c9f251700 5 -- op tracker -- , seq: 85797, time: 2014-09-18 18:08:27.923464, event: done, op: pg_info(1 pgs e11706:2.29c) -7208> 2014-09-18 18:08:27.944011 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 954 ==== pg_notify(2.517(11) epoch 11705) v5 ==== 1335+0+0 (2078192756 0 0) 0x16251480 con 0x10a878c0 -7207> 2014-09-18 18:08:27.944031 7f4c9f251700 5 -- op tracker -- , seq: 85798, time: 2014-09-18 18:08:27.943917, event: header_read, op: pg_notify(2.517(11) epoch 11705) -7206> 2014-09-18 18:08:27.944043 7f4c9f251700 5 -- op tracker -- , seq: 85798, time: 2014-09-18 18:08:27.943918, event: throttled, op: pg_notify(2.517(11) epoch 11705) -7205> 2014-09-18 18:08:27.944050 7f4c9f251700 5 -- op tracker -- , seq: 85798, time: 2014-09-18 18:08:27.943972, event: all_read, op: pg_notify(2.517(11) epoch 11705) -7204> 2014-09-18 18:08:27.944056 7f4c9f251700 5 -- op tracker -- , seq: 85798, time: 2014-09-18 18:08:27.944026, event: dispatched, op: pg_notify(2.517(11) epoch 11705) -7203> 2014-09-18 18:08:27.944063 7f4c9f251700 5 -- op tracker -- , seq: 85798, time: 2014-09-18 18:08:27.944061, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11705) -7202> 2014-09-18 18:08:27.944071 7f4c9f251700 5 -- op tracker -- , seq: 85798, time: 2014-09-18 18:08:27.944069, event: started, op: pg_notify(2.517(11) epoch 11705) -7201> 2014-09-18 18:08:27.944091 7f4c9f251700 5 -- op tracker -- , seq: 85798, time: 2014-09-18 18:08:27.944090, event: done, op: pg_notify(2.517(11) epoch 11705) -7200> 2014-09-18 18:08:27.954631 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1834 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x1182d600 con 0x10f60520 -7199> 2014-09-18 18:08:27.954656 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11705,11705], i have 11706, src has [1,11705] -7198> 2014-09-18 18:08:27.954694 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1835 ==== pg_notify(2.4f6(15) epoch 11705) v5 ==== 1581+0+0 (2754727393 0 0) 0x2b927fe0 con 0x10f60520 -7197> 2014-09-18 18:08:27.954708 7f4c9f251700 5 -- op tracker -- , seq: 85799, time: 2014-09-18 18:08:27.954634, event: header_read, op: pg_notify(2.4f6(15) epoch 11705) -7196> 2014-09-18 18:08:27.954719 7f4c9f251700 5 -- op tracker -- , seq: 85799, time: 2014-09-18 18:08:27.954634, event: throttled, op: pg_notify(2.4f6(15) epoch 11705) -7195> 2014-09-18 18:08:27.954726 7f4c9f251700 5 -- op tracker -- , seq: 85799, time: 2014-09-18 18:08:27.954683, event: all_read, op: pg_notify(2.4f6(15) epoch 11705) -7194> 2014-09-18 18:08:27.954732 7f4c9f251700 5 -- op tracker -- , seq: 85799, time: 2014-09-18 18:08:27.954704, event: dispatched, op: pg_notify(2.4f6(15) epoch 11705) -7193> 2014-09-18 18:08:27.954738 7f4c9f251700 5 -- op tracker -- , seq: 85799, time: 2014-09-18 18:08:27.954737, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11705) -7192> 2014-09-18 18:08:27.954747 7f4c9f251700 5 -- op tracker -- , seq: 85799, time: 2014-09-18 18:08:27.954745, event: started, op: pg_notify(2.4f6(15) epoch 11705) -7191> 2014-09-18 18:08:27.954768 7f4c9f251700 5 -- op tracker -- , seq: 85799, time: 2014-09-18 18:08:27.954767, event: done, op: pg_notify(2.4f6(15) epoch 11705) -7190> 2014-09-18 18:08:27.970069 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1836 ==== pg_notify(2.517(11) epoch 11705) v5 ==== 1335+0+0 (832212311 0 0) 0xe777480 con 0x10f60520 -7189> 2014-09-18 18:08:27.970098 7f4c9f251700 5 -- op tracker -- , seq: 85800, time: 2014-09-18 18:08:27.969990, event: header_read, op: pg_notify(2.517(11) epoch 11705) -7188> 2014-09-18 18:08:27.970110 7f4c9f251700 5 -- op tracker -- , seq: 85800, time: 2014-09-18 18:08:27.969993, event: throttled, op: pg_notify(2.517(11) epoch 11705) -7187> 2014-09-18 18:08:27.970117 7f4c9f251700 5 -- op tracker -- , seq: 85800, time: 2014-09-18 18:08:27.970052, event: all_read, op: pg_notify(2.517(11) epoch 11705) -7186> 2014-09-18 18:08:27.970123 7f4c9f251700 5 -- op tracker -- , seq: 85800, time: 2014-09-18 18:08:27.970091, event: dispatched, op: pg_notify(2.517(11) epoch 11705) -7185> 2014-09-18 18:08:27.970130 7f4c9f251700 5 -- op tracker -- , seq: 85800, time: 2014-09-18 18:08:27.970128, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11705) -7184> 2014-09-18 18:08:27.970139 7f4c9f251700 5 -- op tracker -- , seq: 85800, time: 2014-09-18 18:08:27.970137, event: started, op: pg_notify(2.517(11) epoch 11705) -7183> 2014-09-18 18:08:27.970171 7f4c9f251700 5 -- op tracker -- , seq: 85800, time: 2014-09-18 18:08:27.970169, event: done, op: pg_notify(2.517(11) epoch 11705) -7182> 2014-09-18 18:08:27.970861 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1837 ==== pg_notify(2.1d8(9) epoch 11705) v5 ==== 1261+0+0 (3551314781 0 0) 0xe7770c0 con 0x10f60520 -7181> 2014-09-18 18:08:27.970879 7f4c9f251700 5 -- op tracker -- , seq: 85801, time: 2014-09-18 18:08:27.970787, event: header_read, op: pg_notify(2.1d8(9) epoch 11705) -7180> 2014-09-18 18:08:27.970890 7f4c9f251700 5 -- op tracker -- , seq: 85801, time: 2014-09-18 18:08:27.970787, event: throttled, op: pg_notify(2.1d8(9) epoch 11705) -7179> 2014-09-18 18:08:27.970896 7f4c9f251700 5 -- op tracker -- , seq: 85801, time: 2014-09-18 18:08:27.970833, event: all_read, op: pg_notify(2.1d8(9) epoch 11705) -7178> 2014-09-18 18:08:27.970902 7f4c9f251700 5 -- op tracker -- , seq: 85801, time: 2014-09-18 18:08:27.970875, event: dispatched, op: pg_notify(2.1d8(9) epoch 11705) -7177> 2014-09-18 18:08:27.970915 7f4c9f251700 5 -- op tracker -- , seq: 85801, time: 2014-09-18 18:08:27.970913, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11705) -7176> 2014-09-18 18:08:27.970922 7f4c9f251700 5 -- op tracker -- , seq: 85801, time: 2014-09-18 18:08:27.970921, event: started, op: pg_notify(2.1d8(9) epoch 11705) -7175> 2014-09-18 18:08:27.970942 7f4c9f251700 5 -- op tracker -- , seq: 85801, time: 2014-09-18 18:08:27.970940, event: done, op: pg_notify(2.1d8(9) epoch 11705) -7174> 2014-09-18 18:08:27.972797 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1838 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x1182b440 con 0x10f60520 -7173> 2014-09-18 18:08:27.972821 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7172> 2014-09-18 18:08:27.972834 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1839 ==== pg_notify(2.4f6(15) epoch 11706) v5 ==== 1581+0+0 (3741277874 0 0) 0xe775c20 con 0x10f60520 -7171> 2014-09-18 18:08:27.972851 7f4c9f251700 5 -- op tracker -- , seq: 85802, time: 2014-09-18 18:08:27.972743, event: header_read, op: pg_notify(2.4f6(15) epoch 11706) -7170> 2014-09-18 18:08:27.972866 7f4c9f251700 5 -- op tracker -- , seq: 85802, time: 2014-09-18 18:08:27.972744, event: throttled, op: pg_notify(2.4f6(15) epoch 11706) -7169> 2014-09-18 18:08:27.972877 7f4c9f251700 5 -- op tracker -- , seq: 85802, time: 2014-09-18 18:08:27.972788, event: all_read, op: pg_notify(2.4f6(15) epoch 11706) -7168> 2014-09-18 18:08:27.972889 7f4c9f251700 5 -- op tracker -- , seq: 85802, time: 2014-09-18 18:08:27.972846, event: dispatched, op: pg_notify(2.4f6(15) epoch 11706) -7167> 2014-09-18 18:08:27.972901 7f4c9f251700 5 -- op tracker -- , seq: 85802, time: 2014-09-18 18:08:27.972898, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11706) -7166> 2014-09-18 18:08:27.972916 7f4c9f251700 5 -- op tracker -- , seq: 85802, time: 2014-09-18 18:08:27.972913, event: started, op: pg_notify(2.4f6(15) epoch 11706) -7165> 2014-09-18 18:08:27.972957 7f4c9f251700 5 -- op tracker -- , seq: 85802, time: 2014-09-18 18:08:27.972954, event: done, op: pg_notify(2.4f6(15) epoch 11706) -7164> 2014-09-18 18:08:27.974768 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1840 ==== pg_notify(2.517(11) epoch 11706) v5 ==== 1335+0+0 (850412567 0 0) 0xe7761c0 con 0x10f60520 -7163> 2014-09-18 18:08:27.974788 7f4c9f251700 5 -- op tracker -- , seq: 85803, time: 2014-09-18 18:08:27.974705, event: header_read, op: pg_notify(2.517(11) epoch 11706) -7162> 2014-09-18 18:08:27.974801 7f4c9f251700 5 -- op tracker -- , seq: 85803, time: 2014-09-18 18:08:27.974706, event: throttled, op: pg_notify(2.517(11) epoch 11706) -7161> 2014-09-18 18:08:27.974808 7f4c9f251700 5 -- op tracker -- , seq: 85803, time: 2014-09-18 18:08:27.974745, event: all_read, op: pg_notify(2.517(11) epoch 11706) -7160> 2014-09-18 18:08:27.974814 7f4c9f251700 5 -- op tracker -- , seq: 85803, time: 2014-09-18 18:08:27.974782, event: dispatched, op: pg_notify(2.517(11) epoch 11706) -7159> 2014-09-18 18:08:27.974825 7f4c9f251700 5 -- op tracker -- , seq: 85803, time: 2014-09-18 18:08:27.974823, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11706) -7158> 2014-09-18 18:08:27.974834 7f4c9f251700 5 -- op tracker -- , seq: 85803, time: 2014-09-18 18:08:27.974832, event: started, op: pg_notify(2.517(11) epoch 11706) -7157> 2014-09-18 18:08:27.974857 7f4c9f251700 5 -- op tracker -- , seq: 85803, time: 2014-09-18 18:08:27.974856, event: done, op: pg_notify(2.517(11) epoch 11706) -7156> 2014-09-18 18:08:27.975814 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1841 ==== pg_notify(2.1d8(9) epoch 11706) v5 ==== 1261+0+0 (3027784763 0 0) 0x21d15c00 con 0x10f60520 -7155> 2014-09-18 18:08:27.975831 7f4c9f251700 5 -- op tracker -- , seq: 85804, time: 2014-09-18 18:08:27.975767, event: header_read, op: pg_notify(2.1d8(9) epoch 11706) -7154> 2014-09-18 18:08:27.975842 7f4c9f251700 5 -- op tracker -- , seq: 85804, time: 2014-09-18 18:08:27.975769, event: throttled, op: pg_notify(2.1d8(9) epoch 11706) -7153> 2014-09-18 18:08:27.975848 7f4c9f251700 5 -- op tracker -- , seq: 85804, time: 2014-09-18 18:08:27.975801, event: all_read, op: pg_notify(2.1d8(9) epoch 11706) -7152> 2014-09-18 18:08:27.975853 7f4c9f251700 5 -- op tracker -- , seq: 85804, time: 2014-09-18 18:08:27.975827, event: dispatched, op: pg_notify(2.1d8(9) epoch 11706) -7151> 2014-09-18 18:08:27.975859 7f4c9f251700 5 -- op tracker -- , seq: 85804, time: 2014-09-18 18:08:27.975858, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11706) -7150> 2014-09-18 18:08:27.975867 7f4c9f251700 5 -- op tracker -- , seq: 85804, time: 2014-09-18 18:08:27.975865, event: started, op: pg_notify(2.1d8(9) epoch 11706) -7149> 2014-09-18 18:08:27.975885 7f4c9f251700 5 -- op tracker -- , seq: 85804, time: 2014-09-18 18:08:27.975883, event: done, op: pg_notify(2.1d8(9) epoch 11706) -7148> 2014-09-18 18:08:27.998085 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 678 ==== osd_map(11705..11705 src has 1..11705) v3 ==== 337+0+0 (2797987199 0 0) 0x19df2f40 con 0x1f103e40 -7147> 2014-09-18 18:08:27.998107 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11705,11705], i have 11706, src has [1,11705] -7146> 2014-09-18 18:08:27.998160 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 679 ==== pg_notify(2.4f6(15) epoch 11705) v5 ==== 1543+0+0 (3704083729 0 0) 0xde5e3c0 con 0x1f103e40 -7145> 2014-09-18 18:08:27.998172 7f4c9f251700 5 -- op tracker -- , seq: 85805, time: 2014-09-18 18:08:27.998105, event: header_read, op: pg_notify(2.4f6(15) epoch 11705) -7144> 2014-09-18 18:08:27.998182 7f4c9f251700 5 -- op tracker -- , seq: 85805, time: 2014-09-18 18:08:27.998106, event: throttled, op: pg_notify(2.4f6(15) epoch 11705) -7143> 2014-09-18 18:08:27.998189 7f4c9f251700 5 -- op tracker -- , seq: 85805, time: 2014-09-18 18:08:27.998150, event: all_read, op: pg_notify(2.4f6(15) epoch 11705) -7142> 2014-09-18 18:08:27.998194 7f4c9f251700 5 -- op tracker -- , seq: 85805, time: 2014-09-18 18:08:27.998168, event: dispatched, op: pg_notify(2.4f6(15) epoch 11705) -7141> 2014-09-18 18:08:27.998201 7f4c9f251700 5 -- op tracker -- , seq: 85805, time: 2014-09-18 18:08:27.998200, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11705) -7140> 2014-09-18 18:08:27.998209 7f4c9f251700 5 -- op tracker -- , seq: 85805, time: 2014-09-18 18:08:27.998208, event: started, op: pg_notify(2.4f6(15) epoch 11705) -7139> 2014-09-18 18:08:27.998230 7f4c9f251700 5 -- op tracker -- , seq: 85805, time: 2014-09-18 18:08:27.998228, event: done, op: pg_notify(2.4f6(15) epoch 11705) -7138> 2014-09-18 18:08:28.011502 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 680 ==== pg_notify(2.799(10),2.4c6(15) epoch 11705) v5 ==== 2779+0+0 (4212032917 0 0) 0x1247d4a0 con 0x1f103e40 -7137> 2014-09-18 18:08:28.011520 7f4c9f251700 5 -- op tracker -- , seq: 85806, time: 2014-09-18 18:08:28.011428, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11705) -7136> 2014-09-18 18:08:28.011531 7f4c9f251700 5 -- op tracker -- , seq: 85806, time: 2014-09-18 18:08:28.011430, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11705) -7135> 2014-09-18 18:08:28.011539 7f4c9f251700 5 -- op tracker -- , seq: 85806, time: 2014-09-18 18:08:28.011486, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11705) -7134> 2014-09-18 18:08:28.011546 7f4c9f251700 5 -- op tracker -- , seq: 85806, time: 2014-09-18 18:08:28.011516, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11705) -7133> 2014-09-18 18:08:28.011554 7f4c9f251700 5 -- op tracker -- , seq: 85806, time: 2014-09-18 18:08:28.011552, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11705) -7132> 2014-09-18 18:08:28.011563 7f4c9f251700 5 -- op tracker -- , seq: 85806, time: 2014-09-18 18:08:28.011561, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11705) -7131> 2014-09-18 18:08:28.011612 7f4c9f251700 5 -- op tracker -- , seq: 85806, time: 2014-09-18 18:08:28.011610, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11705) -7130> 2014-09-18 18:08:28.012459 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 681 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0x11bb93c0 con 0x1f103e40 -7129> 2014-09-18 18:08:28.012476 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7128> 2014-09-18 18:08:28.012504 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 682 ==== pg_notify(2.4f6(15) epoch 11706) v5 ==== 1543+0+0 (880026891 0 0) 0x21e114a0 con 0x1f103e40 -7127> 2014-09-18 18:08:28.012524 7f4c9f251700 5 -- op tracker -- , seq: 85807, time: 2014-09-18 18:08:28.012461, event: header_read, op: pg_notify(2.4f6(15) epoch 11706) -7126> 2014-09-18 18:08:28.012534 7f4c9f251700 5 -- op tracker -- , seq: 85807, time: 2014-09-18 18:08:28.012462, event: throttled, op: pg_notify(2.4f6(15) epoch 11706) -7125> 2014-09-18 18:08:28.012540 7f4c9f251700 5 -- op tracker -- , seq: 85807, time: 2014-09-18 18:08:28.012493, event: all_read, op: pg_notify(2.4f6(15) epoch 11706) -7124> 2014-09-18 18:08:28.012546 7f4c9f251700 5 -- op tracker -- , seq: 85807, time: 2014-09-18 18:08:28.012521, event: dispatched, op: pg_notify(2.4f6(15) epoch 11706) -7123> 2014-09-18 18:08:28.012553 7f4c9f251700 5 -- op tracker -- , seq: 85807, time: 2014-09-18 18:08:28.012551, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11706) -7122> 2014-09-18 18:08:28.012561 7f4c9f251700 5 -- op tracker -- , seq: 85807, time: 2014-09-18 18:08:28.012559, event: started, op: pg_notify(2.4f6(15) epoch 11706) -7121> 2014-09-18 18:08:28.012579 7f4c9f251700 5 -- op tracker -- , seq: 85807, time: 2014-09-18 18:08:28.012577, event: done, op: pg_notify(2.4f6(15) epoch 11706) -7120> 2014-09-18 18:08:28.015436 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 683 ==== pg_notify(2.799(10),2.4c6(15) epoch 11706) v5 ==== 2779+0+0 (3880454260 0 0) 0x21e10000 con 0x1f103e40 -7119> 2014-09-18 18:08:28.015457 7f4c9f251700 5 -- op tracker -- , seq: 85808, time: 2014-09-18 18:08:28.015376, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11706) -7118> 2014-09-18 18:08:28.015466 7f4c9f251700 5 -- op tracker -- , seq: 85808, time: 2014-09-18 18:08:28.015377, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11706) -7117> 2014-09-18 18:08:28.015470 7f4c9f251700 5 -- op tracker -- , seq: 85808, time: 2014-09-18 18:08:28.015422, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11706) -7116> 2014-09-18 18:08:28.015474 7f4c9f251700 5 -- op tracker -- , seq: 85808, time: 2014-09-18 18:08:28.015452, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11706) -7115> 2014-09-18 18:08:28.015479 7f4c9f251700 5 -- op tracker -- , seq: 85808, time: 2014-09-18 18:08:28.015477, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11706) -7114> 2014-09-18 18:08:28.015485 7f4c9f251700 5 -- op tracker -- , seq: 85808, time: 2014-09-18 18:08:28.015484, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11706) -7113> 2014-09-18 18:08:28.015516 7f4c9f251700 5 -- op tracker -- , seq: 85808, time: 2014-09-18 18:08:28.015514, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11706) -7112> 2014-09-18 18:08:28.030428 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1274 ==== osd_map(11706..11706 src has 1..11706) v3 ==== 272+0+0 (2643785718 0 0) 0xeee8900 con 0x1f102840 -7111> 2014-09-18 18:08:28.030445 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11706,11706], i have 11706, src has [1,11706] -7110> 2014-09-18 18:08:28.030487 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1275 ==== pg_notify(2.4f6(15) epoch 11706) v5 ==== 1543+0+0 (3223317981 0 0) 0xf099480 con 0x1f102840 -7109> 2014-09-18 18:08:28.030498 7f4c9f251700 5 -- op tracker -- , seq: 85809, time: 2014-09-18 18:08:28.030445, event: header_read, op: pg_notify(2.4f6(15) epoch 11706) -7108> 2014-09-18 18:08:28.030507 7f4c9f251700 5 -- op tracker -- , seq: 85809, time: 2014-09-18 18:08:28.030446, event: throttled, op: pg_notify(2.4f6(15) epoch 11706) -7107> 2014-09-18 18:08:28.030513 7f4c9f251700 5 -- op tracker -- , seq: 85809, time: 2014-09-18 18:08:28.030477, event: all_read, op: pg_notify(2.4f6(15) epoch 11706) -7106> 2014-09-18 18:08:28.030519 7f4c9f251700 5 -- op tracker -- , seq: 85809, time: 2014-09-18 18:08:28.030495, event: dispatched, op: pg_notify(2.4f6(15) epoch 11706) -7105> 2014-09-18 18:08:28.030534 7f4c9f251700 5 -- op tracker -- , seq: 85809, time: 2014-09-18 18:08:28.030532, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11706) -7104> 2014-09-18 18:08:28.030550 7f4c9f251700 5 -- op tracker -- , seq: 85809, time: 2014-09-18 18:08:28.030548, event: started, op: pg_notify(2.4f6(15) epoch 11706) -7103> 2014-09-18 18:08:28.030566 7f4c9f251700 5 -- op tracker -- , seq: 85809, time: 2014-09-18 18:08:28.030565, event: done, op: pg_notify(2.4f6(15) epoch 11706) -7102> 2014-09-18 18:08:28.043748 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3169 ==== pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11706) v5 ==== 6445+0+0 (2906108953 0 0) 0xf810580 con 0x1395ec60 -7101> 2014-09-18 18:08:28.043769 7f4c9f251700 5 -- op tracker -- , seq: 85810, time: 2014-09-18 18:08:28.043619, event: header_read, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11706) -7100> 2014-09-18 18:08:28.043779 7f4c9f251700 5 -- op tracker -- , seq: 85810, time: 2014-09-18 18:08:28.043620, event: throttled, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11706) -7099> 2014-09-18 18:08:28.043784 7f4c9f251700 5 -- op tracker -- , seq: 85810, time: 2014-09-18 18:08:28.043709, event: all_read, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11706) -7098> 2014-09-18 18:08:28.043795 7f4c9f251700 5 -- op tracker -- , seq: 85810, time: 2014-09-18 18:08:28.043765, event: dispatched, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11706) -7097> 2014-09-18 18:08:28.043813 7f4c9f251700 5 -- op tracker -- , seq: 85810, time: 2014-09-18 18:08:28.043811, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11706) -7096> 2014-09-18 18:08:28.043822 7f4c9f251700 5 -- op tracker -- , seq: 85810, time: 2014-09-18 18:08:28.043820, event: started, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11706) -7095> 2014-09-18 18:08:28.043856 7f4c9f251700 5 -- op tracker -- , seq: 85810, time: 2014-09-18 18:08:28.043855, event: done, op: pg_notify(2.4c1(15),2.78f(51),2.4c6(15) epoch 11706) -7094> 2014-09-18 18:08:28.043873 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3170 ==== pg_notify(2.4b1(13) epoch 11706) v5 ==== 1441+0+0 (937447090 0 0) 0xf80f680 con 0x1395ec60 -7093> 2014-09-18 18:08:28.043883 7f4c9f251700 5 -- op tracker -- , seq: 85811, time: 2014-09-18 18:08:28.043730, event: header_read, op: pg_notify(2.4b1(13) epoch 11706) -7092> 2014-09-18 18:08:28.043890 7f4c9f251700 5 -- op tracker -- , seq: 85811, time: 2014-09-18 18:08:28.043731, event: throttled, op: pg_notify(2.4b1(13) epoch 11706) -7091> 2014-09-18 18:08:28.043899 7f4c9f251700 5 -- op tracker -- , seq: 85811, time: 2014-09-18 18:08:28.043763, event: all_read, op: pg_notify(2.4b1(13) epoch 11706) -7090> 2014-09-18 18:08:28.043905 7f4c9f251700 5 -- op tracker -- , seq: 85811, time: 2014-09-18 18:08:28.043881, event: dispatched, op: pg_notify(2.4b1(13) epoch 11706) -7089> 2014-09-18 18:08:28.043911 7f4c9f251700 5 -- op tracker -- , seq: 85811, time: 2014-09-18 18:08:28.043910, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11706) -7088> 2014-09-18 18:08:28.043919 7f4c9f251700 5 -- op tracker -- , seq: 85811, time: 2014-09-18 18:08:28.043917, event: started, op: pg_notify(2.4b1(13) epoch 11706) -7087> 2014-09-18 18:08:28.043934 7f4c9f251700 5 -- op tracker -- , seq: 85811, time: 2014-09-18 18:08:28.043933, event: done, op: pg_notify(2.4b1(13) epoch 11706) -7086> 2014-09-18 18:08:28.044326 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3171 ==== pg_notify(2.799(9),2.4d0(10) epoch 11706) v5 ==== 2509+0+0 (1488151880 0 0) 0xf810940 con 0x1395ec60 -7085> 2014-09-18 18:08:28.044347 7f4c9f251700 5 -- op tracker -- , seq: 85812, time: 2014-09-18 18:08:28.044254, event: header_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11706) -7084> 2014-09-18 18:08:28.044357 7f4c9f251700 5 -- op tracker -- , seq: 85812, time: 2014-09-18 18:08:28.044255, event: throttled, op: pg_notify(2.799(9),2.4d0(10) epoch 11706) -7083> 2014-09-18 18:08:28.044364 7f4c9f251700 5 -- op tracker -- , seq: 85812, time: 2014-09-18 18:08:28.044292, event: all_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11706) -7082> 2014-09-18 18:08:28.044370 7f4c9f251700 5 -- op tracker -- , seq: 85812, time: 2014-09-18 18:08:28.044342, event: dispatched, op: pg_notify(2.799(9),2.4d0(10) epoch 11706) -7081> 2014-09-18 18:08:28.044377 7f4c9f251700 5 -- op tracker -- , seq: 85812, time: 2014-09-18 18:08:28.044375, event: waiting_for_osdmap, op: pg_notify(2.799(9),2.4d0(10) epoch 11706) -7080> 2014-09-18 18:08:28.044385 7f4c9f251700 5 -- op tracker -- , seq: 85812, time: 2014-09-18 18:08:28.044383, event: started, op: pg_notify(2.799(9),2.4d0(10) epoch 11706) -7079> 2014-09-18 18:08:28.044419 7f4c9f251700 5 -- op tracker -- , seq: 85812, time: 2014-09-18 18:08:28.044417, event: done, op: pg_notify(2.799(9),2.4d0(10) epoch 11706) -7078> 2014-09-18 18:08:28.086283 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.35 10.242.42.164:0/17421 164 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.084472) v2 ==== 47+0+0 (3843232769 0 0) 0xe76e1c0 con 0x142e9700 -7077> 2014-09-18 18:08:28.086327 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.35 10.242.42.164:0/17421 164 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.084472) v2 ==== 47+0+0 (3843232769 0 0) 0xf098940 con 0x1d5bcd60 -7076> 2014-09-18 18:08:28.086338 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/17421 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.084472) v2 -- ?+0 0xe1a21e0 con 0x142e9700 -7075> 2014-09-18 18:08:28.086392 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/17421 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.084472) v2 -- ?+0 0x2b92fe00 con 0x1d5bcd60 -7074> 2014-09-18 18:08:28.145884 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=34 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).connect error 10.242.42.164:6884/38815, (111) Connection refused -7073> 2014-09-18 18:08:28.145929 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6884/38815 pipe(0x10b62580 sd=34 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3a680).fault (111) Connection refused -7072> 2014-09-18 18:08:28.146067 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=39 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).connect error 10.242.42.164:6886/38815, (111) Connection refused -7071> 2014-09-18 18:08:28.146086 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=39 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault (111) Connection refused -7070> 2014-09-18 18:08:28.146078 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).connect error 10.242.42.164:6885/38815, (111) Connection refused -7069> 2014-09-18 18:08:28.146096 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault (111) Connection refused -7068> 2014-09-18 18:08:28.189322 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.10 10.242.42.168:0/10695 688 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.056111) v2 ==== 47+0+0 (3349649900 0 0) 0xf4e7c20 con 0x140f9b20 -7067> 2014-09-18 18:08:28.189358 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.10 10.242.42.168:0/10695 688 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.056111) v2 ==== 47+0+0 (3349649900 0 0) 0xf2b8d00 con 0x109f9b20 -7066> 2014-09-18 18:08:28.189361 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10695 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.056111) v2 -- ?+0 0xf098940 con 0x140f9b20 -7065> 2014-09-18 18:08:28.189427 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10695 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.056111) v2 -- ?+0 0xe76e1c0 con 0x109f9b20 -7064> 2014-09-18 18:08:28.237373 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.14 10.242.42.168:0/16237 140 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.104103) v2 ==== 47+0+0 (96182634 0 0) 0xf7a6f00 con 0xcc49a20 -7063> 2014-09-18 18:08:28.237401 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.104103) v2 -- ?+0 0xf2b8d00 con 0xcc49a20 -7062> 2014-09-18 18:08:28.237418 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.14 10.242.42.168:0/16237 140 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.104103) v2 ==== 47+0+0 (96182634 0 0) 0xe0ba5a0 con 0x101cc3c0 -7061> 2014-09-18 18:08:28.237441 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.104103) v2 -- ?+0 0xf4e7c20 con 0x101cc3c0 -7060> 2014-09-18 18:08:28.270427 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 955 ==== pg_notify(2.4c6(15) epoch 11705) v5 ==== 1511+0+0 (4221890713 0 0) 0x162503a0 con 0x10a878c0 -7059> 2014-09-18 18:08:28.270448 7f4c9f251700 5 -- op tracker -- , seq: 85813, time: 2014-09-18 18:08:28.270347, event: header_read, op: pg_notify(2.4c6(15) epoch 11705) -7058> 2014-09-18 18:08:28.270468 7f4c9f251700 5 -- op tracker -- , seq: 85813, time: 2014-09-18 18:08:28.270349, event: throttled, op: pg_notify(2.4c6(15) epoch 11705) -7057> 2014-09-18 18:08:28.270473 7f4c9f251700 5 -- op tracker -- , seq: 85813, time: 2014-09-18 18:08:28.270410, event: all_read, op: pg_notify(2.4c6(15) epoch 11705) -7056> 2014-09-18 18:08:28.270478 7f4c9f251700 5 -- op tracker -- , seq: 85813, time: 2014-09-18 18:08:28.270442, event: dispatched, op: pg_notify(2.4c6(15) epoch 11705) -7055> 2014-09-18 18:08:28.270482 7f4c9f251700 5 -- op tracker -- , seq: 85813, time: 2014-09-18 18:08:28.270481, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11705) -7054> 2014-09-18 18:08:28.270489 7f4c9f251700 5 -- op tracker -- , seq: 85813, time: 2014-09-18 18:08:28.270488, event: started, op: pg_notify(2.4c6(15) epoch 11705) -7053> 2014-09-18 18:08:28.270514 7f4c9f251700 5 -- op tracker -- , seq: 85813, time: 2014-09-18 18:08:28.270513, event: done, op: pg_notify(2.4c6(15) epoch 11705) -7052> 2014-09-18 18:08:28.329264 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5068 ==== pg_notify(2.410 epoch 11704) v5 ==== 722+0+0 (3986850410 0 0) 0x2b94b680 con 0x1088aec0 -7051> 2014-09-18 18:08:28.329282 7f4c9f251700 5 -- op tracker -- , seq: 85814, time: 2014-09-18 18:08:28.329201, event: header_read, op: pg_notify(2.410 epoch 11704) -7050> 2014-09-18 18:08:28.329289 7f4c9f251700 5 -- op tracker -- , seq: 85814, time: 2014-09-18 18:08:28.329202, event: throttled, op: pg_notify(2.410 epoch 11704) -7049> 2014-09-18 18:08:28.329293 7f4c9f251700 5 -- op tracker -- , seq: 85814, time: 2014-09-18 18:08:28.329247, event: all_read, op: pg_notify(2.410 epoch 11704) -7048> 2014-09-18 18:08:28.329297 7f4c9f251700 5 -- op tracker -- , seq: 85814, time: 2014-09-18 18:08:28.329278, event: dispatched, op: pg_notify(2.410 epoch 11704) -7047> 2014-09-18 18:08:28.329302 7f4c9f251700 5 -- op tracker -- , seq: 85814, time: 2014-09-18 18:08:28.329301, event: waiting_for_osdmap, op: pg_notify(2.410 epoch 11704) -7046> 2014-09-18 18:08:28.329308 7f4c9f251700 5 -- op tracker -- , seq: 85814, time: 2014-09-18 18:08:28.329307, event: started, op: pg_notify(2.410 epoch 11704) -7045> 2014-09-18 18:08:28.329330 7f4c9f251700 5 -- op tracker -- , seq: 85814, time: 2014-09-18 18:08:28.329328, event: done, op: pg_notify(2.410 epoch 11704) -7044> 2014-09-18 18:08:28.329383 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11505 n=527 ec=188 les/c 11505/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 5.414936 30 0.000208 -7043> 2014-09-18 18:08:28.329411 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11505 n=527 ec=188 les/c 11505/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -7042> 2014-09-18 18:08:28.329610 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11505 n=527 ec=188 les/c 11505/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000199 0 0.000000 -7041> 2014-09-18 18:08:28.329627 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11505 n=527 ec=188 les/c 11505/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -7040> 2014-09-18 18:08:28.329782 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11505 n=527 ec=188 les/c 11505/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000155 0 0.000000 -7039> 2014-09-18 18:08:28.329821 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11505 n=527 ec=188 les/c 11505/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 5.415382 0 0.000000 -7038> 2014-09-18 18:08:28.329834 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11505 n=527 ec=188 les/c 11505/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -7037> 2014-09-18 18:08:28.330059 7f4c97241700 0 log_channel(default) log [INF] : 2.410 restarting backfill on osd.6 from (0'0,0'0] MAX to 6840'31228 -7036> 2014-09-18 18:08:28.330632 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11705..11706 src has 1..11706) v3 -- ?+0 0xeee9d40 con 0x1088aec0 -7035> 2014-09-18 18:08:28.330646 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_log(2.410 epoch 11706 log log((4052'27523,6840'31228], crt=6840'31228) query_epoch 11706) v4 -- ?+0 0x15e8e300 con 0x1088aec0 -7034> 2014-09-18 18:08:28.330680 7f4c97241700 5 osd.42 pg_epoch: 11706 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11505/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -7033> 2014-09-18 18:08:28.332864 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11706..11706 src has 1..11706) v3 -- ?+0 0x19df1f80 con 0x1395f8c0 -7032> 2014-09-18 18:08:28.332914 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11706:2.410(11)) v4 -- ?+0 0x19e492c0 con 0x1395f8c0 -7031> 2014-09-18 18:08:28.334291 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 919 ==== pg_info(1 pgs e11706:2.410) v4 ==== 734+0+0 (1489180279 0 0) 0x19e492c0 con 0x1395f8c0 -7030> 2014-09-18 18:08:28.334309 7f4c9f251700 5 -- op tracker -- , seq: 85815, time: 2014-09-18 18:08:28.334161, event: header_read, op: pg_info(1 pgs e11706:2.410) -7029> 2014-09-18 18:08:28.334317 7f4c9f251700 5 -- op tracker -- , seq: 85815, time: 2014-09-18 18:08:28.334161, event: throttled, op: pg_info(1 pgs e11706:2.410) -7028> 2014-09-18 18:08:28.334321 7f4c9f251700 5 -- op tracker -- , seq: 85815, time: 2014-09-18 18:08:28.334188, event: all_read, op: pg_info(1 pgs e11706:2.410) -7027> 2014-09-18 18:08:28.334337 7f4c9f251700 5 -- op tracker -- , seq: 85815, time: 2014-09-18 18:08:28.334305, event: dispatched, op: pg_info(1 pgs e11706:2.410) -7026> 2014-09-18 18:08:28.334342 7f4c9f251700 5 -- op tracker -- , seq: 85815, time: 2014-09-18 18:08:28.334341, event: waiting_for_osdmap, op: pg_info(1 pgs e11706:2.410) -7025> 2014-09-18 18:08:28.334348 7f4c9f251700 5 -- op tracker -- , seq: 85815, time: 2014-09-18 18:08:28.334347, event: started, op: pg_info(1 pgs e11706:2.410) -7024> 2014-09-18 18:08:28.334366 7f4c9f251700 5 -- op tracker -- , seq: 85815, time: 2014-09-18 18:08:28.334364, event: done, op: pg_info(1 pgs e11706:2.410) -7023> 2014-09-18 18:08:28.339510 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5069 ==== osd_map(11705..11706 src has 1..11706) v3 ==== 577+0+0 (3285189876 0 0) 0x139a6b40 con 0x1088aec0 -7022> 2014-09-18 18:08:28.339529 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11705,11706], i have 11706, src has [1,11706] -7021> 2014-09-18 18:08:28.339562 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5070 ==== pg_notify(2.62(44) epoch 11706) v5 ==== 3220+0+0 (4133468735 0 0) 0x2b94d2a0 con 0x1088aec0 -7020> 2014-09-18 18:08:28.339574 7f4c9f251700 5 -- op tracker -- , seq: 85816, time: 2014-09-18 18:08:28.339509, event: header_read, op: pg_notify(2.62(44) epoch 11706) -7019> 2014-09-18 18:08:28.339584 7f4c9f251700 5 -- op tracker -- , seq: 85816, time: 2014-09-18 18:08:28.339509, event: throttled, op: pg_notify(2.62(44) epoch 11706) -7018> 2014-09-18 18:08:28.339592 7f4c9f251700 5 -- op tracker -- , seq: 85816, time: 2014-09-18 18:08:28.339553, event: all_read, op: pg_notify(2.62(44) epoch 11706) -7017> 2014-09-18 18:08:28.339598 7f4c9f251700 5 -- op tracker -- , seq: 85816, time: 2014-09-18 18:08:28.339571, event: dispatched, op: pg_notify(2.62(44) epoch 11706) -7016> 2014-09-18 18:08:28.339605 7f4c9f251700 5 -- op tracker -- , seq: 85816, time: 2014-09-18 18:08:28.339603, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11706) -7015> 2014-09-18 18:08:28.339613 7f4c9f251700 5 -- op tracker -- , seq: 85816, time: 2014-09-18 18:08:28.339612, event: started, op: pg_notify(2.62(44) epoch 11706) -7014> 2014-09-18 18:08:28.339632 7f4c9f251700 5 -- op tracker -- , seq: 85816, time: 2014-09-18 18:08:28.339630, event: done, op: pg_notify(2.62(44) epoch 11706) -7013> 2014-09-18 18:08:28.344225 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5071 ==== pg_notify(2.3b6(10) epoch 11706) v5 ==== 1260+0+0 (2368824179 0 0) 0x2b94af00 con 0x1088aec0 -7012> 2014-09-18 18:08:28.344245 7f4c9f251700 5 -- op tracker -- , seq: 85817, time: 2014-09-18 18:08:28.344146, event: header_read, op: pg_notify(2.3b6(10) epoch 11706) -7011> 2014-09-18 18:08:28.344256 7f4c9f251700 5 -- op tracker -- , seq: 85817, time: 2014-09-18 18:08:28.344147, event: throttled, op: pg_notify(2.3b6(10) epoch 11706) -7010> 2014-09-18 18:08:28.344262 7f4c9f251700 5 -- op tracker -- , seq: 85817, time: 2014-09-18 18:08:28.344175, event: all_read, op: pg_notify(2.3b6(10) epoch 11706) -7009> 2014-09-18 18:08:28.344268 7f4c9f251700 5 -- op tracker -- , seq: 85817, time: 2014-09-18 18:08:28.344241, event: dispatched, op: pg_notify(2.3b6(10) epoch 11706) -7008> 2014-09-18 18:08:28.344275 7f4c9f251700 5 -- op tracker -- , seq: 85817, time: 2014-09-18 18:08:28.344273, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11706) -7007> 2014-09-18 18:08:28.344283 7f4c9f251700 5 -- op tracker -- , seq: 85817, time: 2014-09-18 18:08:28.344281, event: started, op: pg_notify(2.3b6(10) epoch 11706) -7006> 2014-09-18 18:08:28.344302 7f4c9f251700 5 -- op tracker -- , seq: 85817, time: 2014-09-18 18:08:28.344300, event: done, op: pg_notify(2.3b6(10) epoch 11706) -7005> 2014-09-18 18:08:28.347137 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5072 ==== pg_notify(2.517(11) epoch 11706) v5 ==== 1335+0+0 (1003676982 0 0) 0x2b94d0c0 con 0x1088aec0 -7004> 2014-09-18 18:08:28.347156 7f4c9f251700 5 -- op tracker -- , seq: 85818, time: 2014-09-18 18:08:28.347067, event: header_read, op: pg_notify(2.517(11) epoch 11706) -7003> 2014-09-18 18:08:28.347167 7f4c9f251700 5 -- op tracker -- , seq: 85818, time: 2014-09-18 18:08:28.347068, event: throttled, op: pg_notify(2.517(11) epoch 11706) -7002> 2014-09-18 18:08:28.347173 7f4c9f251700 5 -- op tracker -- , seq: 85818, time: 2014-09-18 18:08:28.347100, event: all_read, op: pg_notify(2.517(11) epoch 11706) -7001> 2014-09-18 18:08:28.347180 7f4c9f251700 5 -- op tracker -- , seq: 85818, time: 2014-09-18 18:08:28.347152, event: dispatched, op: pg_notify(2.517(11) epoch 11706) -7000> 2014-09-18 18:08:28.347186 7f4c9f251700 5 -- op tracker -- , seq: 85818, time: 2014-09-18 18:08:28.347185, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11706) -6999> 2014-09-18 18:08:28.347194 7f4c9f251700 5 -- op tracker -- , seq: 85818, time: 2014-09-18 18:08:28.347193, event: started, op: pg_notify(2.517(11) epoch 11706) -6998> 2014-09-18 18:08:28.347214 7f4c9f251700 5 -- op tracker -- , seq: 85818, time: 2014-09-18 18:08:28.347212, event: done, op: pg_notify(2.517(11) epoch 11706) -6997> 2014-09-18 18:08:28.351676 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5073 ==== pg_notify(2.40a(162) epoch 11706) v5 ==== 9736+0+0 (1178869868 0 0) 0x2b94b4a0 con 0x1088aec0 -6996> 2014-09-18 18:08:28.351694 7f4c9f251700 5 -- op tracker -- , seq: 85819, time: 2014-09-18 18:08:28.351534, event: header_read, op: pg_notify(2.40a(162) epoch 11706) -6995> 2014-09-18 18:08:28.351705 7f4c9f251700 5 -- op tracker -- , seq: 85819, time: 2014-09-18 18:08:28.351534, event: throttled, op: pg_notify(2.40a(162) epoch 11706) -6994> 2014-09-18 18:08:28.351711 7f4c9f251700 5 -- op tracker -- , seq: 85819, time: 2014-09-18 18:08:28.351664, event: all_read, op: pg_notify(2.40a(162) epoch 11706) -6993> 2014-09-18 18:08:28.351717 7f4c9f251700 5 -- op tracker -- , seq: 85819, time: 2014-09-18 18:08:28.351690, event: dispatched, op: pg_notify(2.40a(162) epoch 11706) -6992> 2014-09-18 18:08:28.351724 7f4c9f251700 5 -- op tracker -- , seq: 85819, time: 2014-09-18 18:08:28.351722, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11706) -6991> 2014-09-18 18:08:28.351732 7f4c9f251700 5 -- op tracker -- , seq: 85819, time: 2014-09-18 18:08:28.351730, event: started, op: pg_notify(2.40a(162) epoch 11706) -6990> 2014-09-18 18:08:28.351754 7f4c9f251700 5 -- op tracker -- , seq: 85819, time: 2014-09-18 18:08:28.351751, event: done, op: pg_notify(2.40a(162) epoch 11706) -6989> 2014-09-18 18:08:28.352357 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5074 ==== pg_notify(2.120(11) epoch 11706) v5 ==== 1315+0+0 (3296384210 0 0) 0x2b94bfe0 con 0x1088aec0 -6988> 2014-09-18 18:08:28.352378 7f4c9f251700 5 -- op tracker -- , seq: 85820, time: 2014-09-18 18:08:28.352296, event: header_read, op: pg_notify(2.120(11) epoch 11706) -6987> 2014-09-18 18:08:28.352388 7f4c9f251700 5 -- op tracker -- , seq: 85820, time: 2014-09-18 18:08:28.352297, event: throttled, op: pg_notify(2.120(11) epoch 11706) -6986> 2014-09-18 18:08:28.352394 7f4c9f251700 5 -- op tracker -- , seq: 85820, time: 2014-09-18 18:08:28.352320, event: all_read, op: pg_notify(2.120(11) epoch 11706) -6985> 2014-09-18 18:08:28.352400 7f4c9f251700 5 -- op tracker -- , seq: 85820, time: 2014-09-18 18:08:28.352373, event: dispatched, op: pg_notify(2.120(11) epoch 11706) -6984> 2014-09-18 18:08:28.352407 7f4c9f251700 5 -- op tracker -- , seq: 85820, time: 2014-09-18 18:08:28.352405, event: waiting_for_osdmap, op: pg_notify(2.120(11) epoch 11706) -6983> 2014-09-18 18:08:28.352415 7f4c9f251700 5 -- op tracker -- , seq: 85820, time: 2014-09-18 18:08:28.352413, event: started, op: pg_notify(2.120(11) epoch 11706) -6982> 2014-09-18 18:08:28.352433 7f4c9f251700 5 -- op tracker -- , seq: 85820, time: 2014-09-18 18:08:28.352432, event: done, op: pg_notify(2.120(11) epoch 11706) -6981> 2014-09-18 18:08:28.353741 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5075 ==== pg_notify(2.1a2(20) epoch 11706) v5 ==== 1798+0+0 (3308571253 0 0) 0x2b94c580 con 0x1088aec0 -6980> 2014-09-18 18:08:28.353782 7f4c9f251700 5 -- op tracker -- , seq: 85821, time: 2014-09-18 18:08:28.353680, event: header_read, op: pg_notify(2.1a2(20) epoch 11706) -6979> 2014-09-18 18:08:28.353790 7f4c9f251700 5 -- op tracker -- , seq: 85821, time: 2014-09-18 18:08:28.353681, event: throttled, op: pg_notify(2.1a2(20) epoch 11706) -6978> 2014-09-18 18:08:28.353796 7f4c9f251700 5 -- op tracker -- , seq: 85821, time: 2014-09-18 18:08:28.353730, event: all_read, op: pg_notify(2.1a2(20) epoch 11706) -6977> 2014-09-18 18:08:28.353802 7f4c9f251700 5 -- op tracker -- , seq: 85821, time: 2014-09-18 18:08:28.353778, event: dispatched, op: pg_notify(2.1a2(20) epoch 11706) -6976> 2014-09-18 18:08:28.353808 7f4c9f251700 5 -- op tracker -- , seq: 85821, time: 2014-09-18 18:08:28.353806, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11706) -6975> 2014-09-18 18:08:28.353815 7f4c9f251700 5 -- op tracker -- , seq: 85821, time: 2014-09-18 18:08:28.353814, event: started, op: pg_notify(2.1a2(20) epoch 11706) -6974> 2014-09-18 18:08:28.353946 7f4c9f251700 5 -- op tracker -- , seq: 85821, time: 2014-09-18 18:08:28.353943, event: done, op: pg_notify(2.1a2(20) epoch 11706) -6973> 2014-09-18 18:08:28.356194 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5076 ==== pg_notify(2.4c1(15) epoch 11706) v5 ==== 1507+0+0 (949648608 0 0) 0x2b94bc20 con 0x1088aec0 -6972> 2014-09-18 18:08:28.356208 7f4c9f251700 5 -- op tracker -- , seq: 85822, time: 2014-09-18 18:08:28.356141, event: header_read, op: pg_notify(2.4c1(15) epoch 11706) -6971> 2014-09-18 18:08:28.356217 7f4c9f251700 5 -- op tracker -- , seq: 85822, time: 2014-09-18 18:08:28.356143, event: throttled, op: pg_notify(2.4c1(15) epoch 11706) -6970> 2014-09-18 18:08:28.356223 7f4c9f251700 5 -- op tracker -- , seq: 85822, time: 2014-09-18 18:08:28.356181, event: all_read, op: pg_notify(2.4c1(15) epoch 11706) -6969> 2014-09-18 18:08:28.356229 7f4c9f251700 5 -- op tracker -- , seq: 85822, time: 2014-09-18 18:08:28.356205, event: dispatched, op: pg_notify(2.4c1(15) epoch 11706) -6968> 2014-09-18 18:08:28.356236 7f4c9f251700 5 -- op tracker -- , seq: 85822, time: 2014-09-18 18:08:28.356234, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11706) -6967> 2014-09-18 18:08:28.356243 7f4c9f251700 5 -- op tracker -- , seq: 85822, time: 2014-09-18 18:08:28.356242, event: started, op: pg_notify(2.4c1(15) epoch 11706) -6966> 2014-09-18 18:08:28.356261 7f4c9f251700 5 -- op tracker -- , seq: 85822, time: 2014-09-18 18:08:28.356260, event: done, op: pg_notify(2.4c1(15) epoch 11706) -6965> 2014-09-18 18:08:28.403321 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.51 10.242.42.168:0/15056 149 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.270182) v2 ==== 47+0+0 (3534447056 0 0) 0xf786b40 con 0x17de92e0 -6964> 2014-09-18 18:08:28.403337 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.51 10.242.42.168:0/15056 149 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.270182) v2 ==== 47+0+0 (3534447056 0 0) 0x10507860 con 0xcc76f20 -6963> 2014-09-18 18:08:28.403355 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/15056 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.270182) v2 -- ?+0 0xf7a6f00 con 0x17de92e0 -6962> 2014-09-18 18:08:28.403385 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/15056 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.270182) v2 -- ?+0 0xe0ba5a0 con 0xcc76f20 -6961> 2014-09-18 18:08:28.455176 7f4ca853b700 1 -- 10.242.42.178:6807/2483 >> :/0 pipe(0x10b64940 sd=127 :6807 s=0 pgs=0 cs=0 l=0 c=0x12b3aec0).accept sd=127 10.242.42.164:40058/0 -6960> 2014-09-18 18:08:28.614490 7f4c8fa32700 5 osd.42 11706 heartbeat: osd_stat(230 GB used, 136 GB avail, 366 GB total, peers [0,1,2,3,4,6,9,11,20,22,23,24,27,30,32,35,39,43,47,48,53,55,56,57,58,63]/[] op hist []) -6959> 2014-09-18 18:08:28.614524 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6802/15878 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xf09a1e0 con 0x4751e40 -6958> 2014-09-18 18:08:28.614573 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6803/15878 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xf09b680 con 0x149b0ec0 -6957> 2014-09-18 18:08:28.614586 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6885/38815 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xf09b0e0 con 0x21de0940 -6956> 2014-09-18 18:08:28.614611 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6886/38815 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f792c0 con 0x21dde2c0 -6955> 2014-09-18 18:08:28.614622 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6818/19139 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f7a760 con 0x12b6a940 -6954> 2014-09-18 18:08:28.614634 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6820/19139 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f7b0c0 con 0x1d518aa0 -6953> 2014-09-18 18:08:28.614645 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6824/20623 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f7a940 con 0x1d662b00 -6952> 2014-09-18 18:08:28.614668 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6825/20623 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f79680 con 0x1d6629a0 -6951> 2014-09-18 18:08:28.614680 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6826/22324 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f783c0 con 0x1277db20 -6950> 2014-09-18 18:08:28.614692 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6828/22324 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f7b660 con 0x1277d180 -6949> 2014-09-18 18:08:28.614704 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6810/22481 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f79fe0 con 0x1096fa20 -6948> 2014-09-18 18:08:28.614720 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6813/22481 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f78780 con 0x10970520 -6947> 2014-09-18 18:08:28.614750 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6827/24959 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f7b480 con 0x1db5c420 -6946> 2014-09-18 18:08:28.614768 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6840/24959 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f7aee0 con 0x1db5c160 -6945> 2014-09-18 18:08:28.614786 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6822/17298 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f78000 con 0x1217b760 -6944> 2014-09-18 18:08:28.614806 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6823/17298 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f7a3a0 con 0x1217c7e0 -6943> 2014-09-18 18:08:28.614832 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6823/33871 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24f79c20 con 0xe58a9a0 -6942> 2014-09-18 18:08:28.614844 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6825/33871 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x853de00 con 0xe58a840 -6941> 2014-09-18 18:08:28.614855 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6820/37668 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x853d2c0 con 0x10a407e0 -6940> 2014-09-18 18:08:28.614874 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6832/37668 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x853e760 con 0x10a40260 -6939> 2014-09-18 18:08:28.614920 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6826/11463 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x853d4a0 con 0xe417b20 -6938> 2014-09-18 18:08:28.614933 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6827/11463 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x853c000 con 0xe58ab00 -6937> 2014-09-18 18:08:28.614921 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=39 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).connect error 10.242.42.164:6885/38815, (111) Connection refused -6936> 2014-09-18 18:08:28.614946 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6803/5612 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x853fa20 con 0xeff6f20 -6935> 2014-09-18 18:08:28.614955 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6810/5612 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x853eee0 con 0xeff6c60 -6934> 2014-09-18 18:08:28.614923 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).connect error 10.242.42.164:6886/38815, (111) Connection refused -6933> 2014-09-18 18:08:28.614965 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6802/21548 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xf91c3a0 con 0x1da53a20 -6932> 2014-09-18 18:08:28.614965 7f4c84bb5700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6886/38815 pipe(0xe548b00 sd=102 :0 s=1 pgs=0 cs=0 l=1 c=0x21dde2c0).fault (111) Connection refused -6931> 2014-09-18 18:08:28.614964 7f4c82b95700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6885/38815 pipe(0xe5482c0 sd=39 :0 s=1 pgs=0 cs=0 l=1 c=0x21de0940).fault (111) Connection refused -6930> 2014-09-18 18:08:28.614982 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6803/21548 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xf195de0 con 0x1da53760 -6929> 2014-09-18 18:08:28.614997 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6830/1204 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xf1932c0 con 0x1db5cdc0 -6928> 2014-09-18 18:08:28.615009 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6831/1204 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x10509a20 con 0x1db5cc60 -6927> 2014-09-18 18:08:28.615057 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6825/1099 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x10508760 con 0x1db5da20 -6926> 2014-09-18 18:08:28.615077 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6826/1099 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x105074a0 con 0x1db5db80 -6925> 2014-09-18 18:08:28.615093 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6823/33871 1575 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xe959480 con 0xe58a9a0 -6924> 2014-09-18 18:08:28.615109 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6813/17421 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x105065a0 con 0x125239c0 -6923> 2014-09-18 18:08:28.615164 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6814/17421 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x10506d20 con 0x11d45860 -6922> 2014-09-18 18:08:28.615183 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6817/2660 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x10506960 con 0x238dc420 -6921> 2014-09-18 18:08:28.615195 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6818/2660 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x10508940 con 0x238dcb00 -6920> 2014-09-18 18:08:28.615206 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6843/4511 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x105083a0 con 0x149b1de0 -6919> 2014-09-18 18:08:28.615217 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6845/4511 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x10507a40 con 0x149b07e0 -6918> 2014-09-18 18:08:28.615228 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6832/25184 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xf0e4d00 con 0x1217d180 -6917> 2014-09-18 18:08:28.615239 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6833/25184 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x19e48b40 con 0x1217caa0 -6916> 2014-09-18 18:08:28.615251 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6837/27583 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x19e4a3a0 con 0x10db26e0 -6915> 2014-09-18 18:08:28.615262 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6838/27583 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x19e4a940 con 0x10db4ec0 -6914> 2014-09-18 18:08:28.615276 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6826/2318 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24fa0b40 con 0x1db5c6e0 -6913> 2014-09-18 18:08:28.615288 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6832/2318 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24fa30c0 con 0x1db5e260 -6912> 2014-09-18 18:08:28.615300 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6830/19404 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24fa0f00 con 0xe58d180 -6911> 2014-09-18 18:08:28.615314 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6831/19404 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24fa12c0 con 0xe58d020 -6910> 2014-09-18 18:08:28.615327 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6840/36458 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24fa3660 con 0x1364b700 -6909> 2014-09-18 18:08:28.615338 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6842/36458 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24fa2d00 con 0x1364b860 -6908> 2014-09-18 18:08:28.615349 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6802/988 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x24fa14a0 con 0x1db5d600 -6907> 2014-09-18 18:08:28.615361 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6803/988 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xf2b1c00 con 0x1db5dfa0 -6906> 2014-09-18 18:08:28.615382 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6839/46756 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0xec6a1c0 con 0x1d7c7a20 -6905> 2014-09-18 18:08:28.615397 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6842/46756 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x1390a3c0 con 0x1da543c0 -6904> 2014-09-18 18:08:28.615427 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6805/2487 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x14183fe0 con 0x1d996aa0 -6903> 2014-09-18 18:08:28.615453 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6809/2487 -- osd_ping(ping e11706 stamp 2014-09-18 18:08:28.614522) v2 -- ?+0 0x14185c00 con 0x1d996c00 -6902> 2014-09-18 18:08:28.615498 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6810/22481 455 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x2b9232c0 con 0x1096fa20 -6901> 2014-09-18 18:08:28.615523 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6813/22481 455 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x21f043c0 con 0x10970520 -6900> 2014-09-18 18:08:28.615539 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6820/37668 449 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf9725a0 con 0x10a407e0 -6899> 2014-09-18 18:08:28.615567 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6832/37668 449 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf242d20 con 0x10a40260 -6898> 2014-09-18 18:08:28.615594 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6803/5612 341 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf96c3a0 con 0xeff6f20 -6897> 2014-09-18 18:08:28.615631 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6810/5612 341 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x24fa03c0 con 0xeff6c60 -6896> 2014-09-18 18:08:28.615657 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6802/15878 189 ==== osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (3618250017 0 0) 0x21cec940 con 0x4751e40 -6895> 2014-09-18 18:08:28.615672 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6803/15878 189 ==== osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (3618250017 0 0) 0x2b93ba40 con 0x149b0ec0 -6894> 2014-09-18 18:08:28.615684 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6814/17421 186 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x21ceb0e0 con 0x11d45860 -6893> 2014-09-18 18:08:28.615697 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6817/2660 184 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf29d480 con 0x238dc420 -6892> 2014-09-18 18:08:28.615710 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6818/2660 184 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x1393c760 con 0x238dcb00 -6891> 2014-09-18 18:08:28.615722 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6818/19139 180 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x1247c3c0 con 0x12b6a940 -6890> 2014-09-18 18:08:28.615738 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6820/19139 180 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x2b9863c0 con 0x1d518aa0 -6889> 2014-09-18 18:08:28.615751 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6824/20623 169 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x2b930d00 con 0x1d662b00 -6888> 2014-09-18 18:08:28.615763 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6825/20623 169 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x853eb20 con 0x1d6629a0 -6887> 2014-09-18 18:08:28.615774 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6826/22324 163 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x12c10780 con 0x1277db20 -6886> 2014-09-18 18:08:28.615784 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6828/22324 163 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x19f24580 con 0x1277d180 -6885> 2014-09-18 18:08:28.615795 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6843/4511 161 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xfae81c0 con 0x149b1de0 -6884> 2014-09-18 18:08:28.615807 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6845/4511 161 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x1247d680 con 0x149b07e0 -6883> 2014-09-18 18:08:28.615826 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6839/46756 159 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x12139a40 con 0x1d7c7a20 -6882> 2014-09-18 18:08:28.615838 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6842/46756 159 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x2b987860 con 0x1da543c0 -6881> 2014-09-18 18:08:28.615849 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6802/21548 158 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf272d20 con 0x1da53a20 -6880> 2014-09-18 18:08:28.615860 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6803/21548 158 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x1247d2c0 con 0x1da53760 -6879> 2014-09-18 18:08:28.615871 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6822/17298 136 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xe76cb40 con 0x1217b760 -6878> 2014-09-18 18:08:28.615892 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6823/17298 136 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf0dab40 con 0x1217c7e0 -6877> 2014-09-18 18:08:28.615903 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6832/25184 136 ==== osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (3618250017 0 0) 0xf09cb20 con 0x1217d180 -6876> 2014-09-18 18:08:28.615922 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6831/25184 -- osd_map(11706..11706 src has 1..11706) v3 -- ?+0 0xe2ecfc0 con 0x12562260 -6875> 2014-09-18 18:08:28.615938 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6833/25184 136 ==== osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (3618250017 0 0) 0xf866d20 con 0x1217caa0 -6874> 2014-09-18 18:08:28.615984 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6802/988 124 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xe0c6d20 con 0x1db5d600 -6873> 2014-09-18 18:08:28.615998 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6803/988 124 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x12c103c0 con 0x1db5dfa0 -6872> 2014-09-18 18:08:28.616009 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6825/1099 124 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf93c3a0 con 0x1db5da20 -6871> 2014-09-18 18:08:28.616021 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6826/1099 124 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x24de81e0 con 0x1db5db80 -6870> 2014-09-18 18:08:28.616032 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6830/1204 124 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x121134a0 con 0x1db5cdc0 -6869> 2014-09-18 18:08:28.616044 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6831/1204 124 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x10412780 con 0x1db5cc60 -6868> 2014-09-18 18:08:28.616055 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6827/24959 122 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf8721e0 con 0x1db5c420 -6867> 2014-09-18 18:08:28.616066 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6840/24959 122 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x24fd21c0 con 0x1db5c160 -6866> 2014-09-18 18:08:28.616077 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6826/2318 122 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x121150c0 con 0x1db5c6e0 -6865> 2014-09-18 18:08:28.616090 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6832/2318 122 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x2b922960 con 0x1db5e260 -6864> 2014-09-18 18:08:28.616101 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6837/27583 102 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xfae6000 con 0x10db26e0 -6863> 2014-09-18 18:08:28.616113 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6838/27583 102 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf0e52a0 con 0x10db4ec0 -6862> 2014-09-18 18:08:28.616124 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6826/11463 1575 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x21cfa1e0 con 0xe417b20 -6861> 2014-09-18 18:08:28.616135 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6827/11463 1575 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x19f24940 con 0xe58ab00 -6860> 2014-09-18 18:08:28.616146 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6825/33871 1575 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xe9583a0 con 0xe58a840 -6859> 2014-09-18 18:08:28.616156 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6830/19404 1575 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xe1a52a0 con 0xe58d180 -6858> 2014-09-18 18:08:28.616168 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6831/19404 1575 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x11b0ffe0 con 0xe58d020 -6857> 2014-09-18 18:08:28.616179 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6840/36458 655 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf12b680 con 0x1364b700 -6856> 2014-09-18 18:08:28.616190 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6842/36458 655 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf4e9480 con 0x1364b860 -6855> 2014-09-18 18:08:28.616201 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6805/2487 330 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0x1368cd20 con 0x1d996aa0 -6854> 2014-09-18 18:08:28.616213 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6809/2487 330 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xec6b480 con 0x1d996c00 -6853> 2014-09-18 18:08:28.616224 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6813/17421 186 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.614522) v2 ==== 47+0+0 (1581497526 0 0) 0xf0e30e0 con 0x125239c0 -6852> 2014-09-18 18:08:28.639507 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.24 10.242.42.172:0/5612 112 ==== osd_ping(ping e11706 stamp 2014-09-18 23:37:17.743515) v2 ==== 47+0+0 (2440981019 0 0) 0x12c13c00 con 0x110bb020 -6851> 2014-09-18 18:08:28.639536 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11706 stamp 2014-09-18 23:37:17.743515) v2 -- ?+0 0xf786b40 con 0x110bb020 -6850> 2014-09-18 18:08:28.639532 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.24 10.242.42.172:0/5612 112 ==== osd_ping(ping e11706 stamp 2014-09-18 23:37:17.743515) v2 ==== 47+0+0 (2440981019 0 0) 0x104134a0 con 0x10a5bb80 -6849> 2014-09-18 18:08:28.639562 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11706 stamp 2014-09-18 23:37:17.743515) v2 -- ?+0 0x10507860 con 0x10a5bb80 -6848> 2014-09-18 18:08:28.705753 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.43 10.242.42.178:0/4511 146 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.705212) v2 ==== 47+0+0 (1075793614 0 0) 0xf09fc20 con 0xcc76dc0 -6847> 2014-09-18 18:08:28.705794 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/4511 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.705212) v2 -- ?+0 0x104134a0 con 0xcc76dc0 -6846> 2014-09-18 18:08:28.705778 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.43 10.242.42.178:0/4511 146 ==== osd_ping(ping e11706 stamp 2014-09-18 18:08:28.705212) v2 ==== 47+0+0 (1075793614 0 0) 0x21cfc3a0 con 0x17de6000 -6845> 2014-09-18 18:08:28.705975 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/4511 -- osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:28.705212) v2 -- ?+0 0x12c13c00 con 0x17de6000 -6844> 2014-09-18 18:08:28.707699 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.23 10.242.42.172:0/11463 1548 ==== osd_ping(ping e11706 stamp 2014-09-18 23:37:17.811801) v2 ==== 47+0+0 (1574079866 0 0) 0xe1a5a20 con 0xec5c160 -6843> 2014-09-18 18:08:28.707721 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/11463 -- osd_ping(ping_reply e11706 stamp 2014-09-18 23:37:17.811801) v2 -- ?+0 0xf09fc20 con 0xec5c160 -6842> 2014-09-18 18:08:28.707727 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.23 10.242.42.172:0/11463 1548 ==== osd_ping(ping e11706 stamp 2014-09-18 23:37:17.811801) v2 ==== 47+0+0 (1574079866 0 0) 0xfcc8960 con 0xe567080 -6841> 2014-09-18 18:08:28.707747 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/11463 -- osd_ping(ping_reply e11706 stamp 2014-09-18 23:37:17.811801) v2 -- ?+0 0x21cfc3a0 con 0xe567080 -6840> 2014-09-18 18:08:28.707924 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1260 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0xeeebcc0 con 0xe8db760 -6839> 2014-09-18 18:08:28.707972 7f4c9f251700 3 osd.42 11706 handle_osd_map epochs [11707,11707], i have 11706, src has [1,11707] -6838> 2014-09-18 18:08:28.710583 7f4c9f251700 1 -- 10.242.42.178:6807/2483 mark_down 10.242.42.164:6884/38815 -- 0x10b62580 -6837> 2014-09-18 18:08:28.710610 7f4c9f251700 1 -- 10.242.42.178:0/2483 mark_down 0x21de0940 -- 0xe5482c0 -6836> 2014-09-18 18:08:28.710626 7f4c9f251700 1 -- 10.242.42.178:0/2483 mark_down 0x21dde2c0 -- 0xe548b00 -6835> 2014-09-18 18:08:28.711368 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xeee9f80 con 0xdd8c940 -6834> 2014-09-18 18:08:28.711396 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- pg_notify(2.280(6) epoch 11707) v5 -- ?+0 0xf106d20 con 0xdd8c940 -6833> 2014-09-18 18:08:28.711417 7f4c9f251700 1 osd.42 11707 ms_handle_reset con 0x12b3a680 session 0xf148580 -6832> 2014-09-18 18:08:28.711428 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1261 ==== pg_notify(2.17e(17) epoch 11707) v5 ==== 1593+0+0 (224007619 0 0) 0xf109840 con 0xe8db760 -6831> 2014-09-18 18:08:28.711455 7f4c9f251700 5 -- op tracker -- , seq: 85823, time: 2014-09-18 18:08:28.707923, event: header_read, op: pg_notify(2.17e(17) epoch 11707) -6830> 2014-09-18 18:08:28.711469 7f4c9f251700 5 -- op tracker -- , seq: 85823, time: 2014-09-18 18:08:28.707924, event: throttled, op: pg_notify(2.17e(17) epoch 11707) -6829> 2014-09-18 18:08:28.711476 7f4c9f251700 5 -- op tracker -- , seq: 85823, time: 2014-09-18 18:08:28.708007, event: all_read, op: pg_notify(2.17e(17) epoch 11707) -6828> 2014-09-18 18:08:28.711514 7f4c9f251700 5 -- op tracker -- , seq: 85823, time: 2014-09-18 18:08:28.711449, event: dispatched, op: pg_notify(2.17e(17) epoch 11707) -6827> 2014-09-18 18:08:28.711524 7f4c9f251700 5 -- op tracker -- , seq: 85823, time: 2014-09-18 18:08:28.711522, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11707) -6826> 2014-09-18 18:08:28.711546 7f4c9f251700 5 -- op tracker -- , seq: 85823, time: 2014-09-18 18:08:28.711543, event: started, op: pg_notify(2.17e(17) epoch 11707) -6825> 2014-09-18 18:08:28.711669 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xeeeb840 con 0x1395f1e0 -6824> 2014-09-18 18:08:28.711703 7f4c9f251700 5 -- op tracker -- , seq: 85823, time: 2014-09-18 18:08:28.711699, event: done, op: pg_notify(2.17e(17) epoch 11707) -6823> 2014-09-18 18:08:28.711760 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11707) v5 -- ?+0 0xf09b0e0 con 0x1395f1e0 -6822> 2014-09-18 18:08:28.711772 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2592 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0x19df0fc0 con 0x4900b00 -6821> 2014-09-18 18:08:28.711797 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6820> 2014-09-18 18:08:28.711806 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2593 ==== pg_notify(2.71c(12) epoch 11707) v5 ==== 1424+0+0 (1411448511 0 0) 0xf2450c0 con 0x4900b00 -6819> 2014-09-18 18:08:28.711930 7f4c9f251700 5 -- op tracker -- , seq: 85824, time: 2014-09-18 18:08:28.710423, event: header_read, op: pg_notify(2.71c(12) epoch 11707) -6818> 2014-09-18 18:08:28.711943 7f4c9f251700 5 -- op tracker -- , seq: 85824, time: 2014-09-18 18:08:28.710424, event: throttled, op: pg_notify(2.71c(12) epoch 11707) -6817> 2014-09-18 18:08:28.711950 7f4c9f251700 5 -- op tracker -- , seq: 85824, time: 2014-09-18 18:08:28.710463, event: all_read, op: pg_notify(2.71c(12) epoch 11707) -6816> 2014-09-18 18:08:28.711957 7f4c9f251700 5 -- op tracker -- , seq: 85824, time: 2014-09-18 18:08:28.711923, event: dispatched, op: pg_notify(2.71c(12) epoch 11707) -6815> 2014-09-18 18:08:28.711964 7f4c9f251700 5 -- op tracker -- , seq: 85824, time: 2014-09-18 18:08:28.711961, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11707) -6814> 2014-09-18 18:08:28.711972 7f4c9f251700 5 -- op tracker -- , seq: 85824, time: 2014-09-18 18:08:28.711970, event: started, op: pg_notify(2.71c(12) epoch 11707) -6813> 2014-09-18 18:08:28.712031 7f4c9f251700 5 -- op tracker -- , seq: 85824, time: 2014-09-18 18:08:28.712027, event: done, op: pg_notify(2.71c(12) epoch 11707) -6812> 2014-09-18 18:08:28.712254 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xf953680 con 0x140f95a0 -6811> 2014-09-18 18:08:28.712270 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11707) v5 -- ?+0 0x21e11e00 con 0x140f95a0 -6810> 2014-09-18 18:08:28.712301 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xeeebcc0 con 0x1395edc0 -6809> 2014-09-18 18:08:28.712311 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11707) v5 -- ?+0 0x21e110e0 con 0x1395edc0 -6808> 2014-09-18 18:08:28.712461 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11707) v5 -- ?+0 0xf811de0 con 0x140f95a0 -6807> 2014-09-18 18:08:28.712767 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 3.656002 13 0.000086 -6806> 2014-09-18 18:08:28.712891 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 3.802206 0 0.000000 -6805> 2014-09-18 18:08:28.712913 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 4.793348 0 0.000000 -6804> 2014-09-18 18:08:28.712986 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+remapped] exit Started 4.793455 0 0.000000 -6803> 2014-09-18 18:08:28.712998 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11702/11702/11258) [27,42,1]/[42,35] r=0 lpr=11702 pi=5697-11701/153 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -6802> 2014-09-18 18:08:28.713037 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xeee8b40 con 0x1395ef20 -6801> 2014-09-18 18:08:28.713040 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000041 1 0.000270 -6800> 2014-09-18 18:08:28.713050 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started -6799> 2014-09-18 18:08:28.713052 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(10) epoch 11707) v5 -- ?+0 0x21e12580 con 0x1395ef20 -6798> 2014-09-18 18:08:28.713060 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Start -6797> 2014-09-18 18:08:28.713100 7f4c96a40700 1 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -6796> 2014-09-18 18:08:28.713111 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000051 0 0.000000 -6795> 2014-09-18 18:08:28.713121 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -6794> 2014-09-18 18:08:28.713130 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -6793> 2014-09-18 18:08:28.713139 7f4c96a40700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -6792> 2014-09-18 18:08:28.713758 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -6791> 2014-09-18 18:08:28.713767 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_alive(want up_thru 11707 have 11707) v1 -- ?+0 0xf15e5a0 con 0x111771e0 -6790> 2014-09-18 18:08:28.713840 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xcc00240 con 0x1395f8c0 -6789> 2014-09-18 18:08:28.713905 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.78f epoch 11707) v3 -- ?+0 0xf1f1840 con 0x1395f8c0 -6788> 2014-09-18 18:08:28.713947 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3286 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0x12bf1440 con 0x4900580 -6787> 2014-09-18 18:08:28.713964 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6786> 2014-09-18 18:08:28.714038 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xcc03cc0 con 0xcbecec0 -6785> 2014-09-18 18:08:28.714058 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_query(2.78f epoch 11707) v3 -- ?+0 0xe0c92a0 con 0xcbecec0 -6784> 2014-09-18 18:08:28.714156 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1034 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0xf955a80 con 0x14358260 -6783> 2014-09-18 18:08:28.714172 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6782> 2014-09-18 18:08:28.714224 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3287 ==== pg_notify(2.4f6(184) epoch 11707) v5 ==== 11296+0+0 (564961195 0 0) 0x1393cd00 con 0x4900580 -6781> 2014-09-18 18:08:28.714225 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0x1182b8c0 con 0x23a74f20 -6780> 2014-09-18 18:08:28.714247 7f4c9f251700 5 -- op tracker -- , seq: 85825, time: 2014-09-18 18:08:28.713975, event: header_read, op: pg_notify(2.4f6(184) epoch 11707) -6779> 2014-09-18 18:08:28.714262 7f4c9f251700 5 -- op tracker -- , seq: 85825, time: 2014-09-18 18:08:28.713976, event: throttled, op: pg_notify(2.4f6(184) epoch 11707) -6778> 2014-09-18 18:08:28.714265 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11707) v5 -- ?+0 0x11bc1480 con 0x23a74f20 -6777> 2014-09-18 18:08:28.714273 7f4c9f251700 5 -- op tracker -- , seq: 85825, time: 2014-09-18 18:08:28.714191, event: all_read, op: pg_notify(2.4f6(184) epoch 11707) -6776> 2014-09-18 18:08:28.714283 7f4c9f251700 5 -- op tracker -- , seq: 85825, time: 2014-09-18 18:08:28.714242, event: dispatched, op: pg_notify(2.4f6(184) epoch 11707) -6775> 2014-09-18 18:08:28.714293 7f4c9f251700 5 -- op tracker -- , seq: 85825, time: 2014-09-18 18:08:28.714291, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11707) -6774> 2014-09-18 18:08:28.714306 7f4c9f251700 5 -- op tracker -- , seq: 85825, time: 2014-09-18 18:08:28.714304, event: started, op: pg_notify(2.4f6(184) epoch 11707) -6773> 2014-09-18 18:08:28.714344 7f4c9f251700 5 -- op tracker -- , seq: 85825, time: 2014-09-18 18:08:28.714341, event: done, op: pg_notify(2.4f6(184) epoch 11707) -6772> 2014-09-18 18:08:28.714368 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1035 ==== pg_notify(2.719(17) epoch 11707) v5 ==== 1601+0+0 (3423989217 0 0) 0xf72f2c0 con 0x14358260 -6771> 2014-09-18 18:08:28.714380 7f4c9f251700 5 -- op tracker -- , seq: 85826, time: 2014-09-18 18:08:28.714156, event: header_read, op: pg_notify(2.719(17) epoch 11707) -6770> 2014-09-18 18:08:28.714434 7f4c9f251700 5 -- op tracker -- , seq: 85826, time: 2014-09-18 18:08:28.714156, event: throttled, op: pg_notify(2.719(17) epoch 11707) -6769> 2014-09-18 18:08:28.714445 7f4c9f251700 5 -- op tracker -- , seq: 85826, time: 2014-09-18 18:08:28.714211, event: all_read, op: pg_notify(2.719(17) epoch 11707) -6768> 2014-09-18 18:08:28.714468 7f4c9f251700 5 -- op tracker -- , seq: 85826, time: 2014-09-18 18:08:28.714377, event: dispatched, op: pg_notify(2.719(17) epoch 11707) -6767> 2014-09-18 18:08:28.714480 7f4c9f251700 5 -- op tracker -- , seq: 85826, time: 2014-09-18 18:08:28.714478, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11707) -6766> 2014-09-18 18:08:28.714489 7f4c9f251700 5 -- op tracker -- , seq: 85826, time: 2014-09-18 18:08:28.714488, event: started, op: pg_notify(2.719(17) epoch 11707) -6765> 2014-09-18 18:08:28.714590 7f4c9f251700 5 -- op tracker -- , seq: 85826, time: 2014-09-18 18:08:28.714585, event: done, op: pg_notify(2.719(17) epoch 11707) -6764> 2014-09-18 18:08:28.714937 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3620 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0xcc03cc0 con 0xcbecec0 -6763> 2014-09-18 18:08:28.714958 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6762> 2014-09-18 18:08:28.714966 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3621 ==== pg_notify(2.4f6(12) epoch 11707) v5 ==== 1382+0+0 (2266853236 0 0) 0xe0c92a0 con 0xcbecec0 -6761> 2014-09-18 18:08:28.714978 7f4c9f251700 5 -- op tracker -- , seq: 85827, time: 2014-09-18 18:08:28.714863, event: header_read, op: pg_notify(2.4f6(12) epoch 11707) -6760> 2014-09-18 18:08:28.714993 7f4c9f251700 5 -- op tracker -- , seq: 85827, time: 2014-09-18 18:08:28.714864, event: throttled, op: pg_notify(2.4f6(12) epoch 11707) -6759> 2014-09-18 18:08:28.715001 7f4c9f251700 5 -- op tracker -- , seq: 85827, time: 2014-09-18 18:08:28.714908, event: all_read, op: pg_notify(2.4f6(12) epoch 11707) -6758> 2014-09-18 18:08:28.715007 7f4c9f251700 5 -- op tracker -- , seq: 85827, time: 2014-09-18 18:08:28.714974, event: dispatched, op: pg_notify(2.4f6(12) epoch 11707) -6757> 2014-09-18 18:08:28.715016 7f4c9f251700 5 -- op tracker -- , seq: 85827, time: 2014-09-18 18:08:28.715013, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11707) -6756> 2014-09-18 18:08:28.715025 7f4c9f251700 5 -- op tracker -- , seq: 85827, time: 2014-09-18 18:08:28.715022, event: started, op: pg_notify(2.4f6(12) epoch 11707) -6755> 2014-09-18 18:08:28.715051 7f4c9f251700 5 -- op tracker -- , seq: 85827, time: 2014-09-18 18:08:28.715048, event: done, op: pg_notify(2.4f6(12) epoch 11707) -6754> 2014-09-18 18:08:28.715367 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3288 ==== pg_notify(2.17e(17) epoch 11707) v5 ==== 1593+0+0 (224007619 0 0) 0x1393d2a0 con 0x4900580 -6753> 2014-09-18 18:08:28.715387 7f4c9f251700 5 -- op tracker -- , seq: 85828, time: 2014-09-18 18:08:28.715294, event: header_read, op: pg_notify(2.17e(17) epoch 11707) -6752> 2014-09-18 18:08:28.715396 7f4c9f251700 5 -- op tracker -- , seq: 85828, time: 2014-09-18 18:08:28.715295, event: throttled, op: pg_notify(2.17e(17) epoch 11707) -6751> 2014-09-18 18:08:28.715404 7f4c9f251700 5 -- op tracker -- , seq: 85828, time: 2014-09-18 18:08:28.715348, event: all_read, op: pg_notify(2.17e(17) epoch 11707) -6750> 2014-09-18 18:08:28.715411 7f4c9f251700 5 -- op tracker -- , seq: 85828, time: 2014-09-18 18:08:28.715382, event: dispatched, op: pg_notify(2.17e(17) epoch 11707) -6749> 2014-09-18 18:08:28.715419 7f4c9f251700 5 -- op tracker -- , seq: 85828, time: 2014-09-18 18:08:28.715417, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11707) -6748> 2014-09-18 18:08:28.715427 7f4c9f251700 5 -- op tracker -- , seq: 85828, time: 2014-09-18 18:08:28.715425, event: started, op: pg_notify(2.17e(17) epoch 11707) -6747> 2014-09-18 18:08:28.715453 7f4c9f251700 5 -- op tracker -- , seq: 85828, time: 2014-09-18 18:08:28.715450, event: done, op: pg_notify(2.17e(17) epoch 11707) -6746> 2014-09-18 18:08:28.716542 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3289 ==== pg_notify(2.1bb(9) epoch 11707) v5 ==== 1189+0+0 (123493290 0 0) 0x1393c940 con 0x4900580 -6745> 2014-09-18 18:08:28.716562 7f4c9f251700 5 -- op tracker -- , seq: 85829, time: 2014-09-18 18:08:28.716484, event: header_read, op: pg_notify(2.1bb(9) epoch 11707) -6744> 2014-09-18 18:08:28.716572 7f4c9f251700 5 -- op tracker -- , seq: 85829, time: 2014-09-18 18:08:28.716485, event: throttled, op: pg_notify(2.1bb(9) epoch 11707) -6743> 2014-09-18 18:08:28.716576 7f4c9f251700 5 -- op tracker -- , seq: 85829, time: 2014-09-18 18:08:28.716525, event: all_read, op: pg_notify(2.1bb(9) epoch 11707) -6742> 2014-09-18 18:08:28.716583 7f4c9f251700 5 -- op tracker -- , seq: 85829, time: 2014-09-18 18:08:28.716557, event: dispatched, op: pg_notify(2.1bb(9) epoch 11707) -6741> 2014-09-18 18:08:28.716590 7f4c9f251700 5 -- op tracker -- , seq: 85829, time: 2014-09-18 18:08:28.716589, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11707) -6740> 2014-09-18 18:08:28.716597 7f4c9f251700 5 -- op tracker -- , seq: 85829, time: 2014-09-18 18:08:28.716595, event: started, op: pg_notify(2.1bb(9) epoch 11707) -6739> 2014-09-18 18:08:28.716626 7f4c9f251700 5 -- op tracker -- , seq: 85829, time: 2014-09-18 18:08:28.716624, event: done, op: pg_notify(2.1bb(9) epoch 11707) -6738> 2014-09-18 18:08:28.717480 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 525 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0xeee8b40 con 0x1395ef20 -6737> 2014-09-18 18:08:28.717500 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6736> 2014-09-18 18:08:28.717532 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 526 ==== pg_notify(2.4d0(10) epoch 11707) v5 ==== 1292+0+0 (1632111770 0 0) 0x21e12580 con 0x1395ef20 -6735> 2014-09-18 18:08:28.717554 7f4c9f251700 5 -- op tracker -- , seq: 85830, time: 2014-09-18 18:08:28.717485, event: header_read, op: pg_notify(2.4d0(10) epoch 11707) -6734> 2014-09-18 18:08:28.717563 7f4c9f251700 5 -- op tracker -- , seq: 85830, time: 2014-09-18 18:08:28.717485, event: throttled, op: pg_notify(2.4d0(10) epoch 11707) -6733> 2014-09-18 18:08:28.717596 7f4c9f251700 5 -- op tracker -- , seq: 85830, time: 2014-09-18 18:08:28.717517, event: all_read, op: pg_notify(2.4d0(10) epoch 11707) -6732> 2014-09-18 18:08:28.717604 7f4c9f251700 5 -- op tracker -- , seq: 85830, time: 2014-09-18 18:08:28.717545, event: dispatched, op: pg_notify(2.4d0(10) epoch 11707) -6731> 2014-09-18 18:08:28.717613 7f4c9f251700 5 -- op tracker -- , seq: 85830, time: 2014-09-18 18:08:28.717610, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11707) -6730> 2014-09-18 18:08:28.717622 7f4c9f251700 5 -- op tracker -- , seq: 85830, time: 2014-09-18 18:08:28.717619, event: started, op: pg_notify(2.4d0(10) epoch 11707) -6729> 2014-09-18 18:08:28.717651 7f4c9f251700 5 -- op tracker -- , seq: 85830, time: 2014-09-18 18:08:28.717648, event: done, op: pg_notify(2.4d0(10) epoch 11707) -6728> 2014-09-18 18:08:28.718473 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2007 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0x11175a80 con 0xe2d9a20 -6727> 2014-09-18 18:08:28.718495 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6726> 2014-09-18 18:08:28.718505 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2008 ==== pg_notify(2.26d(9) epoch 11707) v5 ==== 1257+0+0 (2826352793 0 0) 0x100eed20 con 0xe2d9a20 -6725> 2014-09-18 18:08:28.718519 7f4c9f251700 5 -- op tracker -- , seq: 85831, time: 2014-09-18 18:08:28.717837, event: header_read, op: pg_notify(2.26d(9) epoch 11707) -6724> 2014-09-18 18:08:28.718544 7f4c9f251700 5 -- op tracker -- , seq: 85831, time: 2014-09-18 18:08:28.717837, event: throttled, op: pg_notify(2.26d(9) epoch 11707) -6723> 2014-09-18 18:08:28.718555 7f4c9f251700 5 -- op tracker -- , seq: 85831, time: 2014-09-18 18:08:28.717876, event: all_read, op: pg_notify(2.26d(9) epoch 11707) -6722> 2014-09-18 18:08:28.718565 7f4c9f251700 5 -- op tracker -- , seq: 85831, time: 2014-09-18 18:08:28.718515, event: dispatched, op: pg_notify(2.26d(9) epoch 11707) -6721> 2014-09-18 18:08:28.718575 7f4c9f251700 5 -- op tracker -- , seq: 85831, time: 2014-09-18 18:08:28.718573, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11707) -6720> 2014-09-18 18:08:28.718587 7f4c9f251700 5 -- op tracker -- , seq: 85831, time: 2014-09-18 18:08:28.718585, event: started, op: pg_notify(2.26d(9) epoch 11707) -6719> 2014-09-18 18:08:28.718614 7f4c9f251700 5 -- op tracker -- , seq: 85831, time: 2014-09-18 18:08:28.718612, event: done, op: pg_notify(2.26d(9) epoch 11707) -6718> 2014-09-18 18:08:28.718631 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1719 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0xef593c0 con 0x4900000 -6717> 2014-09-18 18:08:28.718644 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6716> 2014-09-18 18:08:28.718667 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 920 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0xcc00240 con 0x1395f8c0 -6715> 2014-09-18 18:08:28.718679 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6714> 2014-09-18 18:08:28.718826 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 921 ==== pg_notify(2.78f(154) epoch 11707) v5 ==== 9280+0+0 (3668361535 0 0) 0xf1f1840 con 0x1395f8c0 -6713> 2014-09-18 18:08:28.718847 7f4c9f251700 5 -- op tracker -- , seq: 85832, time: 2014-09-18 18:08:28.718587, event: header_read, op: pg_notify(2.78f(154) epoch 11707) -6712> 2014-09-18 18:08:28.718857 7f4c9f251700 5 -- op tracker -- , seq: 85832, time: 2014-09-18 18:08:28.718588, event: throttled, op: pg_notify(2.78f(154) epoch 11707) -6711> 2014-09-18 18:08:28.718865 7f4c9f251700 5 -- op tracker -- , seq: 85832, time: 2014-09-18 18:08:28.718808, event: all_read, op: pg_notify(2.78f(154) epoch 11707) -6710> 2014-09-18 18:08:28.718872 7f4c9f251700 5 -- op tracker -- , seq: 85832, time: 2014-09-18 18:08:28.718841, event: dispatched, op: pg_notify(2.78f(154) epoch 11707) -6709> 2014-09-18 18:08:28.718879 7f4c9f251700 5 -- op tracker -- , seq: 85832, time: 2014-09-18 18:08:28.718877, event: waiting_for_osdmap, op: pg_notify(2.78f(154) epoch 11707) -6708> 2014-09-18 18:08:28.718888 7f4c9f251700 5 -- op tracker -- , seq: 85832, time: 2014-09-18 18:08:28.718886, event: started, op: pg_notify(2.78f(154) epoch 11707) -6707> 2014-09-18 18:08:28.718913 7f4c9f251700 5 -- op tracker -- , seq: 85832, time: 2014-09-18 18:08:28.718910, event: done, op: pg_notify(2.78f(154) epoch 11707) -6706> 2014-09-18 18:08:28.719584 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 782 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0x2d7fd600 con 0xe9a27e0 -6705> 2014-09-18 18:08:28.719652 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6704> 2014-09-18 18:08:28.719662 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2347 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0xeee9f80 con 0xdd8c940 -6703> 2014-09-18 18:08:28.719674 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6702> 2014-09-18 18:08:28.719680 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1720 ==== pg_notify(2.e0(79) epoch 11707) v5 ==== 5119+0+0 (2729877144 0 0) 0xf1ebc20 con 0x4900000 -6701> 2014-09-18 18:08:28.719694 7f4c9f251700 5 -- op tracker -- , seq: 85833, time: 2014-09-18 18:08:28.719459, event: header_read, op: pg_notify(2.e0(79) epoch 11707) -6700> 2014-09-18 18:08:28.719706 7f4c9f251700 5 -- op tracker -- , seq: 85833, time: 2014-09-18 18:08:28.719459, event: throttled, op: pg_notify(2.e0(79) epoch 11707) -6699> 2014-09-18 18:08:28.719716 7f4c9f251700 5 -- op tracker -- , seq: 85833, time: 2014-09-18 18:08:28.719556, event: all_read, op: pg_notify(2.e0(79) epoch 11707) -6698> 2014-09-18 18:08:28.719726 7f4c9f251700 5 -- op tracker -- , seq: 85833, time: 2014-09-18 18:08:28.719689, event: dispatched, op: pg_notify(2.e0(79) epoch 11707) -6697> 2014-09-18 18:08:28.719737 7f4c9f251700 5 -- op tracker -- , seq: 85833, time: 2014-09-18 18:08:28.719734, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11707) -6696> 2014-09-18 18:08:28.719748 7f4c9f251700 5 -- op tracker -- , seq: 85833, time: 2014-09-18 18:08:28.719745, event: started, op: pg_notify(2.e0(79) epoch 11707) -6695> 2014-09-18 18:08:28.719772 7f4c9f251700 5 -- op tracker -- , seq: 85833, time: 2014-09-18 18:08:28.719771, event: done, op: pg_notify(2.e0(79) epoch 11707) -6694> 2014-09-18 18:08:28.719825 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 783 ==== pg_notify(2.2d8(173) epoch 11707) v5 ==== 10757+0+0 (3754305919 0 0) 0x142212a0 con 0xe9a27e0 -6693> 2014-09-18 18:08:28.719845 7f4c9f251700 5 -- op tracker -- , seq: 85834, time: 2014-09-18 18:08:28.719637, event: header_read, op: pg_notify(2.2d8(173) epoch 11707) -6692> 2014-09-18 18:08:28.719854 7f4c9f251700 5 -- op tracker -- , seq: 85834, time: 2014-09-18 18:08:28.719637, event: throttled, op: pg_notify(2.2d8(173) epoch 11707) -6691> 2014-09-18 18:08:28.719861 7f4c9f251700 5 -- op tracker -- , seq: 85834, time: 2014-09-18 18:08:28.719812, event: all_read, op: pg_notify(2.2d8(173) epoch 11707) -6690> 2014-09-18 18:08:28.719869 7f4c9f251700 5 -- op tracker -- , seq: 85834, time: 2014-09-18 18:08:28.719840, event: dispatched, op: pg_notify(2.2d8(173) epoch 11707) -6689> 2014-09-18 18:08:28.719881 7f4c9f251700 5 -- op tracker -- , seq: 85834, time: 2014-09-18 18:08:28.719878, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11707) -6688> 2014-09-18 18:08:28.719893 7f4c9f251700 5 -- op tracker -- , seq: 85834, time: 2014-09-18 18:08:28.719890, event: started, op: pg_notify(2.2d8(173) epoch 11707) -6687> 2014-09-18 18:08:28.719916 7f4c9f251700 5 -- op tracker -- , seq: 85834, time: 2014-09-18 18:08:28.719915, event: done, op: pg_notify(2.2d8(173) epoch 11707) -6686> 2014-09-18 18:08:28.719938 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2348 ==== pg_notify(2.2c7(139) epoch 11707) v5 ==== 8651+0+0 (4179231680 0 0) 0xf106d20 con 0xdd8c940 -6685> 2014-09-18 18:08:28.719951 7f4c9f251700 5 -- op tracker -- , seq: 85835, time: 2014-09-18 18:08:28.719556, event: header_read, op: pg_notify(2.2c7(139) epoch 11707) -6684> 2014-09-18 18:08:28.719962 7f4c9f251700 5 -- op tracker -- , seq: 85835, time: 2014-09-18 18:08:28.719558, event: throttled, op: pg_notify(2.2c7(139) epoch 11707) -6683> 2014-09-18 18:08:28.719969 7f4c9f251700 5 -- op tracker -- , seq: 85835, time: 2014-09-18 18:08:28.719821, event: all_read, op: pg_notify(2.2c7(139) epoch 11707) -6682> 2014-09-18 18:08:28.719979 7f4c9f251700 5 -- op tracker -- , seq: 85835, time: 2014-09-18 18:08:28.719947, event: dispatched, op: pg_notify(2.2c7(139) epoch 11707) -6681> 2014-09-18 18:08:28.719990 7f4c9f251700 5 -- op tracker -- , seq: 85835, time: 2014-09-18 18:08:28.719987, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11707) -6680> 2014-09-18 18:08:28.720003 7f4c9f251700 5 -- op tracker -- , seq: 85835, time: 2014-09-18 18:08:28.720000, event: started, op: pg_notify(2.2c7(139) epoch 11707) -6679> 2014-09-18 18:08:28.720026 7f4c9f251700 5 -- op tracker -- , seq: 85835, time: 2014-09-18 18:08:28.720025, event: done, op: pg_notify(2.2c7(139) epoch 11707) -6678> 2014-09-18 18:08:28.720048 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3622 ==== pg_notify(2.3b6(10) epoch 11707) v5 ==== 1260+0+0 (4224248248 0 0) 0xe1adc00 con 0xcbecec0 -6677> 2014-09-18 18:08:28.720061 7f4c9f251700 5 -- op tracker -- , seq: 85836, time: 2014-09-18 18:08:28.719949, event: header_read, op: pg_notify(2.3b6(10) epoch 11707) -6676> 2014-09-18 18:08:28.720072 7f4c9f251700 5 -- op tracker -- , seq: 85836, time: 2014-09-18 18:08:28.719950, event: throttled, op: pg_notify(2.3b6(10) epoch 11707) -6675> 2014-09-18 18:08:28.720082 7f4c9f251700 5 -- op tracker -- , seq: 85836, time: 2014-09-18 18:08:28.720009, event: all_read, op: pg_notify(2.3b6(10) epoch 11707) -6674> 2014-09-18 18:08:28.720092 7f4c9f251700 5 -- op tracker -- , seq: 85836, time: 2014-09-18 18:08:28.720057, event: dispatched, op: pg_notify(2.3b6(10) epoch 11707) -6673> 2014-09-18 18:08:28.720103 7f4c9f251700 5 -- op tracker -- , seq: 85836, time: 2014-09-18 18:08:28.720100, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11707) -6672> 2014-09-18 18:08:28.720115 7f4c9f251700 5 -- op tracker -- , seq: 85836, time: 2014-09-18 18:08:28.720112, event: started, op: pg_notify(2.3b6(10) epoch 11707) -6671> 2014-09-18 18:08:28.720137 7f4c9f251700 5 -- op tracker -- , seq: 85836, time: 2014-09-18 18:08:28.720136, event: done, op: pg_notify(2.3b6(10) epoch 11707) -6670> 2014-09-18 18:08:28.720176 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2349 ==== pg_notify(2.457(21) epoch 11707) v5 ==== 1875+0+0 (962822206 0 0) 0xf2b90c0 con 0xdd8c940 -6669> 2014-09-18 18:08:28.720190 7f4c9f251700 5 -- op tracker -- , seq: 85837, time: 2014-09-18 18:08:28.720085, event: header_read, op: pg_notify(2.457(21) epoch 11707) -6668> 2014-09-18 18:08:28.720200 7f4c9f251700 5 -- op tracker -- , seq: 85837, time: 2014-09-18 18:08:28.720086, event: throttled, op: pg_notify(2.457(21) epoch 11707) -6667> 2014-09-18 18:08:28.720210 7f4c9f251700 5 -- op tracker -- , seq: 85837, time: 2014-09-18 18:08:28.720125, event: all_read, op: pg_notify(2.457(21) epoch 11707) -6666> 2014-09-18 18:08:28.720217 7f4c9f251700 5 -- op tracker -- , seq: 85837, time: 2014-09-18 18:08:28.720186, event: dispatched, op: pg_notify(2.457(21) epoch 11707) -6665> 2014-09-18 18:08:28.720223 7f4c9f251700 5 -- op tracker -- , seq: 85837, time: 2014-09-18 18:08:28.720221, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11707) -6664> 2014-09-18 18:08:28.720231 7f4c9f251700 5 -- op tracker -- , seq: 85837, time: 2014-09-18 18:08:28.720228, event: started, op: pg_notify(2.457(21) epoch 11707) -6663> 2014-09-18 18:08:28.720252 7f4c9f251700 5 -- op tracker -- , seq: 85837, time: 2014-09-18 18:08:28.720249, event: done, op: pg_notify(2.457(21) epoch 11707) -6662> 2014-09-18 18:08:28.724361 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3623 ==== pg_notify(2.517(11) epoch 11707) v5 ==== 1343+0+0 (4259102304 0 0) 0xe1aa5a0 con 0xcbecec0 -6661> 2014-09-18 18:08:28.724386 7f4c9f251700 5 -- op tracker -- , seq: 85838, time: 2014-09-18 18:08:28.724293, event: header_read, op: pg_notify(2.517(11) epoch 11707) -6660> 2014-09-18 18:08:28.724399 7f4c9f251700 5 -- op tracker -- , seq: 85838, time: 2014-09-18 18:08:28.724295, event: throttled, op: pg_notify(2.517(11) epoch 11707) -6659> 2014-09-18 18:08:28.724409 7f4c9f251700 5 -- op tracker -- , seq: 85838, time: 2014-09-18 18:08:28.724343, event: all_read, op: pg_notify(2.517(11) epoch 11707) -6658> 2014-09-18 18:08:28.724419 7f4c9f251700 5 -- op tracker -- , seq: 85838, time: 2014-09-18 18:08:28.724379, event: dispatched, op: pg_notify(2.517(11) epoch 11707) -6657> 2014-09-18 18:08:28.724430 7f4c9f251700 5 -- op tracker -- , seq: 85838, time: 2014-09-18 18:08:28.724427, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11707) -6656> 2014-09-18 18:08:28.724443 7f4c9f251700 5 -- op tracker -- , seq: 85838, time: 2014-09-18 18:08:28.724440, event: started, op: pg_notify(2.517(11) epoch 11707) -6655> 2014-09-18 18:08:28.724490 7f4c9f251700 5 -- op tracker -- , seq: 85838, time: 2014-09-18 18:08:28.724488, event: done, op: pg_notify(2.517(11) epoch 11707) -6654> 2014-09-18 18:08:28.726092 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3624 ==== pg_notify(2.2c7(139) epoch 11707) v5 ==== 8651+0+0 (950926379 0 0) 0xe1ac3a0 con 0xcbecec0 -6653> 2014-09-18 18:08:28.726110 7f4c9f251700 5 -- op tracker -- , seq: 85839, time: 2014-09-18 18:08:28.725953, event: header_read, op: pg_notify(2.2c7(139) epoch 11707) -6652> 2014-09-18 18:08:28.726121 7f4c9f251700 5 -- op tracker -- , seq: 85839, time: 2014-09-18 18:08:28.725954, event: throttled, op: pg_notify(2.2c7(139) epoch 11707) -6651> 2014-09-18 18:08:28.726128 7f4c9f251700 5 -- op tracker -- , seq: 85839, time: 2014-09-18 18:08:28.726077, event: all_read, op: pg_notify(2.2c7(139) epoch 11707) -6650> 2014-09-18 18:08:28.726135 7f4c9f251700 5 -- op tracker -- , seq: 85839, time: 2014-09-18 18:08:28.726106, event: dispatched, op: pg_notify(2.2c7(139) epoch 11707) -6649> 2014-09-18 18:08:28.726140 7f4c9f251700 5 -- op tracker -- , seq: 85839, time: 2014-09-18 18:08:28.726138, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11707) -6648> 2014-09-18 18:08:28.726148 7f4c9f251700 5 -- op tracker -- , seq: 85839, time: 2014-09-18 18:08:28.726146, event: started, op: pg_notify(2.2c7(139) epoch 11707) -6647> 2014-09-18 18:08:28.726169 7f4c9f251700 5 -- op tracker -- , seq: 85839, time: 2014-09-18 18:08:28.726167, event: done, op: pg_notify(2.2c7(139) epoch 11707) -6646> 2014-09-18 18:08:28.734249 7f4caae41700 5 osd.42 11707 tick -6645> 2014-09-18 18:08:28.736076 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 922 ==== pg_notify(2.78f(154) epoch 11707) v5 ==== 9280+0+0 (3668361535 0 0) 0xe9572c0 con 0x1395f8c0 -6644> 2014-09-18 18:08:28.736099 7f4c9f251700 5 -- op tracker -- , seq: 85840, time: 2014-09-18 18:08:28.735868, event: header_read, op: pg_notify(2.78f(154) epoch 11707) -6643> 2014-09-18 18:08:28.736111 7f4c9f251700 5 -- op tracker -- , seq: 85840, time: 2014-09-18 18:08:28.735873, event: throttled, op: pg_notify(2.78f(154) epoch 11707) -6642> 2014-09-18 18:08:28.736119 7f4c9f251700 5 -- op tracker -- , seq: 85840, time: 2014-09-18 18:08:28.736053, event: all_read, op: pg_notify(2.78f(154) epoch 11707) -6641> 2014-09-18 18:08:28.736126 7f4c9f251700 5 -- op tracker -- , seq: 85840, time: 2014-09-18 18:08:28.736094, event: dispatched, op: pg_notify(2.78f(154) epoch 11707) -6640> 2014-09-18 18:08:28.736134 7f4c9f251700 5 -- op tracker -- , seq: 85840, time: 2014-09-18 18:08:28.736133, event: waiting_for_osdmap, op: pg_notify(2.78f(154) epoch 11707) -6639> 2014-09-18 18:08:28.736145 7f4c9f251700 5 -- op tracker -- , seq: 85840, time: 2014-09-18 18:08:28.736144, event: started, op: pg_notify(2.78f(154) epoch 11707) -6638> 2014-09-18 18:08:28.736167 7f4c9f251700 5 -- op tracker -- , seq: 85840, time: 2014-09-18 18:08:28.736166, event: done, op: pg_notify(2.78f(154) epoch 11707) -6637> 2014-09-18 18:08:28.739178 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2350 ==== osd pg remove(epoch 11707; pg2.280; ) v2 ==== 30+0+0 (1594435809 0 0) 0xf2b81c0 con 0xdd8c940 -6636> 2014-09-18 18:08:28.739196 7f4c9f251700 5 -- op tracker -- , seq: 85841, time: 2014-09-18 18:08:28.739140, event: header_read, op: osd pg remove(epoch 11707; pg2.280; ) -6635> 2014-09-18 18:08:28.739205 7f4c9f251700 5 -- op tracker -- , seq: 85841, time: 2014-09-18 18:08:28.739141, event: throttled, op: osd pg remove(epoch 11707; pg2.280; ) -6634> 2014-09-18 18:08:28.739213 7f4c9f251700 5 -- op tracker -- , seq: 85841, time: 2014-09-18 18:08:28.739165, event: all_read, op: osd pg remove(epoch 11707; pg2.280; ) -6633> 2014-09-18 18:08:28.739221 7f4c9f251700 5 -- op tracker -- , seq: 85841, time: 2014-09-18 18:08:28.739192, event: dispatched, op: osd pg remove(epoch 11707; pg2.280; ) -6632> 2014-09-18 18:08:28.739228 7f4c9f251700 5 -- op tracker -- , seq: 85841, time: 2014-09-18 18:08:28.739226, event: waiting_for_osdmap, op: osd pg remove(epoch 11707; pg2.280; ) -6631> 2014-09-18 18:08:28.739236 7f4c9f251700 5 -- op tracker -- , seq: 85841, time: 2014-09-18 18:08:28.739235, event: started, op: osd pg remove(epoch 11707; pg2.280; ) -6630> 2014-09-18 18:08:28.739242 7f4c9f251700 5 osd.42 11707 queue_pg_for_deletion: 2.280 -6629> 2014-09-18 18:08:28.739338 7f4c9f251700 5 -- op tracker -- , seq: 85841, time: 2014-09-18 18:08:28.739335, event: done, op: osd pg remove(epoch 11707; pg2.280; ) -6628> 2014-09-18 18:08:28.747625 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 39 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0x11172000 con 0x1395f340 -6627> 2014-09-18 18:08:28.747641 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6626> 2014-09-18 18:08:28.747679 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 40 ==== pg_notify(2.1d8(9) epoch 11707) v5 ==== 1261+0+0 (1785722038 0 0) 0x14220940 con 0x1395f340 -6625> 2014-09-18 18:08:28.747690 7f4c9f251700 5 -- op tracker -- , seq: 85842, time: 2014-09-18 18:08:28.747625, event: header_read, op: pg_notify(2.1d8(9) epoch 11707) -6624> 2014-09-18 18:08:28.747698 7f4c9f251700 5 -- op tracker -- , seq: 85842, time: 2014-09-18 18:08:28.747626, event: throttled, op: pg_notify(2.1d8(9) epoch 11707) -6623> 2014-09-18 18:08:28.747707 7f4c9f251700 5 -- op tracker -- , seq: 85842, time: 2014-09-18 18:08:28.747668, event: all_read, op: pg_notify(2.1d8(9) epoch 11707) -6622> 2014-09-18 18:08:28.747714 7f4c9f251700 5 -- op tracker -- , seq: 85842, time: 2014-09-18 18:08:28.747687, event: dispatched, op: pg_notify(2.1d8(9) epoch 11707) -6621> 2014-09-18 18:08:28.747724 7f4c9f251700 5 -- op tracker -- , seq: 85842, time: 2014-09-18 18:08:28.747723, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11707) -6620> 2014-09-18 18:08:28.747737 7f4c9f251700 5 -- op tracker -- , seq: 85842, time: 2014-09-18 18:08:28.747735, event: started, op: pg_notify(2.1d8(9) epoch 11707) -6619> 2014-09-18 18:08:28.747755 7f4c9f251700 5 -- op tracker -- , seq: 85842, time: 2014-09-18 18:08:28.747754, event: done, op: pg_notify(2.1d8(9) epoch 11707) -6618> 2014-09-18 18:08:28.798038 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1192 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0x1182a240 con 0x1088b440 -6617> 2014-09-18 18:08:28.798073 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6616> 2014-09-18 18:08:28.798342 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1193 ==== pg_notify(2.2d8(173) epoch 11707) v5 ==== 10757+0+0 (3754305919 0 0) 0x1390a5a0 con 0x1088b440 -6615> 2014-09-18 18:08:28.798363 7f4c9f251700 5 -- op tracker -- , seq: 85843, time: 2014-09-18 18:08:28.798139, event: header_read, op: pg_notify(2.2d8(173) epoch 11707) -6614> 2014-09-18 18:08:28.798373 7f4c9f251700 5 -- op tracker -- , seq: 85843, time: 2014-09-18 18:08:28.798140, event: throttled, op: pg_notify(2.2d8(173) epoch 11707) -6613> 2014-09-18 18:08:28.798377 7f4c9f251700 5 -- op tracker -- , seq: 85843, time: 2014-09-18 18:08:28.798328, event: all_read, op: pg_notify(2.2d8(173) epoch 11707) -6612> 2014-09-18 18:08:28.798381 7f4c9f251700 5 -- op tracker -- , seq: 85843, time: 2014-09-18 18:08:28.798358, event: dispatched, op: pg_notify(2.2d8(173) epoch 11707) -6611> 2014-09-18 18:08:28.798386 7f4c9f251700 5 -- op tracker -- , seq: 85843, time: 2014-09-18 18:08:28.798384, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11707) -6610> 2014-09-18 18:08:28.798392 7f4c9f251700 5 -- op tracker -- , seq: 85843, time: 2014-09-18 18:08:28.798391, event: started, op: pg_notify(2.2d8(173) epoch 11707) -6609> 2014-09-18 18:08:28.798418 7f4c9f251700 5 -- op tracker -- , seq: 85843, time: 2014-09-18 18:08:28.798417, event: done, op: pg_notify(2.2d8(173) epoch 11707) -6608> 2014-09-18 18:08:28.800404 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3625 ==== pg_notify(2.40a(162) epoch 11707) v5 ==== 9736+0+0 (349047325 0 0) 0xe1ada20 con 0xcbecec0 -6607> 2014-09-18 18:08:28.800423 7f4c9f251700 5 -- op tracker -- , seq: 85844, time: 2014-09-18 18:08:28.800261, event: header_read, op: pg_notify(2.40a(162) epoch 11707) -6606> 2014-09-18 18:08:28.800437 7f4c9f251700 5 -- op tracker -- , seq: 85844, time: 2014-09-18 18:08:28.800262, event: throttled, op: pg_notify(2.40a(162) epoch 11707) -6605> 2014-09-18 18:08:28.800445 7f4c9f251700 5 -- op tracker -- , seq: 85844, time: 2014-09-18 18:08:28.800378, event: all_read, op: pg_notify(2.40a(162) epoch 11707) -6604> 2014-09-18 18:08:28.800451 7f4c9f251700 5 -- op tracker -- , seq: 85844, time: 2014-09-18 18:08:28.800419, event: dispatched, op: pg_notify(2.40a(162) epoch 11707) -6603> 2014-09-18 18:08:28.800458 7f4c9f251700 5 -- op tracker -- , seq: 85844, time: 2014-09-18 18:08:28.800457, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11707) -6602> 2014-09-18 18:08:28.800466 7f4c9f251700 5 -- op tracker -- , seq: 85844, time: 2014-09-18 18:08:28.800465, event: started, op: pg_notify(2.40a(162) epoch 11707) -6601> 2014-09-18 18:08:28.800494 7f4c9f251700 5 -- op tracker -- , seq: 85844, time: 2014-09-18 18:08:28.800492, event: done, op: pg_notify(2.40a(162) epoch 11707) -6600> 2014-09-18 18:08:28.820090 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3172 ==== osd_map(11707..11707 src has 1..11707) v3 ==== 309+0+0 (2826630847 0 0) 0x15e32f40 con 0x1395ec60 -6599> 2014-09-18 18:08:28.820117 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11707,11707], i have 11707, src has [1,11707] -6598> 2014-09-18 18:08:28.820205 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3173 ==== pg_notify(2.71(19) epoch 11707) v5 ==== 1795+0+0 (60138472 0 0) 0xf811c00 con 0x1395ec60 -6597> 2014-09-18 18:08:28.820217 7f4c9f251700 5 -- op tracker -- , seq: 85845, time: 2014-09-18 18:08:28.820124, event: header_read, op: pg_notify(2.71(19) epoch 11707) -6596> 2014-09-18 18:08:28.820229 7f4c9f251700 5 -- op tracker -- , seq: 85845, time: 2014-09-18 18:08:28.820125, event: throttled, op: pg_notify(2.71(19) epoch 11707) -6595> 2014-09-18 18:08:28.820238 7f4c9f251700 5 -- op tracker -- , seq: 85845, time: 2014-09-18 18:08:28.820193, event: all_read, op: pg_notify(2.71(19) epoch 11707) -6594> 2014-09-18 18:08:28.820246 7f4c9f251700 5 -- op tracker -- , seq: 85845, time: 2014-09-18 18:08:28.820213, event: dispatched, op: pg_notify(2.71(19) epoch 11707) -6593> 2014-09-18 18:08:28.820254 7f4c9f251700 5 -- op tracker -- , seq: 85845, time: 2014-09-18 18:08:28.820253, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11707) -6592> 2014-09-18 18:08:28.820263 7f4c9f251700 5 -- op tracker -- , seq: 85845, time: 2014-09-18 18:08:28.820262, event: started, op: pg_notify(2.71(19) epoch 11707) -6591> 2014-09-18 18:08:28.820287 7f4c9f251700 5 -- op tracker -- , seq: 85845, time: 2014-09-18 18:08:28.820285, event: done, op: pg_notify(2.71(19) epoch 11707) -6590> 2014-09-18 18:08:28.844436 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3626 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11707) v5 ==== 6174+0+0 (1702438173 0 0) 0xe1ab860 con 0xcbecec0 -6589> 2014-09-18 18:08:28.844457 7f4c9f251700 5 -- op tracker -- , seq: 85846, time: 2014-09-18 18:08:28.844334, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11707) -6588> 2014-09-18 18:08:28.844469 7f4c9f251700 5 -- op tracker -- , seq: 85846, time: 2014-09-18 18:08:28.844335, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11707) -6587> 2014-09-18 18:08:28.844477 7f4c9f251700 5 -- op tracker -- , seq: 85846, time: 2014-09-18 18:08:28.844420, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11707) -6586> 2014-09-18 18:08:28.844485 7f4c9f251700 5 -- op tracker -- , seq: 85846, time: 2014-09-18 18:08:28.844452, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11707) -6585> 2014-09-18 18:08:28.844494 7f4c9f251700 5 -- op tracker -- , seq: 85846, time: 2014-09-18 18:08:28.844492, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11707) -6584> 2014-09-18 18:08:28.844504 7f4c9f251700 5 -- op tracker -- , seq: 85846, time: 2014-09-18 18:08:28.844502, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11707) -6583> 2014-09-18 18:08:28.844547 7f4c9f251700 5 -- op tracker -- , seq: 85846, time: 2014-09-18 18:08:28.844545, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11707) -6582> 2014-09-18 18:08:28.845825 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3627 ==== pg_notify(2.1bb(9) epoch 11707) v5 ==== 1189+0+0 (2882922937 0 0) 0xe1abc20 con 0xcbecec0 -6581> 2014-09-18 18:08:28.845838 7f4c9f251700 5 -- op tracker -- , seq: 85847, time: 2014-09-18 18:08:28.845767, event: header_read, op: pg_notify(2.1bb(9) epoch 11707) -6580> 2014-09-18 18:08:28.845844 7f4c9f251700 5 -- op tracker -- , seq: 85847, time: 2014-09-18 18:08:28.845768, event: throttled, op: pg_notify(2.1bb(9) epoch 11707) -6579> 2014-09-18 18:08:28.845856 7f4c9f251700 5 -- op tracker -- , seq: 85847, time: 2014-09-18 18:08:28.845813, event: all_read, op: pg_notify(2.1bb(9) epoch 11707) -6578> 2014-09-18 18:08:28.845862 7f4c9f251700 5 -- op tracker -- , seq: 85847, time: 2014-09-18 18:08:28.845835, event: dispatched, op: pg_notify(2.1bb(9) epoch 11707) -6577> 2014-09-18 18:08:28.845868 7f4c9f251700 5 -- op tracker -- , seq: 85847, time: 2014-09-18 18:08:28.845867, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11707) -6576> 2014-09-18 18:08:28.845876 7f4c9f251700 5 -- op tracker -- , seq: 85847, time: 2014-09-18 18:08:28.845874, event: started, op: pg_notify(2.1bb(9) epoch 11707) -6575> 2014-09-18 18:08:28.845892 7f4c9f251700 5 -- op tracker -- , seq: 85847, time: 2014-09-18 18:08:28.845891, event: done, op: pg_notify(2.1bb(9) epoch 11707) -6574> 2014-09-18 18:08:28.848855 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3628 ==== pg_notify(2.78f(55) epoch 11707) v5 ==== 3667+0+0 (376043268 0 0) 0xe1acb20 con 0xcbecec0 -6573> 2014-09-18 18:08:28.848873 7f4c9f251700 5 -- op tracker -- , seq: 85848, time: 2014-09-18 18:08:28.848727, event: header_read, op: pg_notify(2.78f(55) epoch 11707) -6572> 2014-09-18 18:08:28.848886 7f4c9f251700 5 -- op tracker -- , seq: 85848, time: 2014-09-18 18:08:28.848728, event: throttled, op: pg_notify(2.78f(55) epoch 11707) -6571> 2014-09-18 18:08:28.848898 7f4c9f251700 5 -- op tracker -- , seq: 85848, time: 2014-09-18 18:08:28.848830, event: all_read, op: pg_notify(2.78f(55) epoch 11707) -6570> 2014-09-18 18:08:28.848905 7f4c9f251700 5 -- op tracker -- , seq: 85848, time: 2014-09-18 18:08:28.848869, event: dispatched, op: pg_notify(2.78f(55) epoch 11707) -6569> 2014-09-18 18:08:28.848912 7f4c9f251700 5 -- op tracker -- , seq: 85848, time: 2014-09-18 18:08:28.848911, event: waiting_for_osdmap, op: pg_notify(2.78f(55) epoch 11707) -6568> 2014-09-18 18:08:28.848921 7f4c9f251700 5 -- op tracker -- , seq: 85848, time: 2014-09-18 18:08:28.848919, event: started, op: pg_notify(2.78f(55) epoch 11707) -6567> 2014-09-18 18:08:28.848940 7f4c9f251700 5 -- op tracker -- , seq: 85848, time: 2014-09-18 18:08:28.848938, event: done, op: pg_notify(2.78f(55) epoch 11707) -6566> 2014-09-18 18:08:28.848961 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.135822 4 0.000232 -6565> 2014-09-18 18:08:28.849041 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -6564> 2014-09-18 18:08:28.849211 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000170 0 0.000000 -6563> 2014-09-18 18:08:28.849335 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -6562> 2014-09-18 18:08:28.849422 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000087 0 0.000000 -6561> 2014-09-18 18:08:28.849438 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -6560> 2014-09-18 18:08:28.850074 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3629 ==== pg_notify(2.78f(55) epoch 11707) v5 ==== 3667+0+0 (376043268 0 0) 0xe1ad660 con 0xcbecec0 -6559> 2014-09-18 18:08:28.850089 7f4c9f251700 5 -- op tracker -- , seq: 85849, time: 2014-09-18 18:08:28.850012, event: header_read, op: pg_notify(2.78f(55) epoch 11707) -6558> 2014-09-18 18:08:28.850098 7f4c9f251700 5 -- op tracker -- , seq: 85849, time: 2014-09-18 18:08:28.850013, event: throttled, op: pg_notify(2.78f(55) epoch 11707) -6557> 2014-09-18 18:08:28.850104 7f4c9f251700 5 -- op tracker -- , seq: 85849, time: 2014-09-18 18:08:28.850061, event: all_read, op: pg_notify(2.78f(55) epoch 11707) -6556> 2014-09-18 18:08:28.850110 7f4c9f251700 5 -- op tracker -- , seq: 85849, time: 2014-09-18 18:08:28.850086, event: dispatched, op: pg_notify(2.78f(55) epoch 11707) -6555> 2014-09-18 18:08:28.850116 7f4c9f251700 5 -- op tracker -- , seq: 85849, time: 2014-09-18 18:08:28.850115, event: waiting_for_osdmap, op: pg_notify(2.78f(55) epoch 11707) -6554> 2014-09-18 18:08:28.850124 7f4c9f251700 5 -- op tracker -- , seq: 85849, time: 2014-09-18 18:08:28.850122, event: started, op: pg_notify(2.78f(55) epoch 11707) -6553> 2014-09-18 18:08:28.850141 7f4c9f251700 5 -- op tracker -- , seq: 85849, time: 2014-09-18 18:08:28.850139, event: done, op: pg_notify(2.78f(55) epoch 11707) -6552> 2014-09-18 18:08:28.901581 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3174 ==== pg_notify(2.410(11) epoch 11707) v5 ==== 1325+0+0 (1642875029 0 0) 0xf80e5a0 con 0x1395ec60 -6551> 2014-09-18 18:08:28.901606 7f4c9f251700 5 -- op tracker -- , seq: 85850, time: 2014-09-18 18:08:28.901436, event: header_read, op: pg_notify(2.410(11) epoch 11707) -6550> 2014-09-18 18:08:28.901626 7f4c9f251700 5 -- op tracker -- , seq: 85850, time: 2014-09-18 18:08:28.901438, event: throttled, op: pg_notify(2.410(11) epoch 11707) -6549> 2014-09-18 18:08:28.901634 7f4c9f251700 5 -- op tracker -- , seq: 85850, time: 2014-09-18 18:08:28.901532, event: all_read, op: pg_notify(2.410(11) epoch 11707) -6548> 2014-09-18 18:08:28.901645 7f4c9f251700 5 -- op tracker -- , seq: 85850, time: 2014-09-18 18:08:28.901601, event: dispatched, op: pg_notify(2.410(11) epoch 11707) -6547> 2014-09-18 18:08:28.901654 7f4c9f251700 5 -- op tracker -- , seq: 85850, time: 2014-09-18 18:08:28.901653, event: waiting_for_osdmap, op: pg_notify(2.410(11) epoch 11707) -6546> 2014-09-18 18:08:28.901664 7f4c9f251700 5 -- op tracker -- , seq: 85850, time: 2014-09-18 18:08:28.901663, event: started, op: pg_notify(2.410(11) epoch 11707) -6545> 2014-09-18 18:08:28.901686 7f4c9f251700 5 -- op tracker -- , seq: 85850, time: 2014-09-18 18:08:28.901684, event: done, op: pg_notify(2.410(11) epoch 11707) -6544> 2014-09-18 18:08:28.903095 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3175 ==== pg_notify(2.186(12) epoch 11707) v5 ==== 1386+0+0 (1610244396 0 0) 0xf810b20 con 0x1395ec60 -6543> 2014-09-18 18:08:28.903113 7f4c9f251700 5 -- op tracker -- , seq: 85851, time: 2014-09-18 18:08:28.903034, event: header_read, op: pg_notify(2.186(12) epoch 11707) -6542> 2014-09-18 18:08:28.903121 7f4c9f251700 5 -- op tracker -- , seq: 85851, time: 2014-09-18 18:08:28.903036, event: throttled, op: pg_notify(2.186(12) epoch 11707) -6541> 2014-09-18 18:08:28.903126 7f4c9f251700 5 -- op tracker -- , seq: 85851, time: 2014-09-18 18:08:28.903080, event: all_read, op: pg_notify(2.186(12) epoch 11707) -6540> 2014-09-18 18:08:28.903130 7f4c9f251700 5 -- op tracker -- , seq: 85851, time: 2014-09-18 18:08:28.903109, event: dispatched, op: pg_notify(2.186(12) epoch 11707) -6539> 2014-09-18 18:08:28.903134 7f4c9f251700 5 -- op tracker -- , seq: 85851, time: 2014-09-18 18:08:28.903133, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11707) -6538> 2014-09-18 18:08:28.903140 7f4c9f251700 5 -- op tracker -- , seq: 85851, time: 2014-09-18 18:08:28.903138, event: started, op: pg_notify(2.186(12) epoch 11707) -6537> 2014-09-18 18:08:28.903163 7f4c9f251700 5 -- op tracker -- , seq: 85851, time: 2014-09-18 18:08:28.903162, event: done, op: pg_notify(2.186(12) epoch 11707) -6536> 2014-09-18 18:08:28.954438 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.57 10.242.42.176:0/988 118 ==== osd_ping(ping e11707 stamp 2014-09-18 23:37:33.702308) v2 ==== 47+0+0 (3721066664 0 0) 0x1368c780 con 0x12561e40 -6535> 2014-09-18 18:08:28.954458 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.57 10.242.42.176:0/988 118 ==== osd_ping(ping e11707 stamp 2014-09-18 23:37:33.702308) v2 ==== 47+0+0 (3721066664 0 0) 0xf728760 con 0x12abfc80 -6534> 2014-09-18 18:08:28.954476 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/988 -- osd_ping(ping_reply e11707 stamp 2014-09-18 23:37:33.702308) v2 -- ?+0 0xfcc8960 con 0x12561e40 -6533> 2014-09-18 18:08:28.954512 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/988 -- osd_ping(ping_reply e11707 stamp 2014-09-18 23:37:33.702308) v2 -- ?+0 0xe1a5a20 con 0x12abfc80 -6532> 2014-09-18 18:08:28.972985 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3176 ==== pg_notify(2.74e(11) epoch 11707) v5 ==== 1303+0+0 (3980031932 0 0) 0xf80e3c0 con 0x1395ec60 -6531> 2014-09-18 18:08:28.973008 7f4c9f251700 5 -- op tracker -- , seq: 85852, time: 2014-09-18 18:08:28.972862, event: header_read, op: pg_notify(2.74e(11) epoch 11707) -6530> 2014-09-18 18:08:28.973016 7f4c9f251700 5 -- op tracker -- , seq: 85852, time: 2014-09-18 18:08:28.972864, event: throttled, op: pg_notify(2.74e(11) epoch 11707) -6529> 2014-09-18 18:08:28.973021 7f4c9f251700 5 -- op tracker -- , seq: 85852, time: 2014-09-18 18:08:28.972925, event: all_read, op: pg_notify(2.74e(11) epoch 11707) -6528> 2014-09-18 18:08:28.973025 7f4c9f251700 5 -- op tracker -- , seq: 85852, time: 2014-09-18 18:08:28.973002, event: dispatched, op: pg_notify(2.74e(11) epoch 11707) -6527> 2014-09-18 18:08:28.973029 7f4c9f251700 5 -- op tracker -- , seq: 85852, time: 2014-09-18 18:08:28.973028, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11707) -6526> 2014-09-18 18:08:28.973044 7f4c9f251700 5 -- op tracker -- , seq: 85852, time: 2014-09-18 18:08:28.973043, event: started, op: pg_notify(2.74e(11) epoch 11707) -6525> 2014-09-18 18:08:28.973067 7f4c9f251700 5 -- op tracker -- , seq: 85852, time: 2014-09-18 18:08:28.973065, event: done, op: pg_notify(2.74e(11) epoch 11707) -6524> 2014-09-18 18:08:29.115600 7f4c8fa32700 5 osd.42 11707 heartbeat: osd_stat(228 GB used, 138 GB avail, 366 GB total, peers [0,2,3,4,6,9,11,20,22,23,24,27,30,32,35,39,43,47,48,53,55,56,57,58,63]/[] op hist []) -6523> 2014-09-18 18:08:29.115624 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6802/15878 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310ed00 con 0x4751e40 -6522> 2014-09-18 18:08:29.115646 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6803/15878 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310fc00 con 0x149b0ec0 -6521> 2014-09-18 18:08:29.115658 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6818/19139 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310e580 con 0x12b6a940 -6520> 2014-09-18 18:08:29.115670 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6820/19139 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310e3a0 con 0x1d518aa0 -6519> 2014-09-18 18:08:29.115694 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6824/20623 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310d2c0 con 0x1d662b00 -6518> 2014-09-18 18:08:29.115705 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6825/20623 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310e940 con 0x1d6629a0 -6517> 2014-09-18 18:08:29.115716 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6826/22324 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310eee0 con 0x1277db20 -6516> 2014-09-18 18:08:29.115728 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6828/22324 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310c5a0 con 0x1277d180 -6515> 2014-09-18 18:08:29.115744 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6810/22481 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310f480 con 0x1096fa20 -6514> 2014-09-18 18:08:29.115775 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6813/22481 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310d860 con 0x10970520 -6513> 2014-09-18 18:08:29.115801 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6827/24959 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310dfe0 con 0x1db5c420 -6512> 2014-09-18 18:08:29.115828 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6840/24959 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310f660 con 0x1db5c160 -6511> 2014-09-18 18:08:29.115845 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6822/17298 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310c1e0 con 0x1217b760 -6510> 2014-09-18 18:08:29.115856 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6823/17298 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310c3c0 con 0x1217c7e0 -6509> 2014-09-18 18:08:29.115867 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6823/33871 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310cf00 con 0xe58a9a0 -6508> 2014-09-18 18:08:29.115879 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6825/33871 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310e1c0 con 0xe58a840 -6507> 2014-09-18 18:08:29.115890 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6820/37668 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310e760 con 0x10a407e0 -6506> 2014-09-18 18:08:29.115903 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6832/37668 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310c780 con 0x10a40260 -6505> 2014-09-18 18:08:29.115937 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6826/11463 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x1310d680 con 0xe417b20 -6504> 2014-09-18 18:08:29.115966 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6827/11463 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd3840 con 0xe58ab00 -6503> 2014-09-18 18:08:29.115962 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6820/19139 181 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (340625957 0 0) 0xf2b5c00 con 0x1d518aa0 -6502> 2014-09-18 18:08:29.115982 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6803/5612 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd1860 con 0xeff6f20 -6501> 2014-09-18 18:08:29.115993 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6810/5612 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd1c20 con 0xeff6c60 -6500> 2014-09-18 18:08:29.116017 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6802/21548 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd03c0 con 0x1da53a20 -6499> 2014-09-18 18:08:29.116047 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6803/21548 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd3a20 con 0x1da53760 -6498> 2014-09-18 18:08:29.116083 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6830/1204 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd1e00 con 0x1db5cdc0 -6497> 2014-09-18 18:08:29.116100 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6831/1204 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd0960 con 0x1db5cc60 -6496> 2014-09-18 18:08:29.116115 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6825/1099 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd3480 con 0x1db5da20 -6495> 2014-09-18 18:08:29.116133 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6826/1099 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd0d20 con 0x1db5db80 -6494> 2014-09-18 18:08:29.116150 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6813/17421 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd21c0 con 0x125239c0 -6493> 2014-09-18 18:08:29.116161 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6814/17421 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd12c0 con 0x11d45860 -6492> 2014-09-18 18:08:29.116175 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6817/2660 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd01e0 con 0x238dc420 -6491> 2014-09-18 18:08:29.116185 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6818/2660 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd23a0 con 0x238dcb00 -6490> 2014-09-18 18:08:29.116200 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6843/4511 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd2760 con 0x149b1de0 -6489> 2014-09-18 18:08:29.116214 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6845/4511 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd1680 con 0x149b07e0 -6488> 2014-09-18 18:08:29.116232 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6832/25184 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd2940 con 0x1217d180 -6487> 2014-09-18 18:08:29.116249 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6833/25184 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd3de0 con 0x1217caa0 -6486> 2014-09-18 18:08:29.116262 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6837/27583 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd3660 con 0x10db26e0 -6485> 2014-09-18 18:08:29.116283 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6838/27583 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd30c0 con 0x10db4ec0 -6484> 2014-09-18 18:08:29.116299 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6826/2318 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0x11fd14a0 con 0x1db5c6e0 -6483> 2014-09-18 18:08:29.116317 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6832/2318 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6dfe00 con 0x1db5e260 -6482> 2014-09-18 18:08:29.116331 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6830/19404 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6e1a20 con 0xe58d180 -6481> 2014-09-18 18:08:29.116362 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6831/19404 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6e03a0 con 0xe58d020 -6480> 2014-09-18 18:08:29.116389 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6840/36458 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6df2c0 con 0x1364b700 -6479> 2014-09-18 18:08:29.116402 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6842/36458 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6e0ee0 con 0x1364b860 -6478> 2014-09-18 18:08:29.116412 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6802/988 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6de5a0 con 0x1db5d600 -6477> 2014-09-18 18:08:29.116435 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6803/988 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6de3c0 con 0x1db5dfa0 -6476> 2014-09-18 18:08:29.116450 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6839/46756 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6df680 con 0x1d7c7a20 -6475> 2014-09-18 18:08:29.116463 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6842/46756 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6de780 con 0x1da543c0 -6474> 2014-09-18 18:08:29.116477 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6805/2487 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6de1e0 con 0x1d996aa0 -6473> 2014-09-18 18:08:29.116492 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6809/2487 -- osd_ping(ping e11707 stamp 2014-09-18 18:08:29.115622) v2 -- ?+0 0xf6df0e0 con 0x1d996c00 -6472> 2014-09-18 18:08:29.116531 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6817/19139 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xe2efa80 con 0x10f60520 -6471> 2014-09-18 18:08:29.116549 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6818/19139 181 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (340625957 0 0) 0xe0bda20 con 0x12b6a940 -6470> 2014-09-18 18:08:29.116587 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6824/20623 170 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (340625957 0 0) 0x12139fe0 con 0x1d662b00 -6469> 2014-09-18 18:08:29.116604 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6823/20623 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0xeee9200 con 0x1f103e40 -6468> 2014-09-18 18:08:29.116616 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6825/20623 170 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (340625957 0 0) 0x21f063a0 con 0x1d6629a0 -6467> 2014-09-18 18:08:29.116645 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6826/22324 164 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (340625957 0 0) 0x1213bc00 con 0x1277db20 -6466> 2014-09-18 18:08:29.116660 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6819/22324 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0x1e285180 con 0x1f102840 -6465> 2014-09-18 18:08:29.116671 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6828/22324 164 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (340625957 0 0) 0x19f25660 con 0x1277d180 -6464> 2014-09-18 18:08:29.116705 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6843/4511 162 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x1421ffe0 con 0x149b1de0 -6463> 2014-09-18 18:08:29.116768 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6845/4511 162 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x19f223c0 con 0x149b07e0 -6462> 2014-09-18 18:08:29.116781 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6802/21548 159 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf272960 con 0x1da53a20 -6461> 2014-09-18 18:08:29.116794 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6803/21548 159 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x1247dfe0 con 0x1da53760 -6460> 2014-09-18 18:08:29.116812 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6822/17298 137 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x24f792c0 con 0x1217b760 -6459> 2014-09-18 18:08:29.116827 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6823/17298 137 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf810760 con 0x1217c7e0 -6458> 2014-09-18 18:08:29.116839 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6832/25184 137 ==== osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (2644782002 0 0) 0x1390b680 con 0x1217d180 -6457> 2014-09-18 18:08:29.116852 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6831/25184 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0x1e285cc0 con 0x12562260 -6456> 2014-09-18 18:08:29.116863 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6833/25184 137 ==== osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (2644782002 0 0) 0xf106d20 con 0x1217caa0 -6455> 2014-09-18 18:08:29.116892 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6802/988 125 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xe0c8760 con 0x1db5d600 -6454> 2014-09-18 18:08:29.116906 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6803/988 125 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x12c13480 con 0x1db5dfa0 -6453> 2014-09-18 18:08:29.116917 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6825/1099 125 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf7283a0 con 0x1db5da20 -6452> 2014-09-18 18:08:29.116929 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6826/1099 125 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x128ca780 con 0x1db5db80 -6451> 2014-09-18 18:08:29.116941 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6830/1204 125 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x12113680 con 0x1db5cdc0 -6450> 2014-09-18 18:08:29.116952 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6831/1204 125 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x10413a40 con 0x1db5cc60 -6449> 2014-09-18 18:08:29.116963 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6827/24959 123 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x11b110c0 con 0x1db5c420 -6448> 2014-09-18 18:08:29.116975 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6840/24959 123 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x24fd3660 con 0x1db5c160 -6447> 2014-09-18 18:08:29.116986 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6826/2318 123 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf29c1c0 con 0x1db5c6e0 -6446> 2014-09-18 18:08:29.116998 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6832/2318 123 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x1368e1c0 con 0x1db5e260 -6445> 2014-09-18 18:08:29.117009 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6837/27583 103 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xfae65a0 con 0x10db26e0 -6444> 2014-09-18 18:08:29.117021 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6838/27583 103 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf0e50c0 con 0x10db4ec0 -6443> 2014-09-18 18:08:29.117032 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6826/11463 1576 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x2b94bc20 con 0xe417b20 -6442> 2014-09-18 18:08:29.117045 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6827/11463 1576 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x19f22960 con 0xe58ab00 -6441> 2014-09-18 18:08:29.117056 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6823/33871 1576 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xe958940 con 0xe58a9a0 -6440> 2014-09-18 18:08:29.117068 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6825/33871 1576 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf9730e0 con 0xe58a840 -6439> 2014-09-18 18:08:29.117079 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6830/19404 1576 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf1e9a20 con 0xe58d180 -6438> 2014-09-18 18:08:29.117091 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6831/19404 1576 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x11b0e000 con 0xe58d020 -6437> 2014-09-18 18:08:29.117102 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6840/36458 656 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf12af00 con 0x1364b700 -6436> 2014-09-18 18:08:29.117113 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6842/36458 656 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf4e9840 con 0x1364b860 -6435> 2014-09-18 18:08:29.117124 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6810/22481 456 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (340625957 0 0) 0x2b925de0 con 0x1096fa20 -6434> 2014-09-18 18:08:29.117190 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0x24dbc000 con 0x1088aec0 -6433> 2014-09-18 18:08:29.117203 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6813/22481 456 ==== osd_ping(ping_reply e11706 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (340625957 0 0) 0xe76c000 con 0x10970520 -6432> 2014-09-18 18:08:29.117230 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6820/37668 450 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf9721e0 con 0x10a407e0 -6431> 2014-09-18 18:08:29.117244 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6832/37668 450 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf243e00 con 0x10a40260 -6430> 2014-09-18 18:08:29.117254 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6803/5612 342 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf0b9840 con 0xeff6f20 -6429> 2014-09-18 18:08:29.117266 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6810/5612 342 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x24fa2760 con 0xeff6c60 -6428> 2014-09-18 18:08:29.117277 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6805/2487 331 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x1368fa20 con 0x1d996aa0 -6427> 2014-09-18 18:08:29.117289 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6809/2487 331 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xec68780 con 0x1d996c00 -6426> 2014-09-18 18:08:29.117300 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6802/15878 190 ==== osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (2644782002 0 0) 0x21ced480 con 0x4751e40 -6425> 2014-09-18 18:08:29.117314 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- osd_map(11707..11707 src has 1..11707) v3 -- ?+0 0x24dbef40 con 0x10a878c0 -6424> 2014-09-18 18:08:29.117324 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6803/15878 190 ==== osd_ping(ping_reply e11705 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (2644782002 0 0) 0x121130e0 con 0x149b0ec0 -6423> 2014-09-18 18:08:29.117477 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6813/17421 187 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x1368e940 con 0x125239c0 -6422> 2014-09-18 18:08:29.117492 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6814/17421 187 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x21cec580 con 0x11d45860 -6421> 2014-09-18 18:08:29.117504 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6817/2660 185 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf2b12a0 con 0x238dc420 -6420> 2014-09-18 18:08:29.117515 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6818/2660 185 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x853e3a0 con 0x238dcb00 -6419> 2014-09-18 18:08:29.117526 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6839/46756 160 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0x121392c0 con 0x1d7c7a20 -6418> 2014-09-18 18:08:29.117538 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6842/46756 160 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.115622) v2 ==== 47+0+0 (1827334824 0 0) 0xf93ad20 con 0x1da543c0 -6417> 2014-09-18 18:08:29.124476 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3177 ==== pg_notify(2.4b1(13) epoch 11707) v5 ==== 1441+0+0 (2617996800 0 0) 0xf80eb40 con 0x1395ec60 -6416> 2014-09-18 18:08:29.124498 7f4c9f251700 5 -- op tracker -- , seq: 85853, time: 2014-09-18 18:08:29.124387, event: header_read, op: pg_notify(2.4b1(13) epoch 11707) -6415> 2014-09-18 18:08:29.124509 7f4c9f251700 5 -- op tracker -- , seq: 85853, time: 2014-09-18 18:08:29.124389, event: throttled, op: pg_notify(2.4b1(13) epoch 11707) -6414> 2014-09-18 18:08:29.124515 7f4c9f251700 5 -- op tracker -- , seq: 85853, time: 2014-09-18 18:08:29.124437, event: all_read, op: pg_notify(2.4b1(13) epoch 11707) -6413> 2014-09-18 18:08:29.124521 7f4c9f251700 5 -- op tracker -- , seq: 85853, time: 2014-09-18 18:08:29.124492, event: dispatched, op: pg_notify(2.4b1(13) epoch 11707) -6412> 2014-09-18 18:08:29.124528 7f4c9f251700 5 -- op tracker -- , seq: 85853, time: 2014-09-18 18:08:29.124526, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11707) -6411> 2014-09-18 18:08:29.124536 7f4c9f251700 5 -- op tracker -- , seq: 85853, time: 2014-09-18 18:08:29.124534, event: started, op: pg_notify(2.4b1(13) epoch 11707) -6410> 2014-09-18 18:08:29.124559 7f4c9f251700 5 -- op tracker -- , seq: 85853, time: 2014-09-18 18:08:29.124557, event: done, op: pg_notify(2.4b1(13) epoch 11707) -6409> 2014-09-18 18:08:29.125554 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3178 ==== pg_notify(2.4c1(15),2.78f(52),2.4c6(15),2.799(9),2.4d0(10) epoch 11707) v5 ==== 8993+0+0 (2480297588 0 0) 0xf80ffe0 con 0x1395ec60 -6408> 2014-09-18 18:08:29.125576 7f4c9f251700 5 -- op tracker -- , seq: 85854, time: 2014-09-18 18:08:29.125418, event: header_read, op: pg_notify(2.4c1(15),2.78f(52),2.4c6(15),2.799(9),2.4d0(10) epoch 11707) -6407> 2014-09-18 18:08:29.125587 7f4c9f251700 5 -- op tracker -- , seq: 85854, time: 2014-09-18 18:08:29.125419, event: throttled, op: pg_notify(2.4c1(15),2.78f(52),2.4c6(15),2.799(9),2.4d0(10) epoch 11707) -6406> 2014-09-18 18:08:29.125594 7f4c9f251700 5 -- op tracker -- , seq: 85854, time: 2014-09-18 18:08:29.125518, event: all_read, op: pg_notify(2.4c1(15),2.78f(52),2.4c6(15),2.799(9),2.4d0(10) epoch 11707) -6405> 2014-09-18 18:08:29.125601 7f4c9f251700 5 -- op tracker -- , seq: 85854, time: 2014-09-18 18:08:29.125571, event: dispatched, op: pg_notify(2.4c1(15),2.78f(52),2.4c6(15),2.799(9),2.4d0(10) epoch 11707) -6404> 2014-09-18 18:08:29.125608 7f4c9f251700 5 -- op tracker -- , seq: 85854, time: 2014-09-18 18:08:29.125606, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(52),2.4c6(15),2.799(9),2.4d0(10) epoch 11707) -6403> 2014-09-18 18:08:29.125617 7f4c9f251700 5 -- op tracker -- , seq: 85854, time: 2014-09-18 18:08:29.125615, event: started, op: pg_notify(2.4c1(15),2.78f(52),2.4c6(15),2.799(9),2.4d0(10) epoch 11707) -6402> 2014-09-18 18:08:29.125676 7f4c9f251700 5 -- op tracker -- , seq: 85854, time: 2014-09-18 18:08:29.125674, event: done, op: pg_notify(2.4c1(15),2.78f(52),2.4c6(15),2.799(9),2.4d0(10) epoch 11707) -6401> 2014-09-18 18:08:29.177828 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.30 10.242.42.176:0/1204 119 ==== osd_ping(ping e11707 stamp 2014-09-18 23:37:33.925580) v2 ==== 47+0+0 (1279453163 0 0) 0xf189a20 con 0x110bbc80 -6400> 2014-09-18 18:08:29.177823 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.30 10.242.42.176:0/1204 119 ==== osd_ping(ping e11707 stamp 2014-09-18 23:37:33.925580) v2 ==== 47+0+0 (1279453163 0 0) 0x24fd1a40 con 0x12abd1e0 -6399> 2014-09-18 18:08:29.177870 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/1204 -- osd_ping(ping_reply e11707 stamp 2014-09-18 23:37:33.925580) v2 -- ?+0 0xf728760 con 0x12abd1e0 -6398> 2014-09-18 18:08:29.177906 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/1204 -- osd_ping(ping_reply e11707 stamp 2014-09-18 23:37:33.925580) v2 -- ?+0 0x1368c780 con 0x110bbc80 -6397> 2014-09-18 18:08:29.190156 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.13 10.242.42.168:0/10879 693 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:29.056890) v2 ==== 47+0+0 (25221496 0 0) 0xe288760 con 0x109f9de0 -6396> 2014-09-18 18:08:29.190171 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.13 10.242.42.168:0/10879 693 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:29.056890) v2 ==== 47+0+0 (25221496 0 0) 0xf93c1c0 con 0x140f7b80 -6395> 2014-09-18 18:08:29.190182 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.056890) v2 -- ?+0 0xf189a20 con 0x109f9de0 -6394> 2014-09-18 18:08:29.190221 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.056890) v2 -- ?+0 0x24fd1a40 con 0x140f7b80 -6393> 2014-09-18 18:08:29.195980 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.11 10.242.42.168:0/17298 125 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:29.062946) v2 ==== 47+0+0 (2123676592 0 0) 0xf09b2c0 con 0x101cd180 -6392> 2014-09-18 18:08:29.196008 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.062946) v2 -- ?+0 0xf93c1c0 con 0x101cd180 -6391> 2014-09-18 18:08:29.196029 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.11 10.242.42.168:0/17298 125 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:29.062946) v2 ==== 47+0+0 (2123676592 0 0) 0xe76c960 con 0xcc49e40 -6390> 2014-09-18 18:08:29.196054 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.062946) v2 -- ?+0 0xe288760 con 0xcc49e40 -6389> 2014-09-18 18:08:29.316303 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5077 ==== pg_info(1 pgs e11706:2.410) v4 ==== 718+0+0 (1084840817 0 0) 0xf09fa40 con 0x1088aec0 -6388> 2014-09-18 18:08:29.316333 7f4c9f251700 5 -- op tracker -- , seq: 85855, time: 2014-09-18 18:08:29.316155, event: header_read, op: pg_info(1 pgs e11706:2.410) -6387> 2014-09-18 18:08:29.316346 7f4c9f251700 5 -- op tracker -- , seq: 85855, time: 2014-09-18 18:08:29.316157, event: throttled, op: pg_info(1 pgs e11706:2.410) -6386> 2014-09-18 18:08:29.316354 7f4c9f251700 5 -- op tracker -- , seq: 85855, time: 2014-09-18 18:08:29.316198, event: all_read, op: pg_info(1 pgs e11706:2.410) -6385> 2014-09-18 18:08:29.316369 7f4c9f251700 5 -- op tracker -- , seq: 85855, time: 2014-09-18 18:08:29.316326, event: dispatched, op: pg_info(1 pgs e11706:2.410) -6384> 2014-09-18 18:08:29.316377 7f4c9f251700 5 -- op tracker -- , seq: 85855, time: 2014-09-18 18:08:29.316376, event: waiting_for_osdmap, op: pg_info(1 pgs e11706:2.410) -6383> 2014-09-18 18:08:29.316387 7f4c9f251700 5 -- op tracker -- , seq: 85855, time: 2014-09-18 18:08:29.316385, event: started, op: pg_info(1 pgs e11706:2.410) -6382> 2014-09-18 18:08:29.316408 7f4c9f251700 5 -- op tracker -- , seq: 85855, time: 2014-09-18 18:08:29.316407, event: done, op: pg_info(1 pgs e11706:2.410) -6381> 2014-09-18 18:08:29.316447 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_info(1 pgs e11707:2.410) v4 -- ?+0 0x2b923a40 con 0x1088aec0 -6380> 2014-09-18 18:08:29.316541 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11707:2.410) v4 -- ?+0 0xe0c9660 con 0x1395f8c0 -6379> 2014-09-18 18:08:29.316763 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=6,27 from 0//0//-1 -6378> 2014-09-18 18:08:29.316818 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 6 from 0//0//-1 -6377> 2014-09-18 18:08:29.316832 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 27 from 0//0//-1 -6376> 2014-09-18 18:08:29.316868 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 0.986188 10 0.001622 -6375> 2014-09-18 18:08:29.316884 7f4c97241700 5 osd.42 pg_epoch: 11707 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -6374> 2014-09-18 18:08:29.355989 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1276 ==== pg_notify(2.4f6(15) epoch 11707) v5 ==== 1543+0+0 (2557559339 0 0) 0xf097c20 con 0x1f102840 -6373> 2014-09-18 18:08:29.356017 7f4c9f251700 5 -- op tracker -- , seq: 85856, time: 2014-09-18 18:08:29.355894, event: header_read, op: pg_notify(2.4f6(15) epoch 11707) -6372> 2014-09-18 18:08:29.356032 7f4c9f251700 5 -- op tracker -- , seq: 85856, time: 2014-09-18 18:08:29.355897, event: throttled, op: pg_notify(2.4f6(15) epoch 11707) -6371> 2014-09-18 18:08:29.356041 7f4c9f251700 5 -- op tracker -- , seq: 85856, time: 2014-09-18 18:08:29.355967, event: all_read, op: pg_notify(2.4f6(15) epoch 11707) -6370> 2014-09-18 18:08:29.356048 7f4c9f251700 5 -- op tracker -- , seq: 85856, time: 2014-09-18 18:08:29.356010, event: dispatched, op: pg_notify(2.4f6(15) epoch 11707) -6369> 2014-09-18 18:08:29.356056 7f4c9f251700 5 -- op tracker -- , seq: 85856, time: 2014-09-18 18:08:29.356055, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11707) -6368> 2014-09-18 18:08:29.356067 7f4c9f251700 5 -- op tracker -- , seq: 85856, time: 2014-09-18 18:08:29.356065, event: started, op: pg_notify(2.4f6(15) epoch 11707) -6367> 2014-09-18 18:08:29.356091 7f4c9f251700 5 -- op tracker -- , seq: 85856, time: 2014-09-18 18:08:29.356090, event: done, op: pg_notify(2.4f6(15) epoch 11707) -6366> 2014-09-18 18:08:29.370786 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 337 ==== pg_notify(2.719(17) epoch 11706) v5 ==== 1601+0+0 (1834606333 0 0) 0xf2b0940 con 0x12562260 -6365> 2014-09-18 18:08:29.370817 7f4c9f251700 5 -- op tracker -- , seq: 85857, time: 2014-09-18 18:08:29.370668, event: header_read, op: pg_notify(2.719(17) epoch 11706) -6364> 2014-09-18 18:08:29.370834 7f4c9f251700 5 -- op tracker -- , seq: 85857, time: 2014-09-18 18:08:29.370671, event: throttled, op: pg_notify(2.719(17) epoch 11706) -6363> 2014-09-18 18:08:29.370842 7f4c9f251700 5 -- op tracker -- , seq: 85857, time: 2014-09-18 18:08:29.370739, event: all_read, op: pg_notify(2.719(17) epoch 11706) -6362> 2014-09-18 18:08:29.370850 7f4c9f251700 5 -- op tracker -- , seq: 85857, time: 2014-09-18 18:08:29.370810, event: dispatched, op: pg_notify(2.719(17) epoch 11706) -6361> 2014-09-18 18:08:29.370859 7f4c9f251700 5 -- op tracker -- , seq: 85857, time: 2014-09-18 18:08:29.370857, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11706) -6360> 2014-09-18 18:08:29.370869 7f4c9f251700 5 -- op tracker -- , seq: 85857, time: 2014-09-18 18:08:29.370868, event: started, op: pg_notify(2.719(17) epoch 11706) -6359> 2014-09-18 18:08:29.370894 7f4c9f251700 5 -- op tracker -- , seq: 85857, time: 2014-09-18 18:08:29.370893, event: done, op: pg_notify(2.719(17) epoch 11706) -6358> 2014-09-18 18:08:29.387827 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 338 ==== pg_notify(2.719(17) epoch 11707) v5 ==== 1601+0+0 (2205226650 0 0) 0x11739680 con 0x12562260 -6357> 2014-09-18 18:08:29.387847 7f4c9f251700 5 -- op tracker -- , seq: 85858, time: 2014-09-18 18:08:29.387726, event: header_read, op: pg_notify(2.719(17) epoch 11707) -6356> 2014-09-18 18:08:29.387862 7f4c9f251700 5 -- op tracker -- , seq: 85858, time: 2014-09-18 18:08:29.387727, event: throttled, op: pg_notify(2.719(17) epoch 11707) -6355> 2014-09-18 18:08:29.387870 7f4c9f251700 5 -- op tracker -- , seq: 85858, time: 2014-09-18 18:08:29.387806, event: all_read, op: pg_notify(2.719(17) epoch 11707) -6354> 2014-09-18 18:08:29.387877 7f4c9f251700 5 -- op tracker -- , seq: 85858, time: 2014-09-18 18:08:29.387843, event: dispatched, op: pg_notify(2.719(17) epoch 11707) -6353> 2014-09-18 18:08:29.387884 7f4c9f251700 5 -- op tracker -- , seq: 85858, time: 2014-09-18 18:08:29.387883, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11707) -6352> 2014-09-18 18:08:29.387893 7f4c9f251700 5 -- op tracker -- , seq: 85858, time: 2014-09-18 18:08:29.387891, event: started, op: pg_notify(2.719(17) epoch 11707) -6351> 2014-09-18 18:08:29.387913 7f4c9f251700 5 -- op tracker -- , seq: 85858, time: 2014-09-18 18:08:29.387912, event: done, op: pg_notify(2.719(17) epoch 11707) -6350> 2014-09-18 18:08:29.439303 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.37 10.242.42.168:0/18319 104 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:29.305897) v2 ==== 47+0+0 (1930642079 0 0) 0x1173b660 con 0x11ffdce0 -6349> 2014-09-18 18:08:29.439345 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.305897) v2 -- ?+0 0xf09b2c0 con 0x11ffdce0 -6348> 2014-09-18 18:08:29.439343 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.37 10.242.42.168:0/18319 104 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:29.305897) v2 ==== 47+0+0 (1930642079 0 0) 0xf09e780 con 0x110bb860 -6347> 2014-09-18 18:08:29.439385 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:29.305897) v2 -- ?+0 0xe76c960 con 0x110bb860 -6346> 2014-09-18 18:08:29.454534 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5078 ==== pg_notify(2.62(44) epoch 11707) v5 ==== 3220+0+0 (1267055609 0 0) 0x2b923a40 con 0x1088aec0 -6345> 2014-09-18 18:08:29.454581 7f4c9f251700 5 -- op tracker -- , seq: 85859, time: 2014-09-18 18:08:29.454423, event: header_read, op: pg_notify(2.62(44) epoch 11707) -6344> 2014-09-18 18:08:29.454604 7f4c9f251700 5 -- op tracker -- , seq: 85859, time: 2014-09-18 18:08:29.454425, event: throttled, op: pg_notify(2.62(44) epoch 11707) -6343> 2014-09-18 18:08:29.454610 7f4c9f251700 5 -- op tracker -- , seq: 85859, time: 2014-09-18 18:08:29.454515, event: all_read, op: pg_notify(2.62(44) epoch 11707) -6342> 2014-09-18 18:08:29.454619 7f4c9f251700 5 -- op tracker -- , seq: 85859, time: 2014-09-18 18:08:29.454574, event: dispatched, op: pg_notify(2.62(44) epoch 11707) -6341> 2014-09-18 18:08:29.454639 7f4c9f251700 5 -- op tracker -- , seq: 85859, time: 2014-09-18 18:08:29.454638, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11707) -6340> 2014-09-18 18:08:29.454653 7f4c9f251700 5 -- op tracker -- , seq: 85859, time: 2014-09-18 18:08:29.454651, event: started, op: pg_notify(2.62(44) epoch 11707) -6339> 2014-09-18 18:08:29.454679 7f4c9f251700 5 -- op tracker -- , seq: 85859, time: 2014-09-18 18:08:29.454677, event: done, op: pg_notify(2.62(44) epoch 11707) -6338> 2014-09-18 18:08:29.457051 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5079 ==== pg_notify(2.3b6(10) epoch 11707) v5 ==== 1260+0+0 (1749223122 0 0) 0xf09f2c0 con 0x1088aec0 -6337> 2014-09-18 18:08:29.457078 7f4c9f251700 5 -- op tracker -- , seq: 85860, time: 2014-09-18 18:08:29.456973, event: header_read, op: pg_notify(2.3b6(10) epoch 11707) -6336> 2014-09-18 18:08:29.457098 7f4c9f251700 5 -- op tracker -- , seq: 85860, time: 2014-09-18 18:08:29.456975, event: throttled, op: pg_notify(2.3b6(10) epoch 11707) -6335> 2014-09-18 18:08:29.457109 7f4c9f251700 5 -- op tracker -- , seq: 85860, time: 2014-09-18 18:08:29.457032, event: all_read, op: pg_notify(2.3b6(10) epoch 11707) -6334> 2014-09-18 18:08:29.457120 7f4c9f251700 5 -- op tracker -- , seq: 85860, time: 2014-09-18 18:08:29.457071, event: dispatched, op: pg_notify(2.3b6(10) epoch 11707) -6333> 2014-09-18 18:08:29.457125 7f4c9f251700 5 -- op tracker -- , seq: 85860, time: 2014-09-18 18:08:29.457124, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11707) -6332> 2014-09-18 18:08:29.457135 7f4c9f251700 5 -- op tracker -- , seq: 85860, time: 2014-09-18 18:08:29.457133, event: started, op: pg_notify(2.3b6(10) epoch 11707) -6331> 2014-09-18 18:08:29.457157 7f4c9f251700 5 -- op tracker -- , seq: 85860, time: 2014-09-18 18:08:29.457156, event: done, op: pg_notify(2.3b6(10) epoch 11707) -6330> 2014-09-18 18:08:29.459817 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5080 ==== pg_notify(2.517(11) epoch 11707) v5 ==== 1335+0+0 (986406390 0 0) 0xf09ffe0 con 0x1088aec0 -6329> 2014-09-18 18:08:29.459843 7f4c9f251700 5 -- op tracker -- , seq: 85861, time: 2014-09-18 18:08:29.459707, event: header_read, op: pg_notify(2.517(11) epoch 11707) -6328> 2014-09-18 18:08:29.459861 7f4c9f251700 5 -- op tracker -- , seq: 85861, time: 2014-09-18 18:08:29.459708, event: throttled, op: pg_notify(2.517(11) epoch 11707) -6327> 2014-09-18 18:08:29.459873 7f4c9f251700 5 -- op tracker -- , seq: 85861, time: 2014-09-18 18:08:29.459776, event: all_read, op: pg_notify(2.517(11) epoch 11707) -6326> 2014-09-18 18:08:29.459884 7f4c9f251700 5 -- op tracker -- , seq: 85861, time: 2014-09-18 18:08:29.459836, event: dispatched, op: pg_notify(2.517(11) epoch 11707) -6325> 2014-09-18 18:08:29.459895 7f4c9f251700 5 -- op tracker -- , seq: 85861, time: 2014-09-18 18:08:29.459892, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11707) -6324> 2014-09-18 18:08:29.459917 7f4c9f251700 5 -- op tracker -- , seq: 85861, time: 2014-09-18 18:08:29.459914, event: started, op: pg_notify(2.517(11) epoch 11707) -6323> 2014-09-18 18:08:29.459944 7f4c9f251700 5 -- op tracker -- , seq: 85861, time: 2014-09-18 18:08:29.459942, event: done, op: pg_notify(2.517(11) epoch 11707) -6322> 2014-09-18 18:08:29.552227 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.39 10.242.42.172:0/2660 1589 ==== osd_ping(ping e11707 stamp 2014-09-18 23:37:18.656246) v2 ==== 47+0+0 (1038745667 0 0) 0xfccb480 con 0x4750840 -6321> 2014-09-18 18:08:29.552266 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11707 stamp 2014-09-18 23:37:18.656246) v2 -- ?+0 0x1173b660 con 0x4750840 -6320> 2014-09-18 18:08:29.552944 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.39 10.242.42.172:0/2660 1589 ==== osd_ping(ping e11707 stamp 2014-09-18 23:37:18.656246) v2 ==== 47+0+0 (1038745667 0 0) 0x11b11840 con 0xec39340 -6319> 2014-09-18 18:08:29.552964 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11707 stamp 2014-09-18 23:37:18.656246) v2 -- ?+0 0xf09e780 con 0xec39340 -6318> 2014-09-18 18:08:29.713736 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1721 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0xef59a80 con 0x4900000 -6317> 2014-09-18 18:08:29.713886 7f4c9f251700 3 osd.42 11707 handle_osd_map epochs [11708,11708], i have 11707, src has [1,11708] -6316> 2014-09-18 18:08:29.716458 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11504/11505/11490) [9,42,27]/[9,42] r=1 lpr=11505 pi=11258-11504/5 luod=0'0 crt=6705'30135 lcod 0'0 active+remapped] exit Started/ReplicaActive/RepNotRecovering 362.044558 606 0.001478 -6315> 2014-09-18 18:08:29.716519 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11504/11505/11490) [9,42,27]/[9,42] r=1 lpr=11505 pi=11258-11504/5 luod=0'0 crt=6705'30135 lcod 0'0 active+remapped] exit Started/ReplicaActive 362.044634 0 0.000000 -6314> 2014-09-18 18:08:29.716544 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11504/11505/11490) [9,42,27]/[9,42] r=1 lpr=11505 pi=11258-11504/5 luod=0'0 crt=6705'30135 lcod 0'0 active+remapped] exit Started 363.064988 0 0.000000 -6313> 2014-09-18 18:08:29.716560 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11504/11505/11490) [9,42,27]/[9,42] r=1 lpr=11505 pi=11258-11504/5 luod=0'0 crt=6705'30135 lcod 0'0 active+remapped] enter Reset -6312> 2014-09-18 18:08:29.716628 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11598/11599/9945) [42,32,9]/[42,9] r=0 lpr=11599 pi=11570-11598/2 bft=32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 146.964923 322 0.002632 -6311> 2014-09-18 18:08:29.716642 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1722 ==== pg_notify(2.e0(79) epoch 11708) v5 ==== 5119+0+0 (2590511559 0 0) 0xf1ecb20 con 0x4900000 -6310> 2014-09-18 18:08:29.716659 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11598/11599/9945) [42,32,9]/[42,9] r=0 lpr=11599 pi=11570-11598/2 bft=32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 147.053241 0 0.000000 -6309> 2014-09-18 18:08:29.716666 7f4c9f251700 5 -- op tracker -- , seq: 85862, time: 2014-09-18 18:08:29.713753, event: header_read, op: pg_notify(2.e0(79) epoch 11708) -6308> 2014-09-18 18:08:29.716676 7f4c9f251700 5 -- op tracker -- , seq: 85862, time: 2014-09-18 18:08:29.713753, event: throttled, op: pg_notify(2.e0(79) epoch 11708) -6307> 2014-09-18 18:08:29.716683 7f4c9f251700 5 -- op tracker -- , seq: 85862, time: 2014-09-18 18:08:29.713854, event: all_read, op: pg_notify(2.e0(79) epoch 11708) -6306> 2014-09-18 18:08:29.716679 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11598/11599/9945) [42,32,9]/[42,9] r=0 lpr=11599 pi=11570-11598/2 bft=32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 148.047358 0 0.000000 -6305> 2014-09-18 18:08:29.716689 7f4c9f251700 5 -- op tracker -- , seq: 85862, time: 2014-09-18 18:08:29.716663, event: dispatched, op: pg_notify(2.e0(79) epoch 11708) -6304> 2014-09-18 18:08:29.716704 7f4c9f251700 5 -- op tracker -- , seq: 85862, time: 2014-09-18 18:08:29.716703, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11708) -6303> 2014-09-18 18:08:29.716697 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11598/11599/9945) [42,32,9]/[42,9] r=0 lpr=11599 pi=11570-11598/2 bft=32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+remapped] exit Started 148.047417 0 0.000000 -6302> 2014-09-18 18:08:29.716711 7f4c9f251700 5 -- op tracker -- , seq: 85862, time: 2014-09-18 18:08:29.716710, event: started, op: pg_notify(2.e0(79) epoch 11708) -6301> 2014-09-18 18:08:29.716725 7f4c9f251700 5 -- op tracker -- , seq: 85862, time: 2014-09-18 18:08:29.716723, event: done, op: pg_notify(2.e0(79) epoch 11708) -6300> 2014-09-18 18:08:29.716711 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11598/11599/9945) [42,32,9]/[42,9] r=0 lpr=11599 pi=11570-11598/2 bft=32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -6299> 2014-09-18 18:08:29.716909 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000198 1 0.000280 -6298> 2014-09-18 18:08:29.716922 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped] enter Started -6297> 2014-09-18 18:08:29.716929 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped] enter Start -6296> 2014-09-18 18:08:29.716918 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] exit Reset 0.000358 1 0.000145 -6295> 2014-09-18 18:08:29.716935 7f4c96a40700 1 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -6294> 2014-09-18 18:08:29.716939 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] enter Started -6293> 2014-09-18 18:08:29.716943 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000014 0 0.000000 -6292> 2014-09-18 18:08:29.716952 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -6291> 2014-09-18 18:08:29.716950 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] enter Start -6290> 2014-09-18 18:08:29.716962 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -6289> 2014-09-18 18:08:29.716964 7f4c97241700 1 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -6288> 2014-09-18 18:08:29.716971 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -6287> 2014-09-18 18:08:29.716985 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] exit Start 0.000035 0 0.000000 -6286> 2014-09-18 18:08:29.717002 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] enter Started/Stray -6285> 2014-09-18 18:08:29.717269 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11482/11482/9945) [42,6,57]/[42,57] r=0 lpr=11482 pi=10656-11481/9 bft=6 crt=6752'29654 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitRemoteBackfillReserved 148.708716 328 0.001783 -6284> 2014-09-18 18:08:29.717398 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11482/11482/9945) [42,6,57]/[42,57] r=0 lpr=11482 pi=10656-11481/9 bft=6 crt=6752'29654 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 383.529627 0 0.000000 -6283> 2014-09-18 18:08:29.717429 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11406/11406/11015) [6,42,27] r=1 lpr=11406 pi=3487-11405/26 crt=0'0 active] exit Started/ReplicaActive/RepNotRecovering 478.541312 898 0.001358 -6282> 2014-09-18 18:08:29.717446 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11406/11406/11015) [6,42,27] r=1 lpr=11406 pi=3487-11405/26 crt=0'0 active] exit Started/ReplicaActive 478.541340 0 0.000000 -6281> 2014-09-18 18:08:29.717460 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11406/11406/11015) [6,42,27] r=1 lpr=11406 pi=3487-11405/26 crt=0'0 active] exit Started 482.088029 0 0.000000 -6280> 2014-09-18 18:08:29.717471 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11406/11406/11015) [6,42,27] r=1 lpr=11406 pi=3487-11405/26 crt=0'0 active] enter Reset -6279> 2014-09-18 18:08:29.717421 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11482/11482/9945) [42,6,57]/[42,57] r=0 lpr=11482 pi=10656-11481/9 bft=6 crt=6752'29654 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 386.752746 0 0.000000 -6278> 2014-09-18 18:08:29.717485 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11482/11482/9945) [42,6,57]/[42,57] r=0 lpr=11482 pi=10656-11481/9 bft=6 crt=6752'29654 lcod 0'0 mlcod 0'0 active+remapped] exit Started 386.752860 0 0.000000 -6277> 2014-09-18 18:08:29.717499 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11482/11482/9945) [42,6,57]/[42,57] r=0 lpr=11482 pi=10656-11481/9 bft=6 crt=6752'29654 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -6276> 2014-09-18 18:08:29.717539 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] exit Reset 0.000067 1 0.000081 -6275> 2014-09-18 18:08:29.717552 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] enter Started -6274> 2014-09-18 18:08:29.717564 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] enter Start -6273> 2014-09-18 18:08:29.717574 7f4c97241700 1 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] state: transitioning to Stray -6272> 2014-09-18 18:08:29.717586 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] exit Start 0.000023 0 0.000000 -6271> 2014-09-18 18:08:29.717598 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] enter Started/Stray -6270> 2014-09-18 18:08:29.717704 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000205 1 0.000434 -6269> 2014-09-18 18:08:29.717720 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped] enter Started -6268> 2014-09-18 18:08:29.717745 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped] enter Start -6267> 2014-09-18 18:08:29.717760 7f4c96a40700 1 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -6266> 2014-09-18 18:08:29.717771 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000026 0 0.000000 -6265> 2014-09-18 18:08:29.717782 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -6264> 2014-09-18 18:08:29.717791 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -6263> 2014-09-18 18:08:29.717801 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -6262> 2014-09-18 18:08:29.718517 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0xe2ef600 con 0x1395ec60 -6261> 2014-09-18 18:08:29.718564 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.13(6),0.12(27) epoch 11708) v5 -- ?+0 0xe9590c0 con 0x1395ec60 -6260> 2014-09-18 18:08:29.718580 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x24d1a480 con 0x140f95a0 -6259> 2014-09-18 18:08:29.718593 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11708) v5 -- ?+0 0xe959840 con 0x140f95a0 -6258> 2014-09-18 18:08:29.718578 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 923 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x19df1f80 con 0x1395f8c0 -6257> 2014-09-18 18:08:29.718645 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6256> 2014-09-18 18:08:29.718695 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x24d1a900 con 0x1395edc0 -6255> 2014-09-18 18:08:29.718712 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11708) v5 -- ?+0 0x21cfa1e0 con 0x1395edc0 -6254> 2014-09-18 18:08:29.718764 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x15e32f40 con 0x1395f1e0 -6253> 2014-09-18 18:08:29.718780 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11708) v5 -- ?+0 0x21cfba40 con 0x1395f1e0 -6252> 2014-09-18 18:08:29.718807 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -6251> 2014-09-18 18:08:29.718826 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_alive(want up_thru 11708 have 11708) v1 -- ?+0 0xda31660 con 0x111771e0 -6250> 2014-09-18 18:08:29.718861 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11708) v5 -- ?+0 0xda30ee0 con 0x140f95a0 -6249> 2014-09-18 18:08:29.718880 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_query(2.178 epoch 11708) v3 -- ?+0 0x1247d0e0 con 0x1395ec60 -6248> 2014-09-18 18:08:29.718899 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6822/33871 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x10d19180 con 0x4900000 -6247> 2014-09-18 18:08:29.718915 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6822/33871 -- pg_query(2.178 epoch 11708) v3 -- ?+0 0x1247ed00 con 0x4900000 -6246> 2014-09-18 18:08:29.718929 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x10d18d00 con 0x1395ef20 -6245> 2014-09-18 18:08:29.718946 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_query(2.178 epoch 11708) v3 -- ?+0 0x1247c780 con 0x1395ef20 -6244> 2014-09-18 18:08:29.718930 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11701/11701/10657) [9,13,30]/[13,42] r=1 lpr=11701 pi=9952-11700/26 luod=0'0 crt=6789'26114 active+remapped] exit Started/ReplicaActive/RepNotRecovering 5.795304 18 0.000458 -6243> 2014-09-18 18:08:29.719000 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11701/11701/10657) [9,13,30]/[13,42] r=1 lpr=11701 pi=9952-11700/26 luod=0'0 crt=6789'26114 active+remapped] exit Started/ReplicaActive 5.795447 0 0.000000 -6242> 2014-09-18 18:08:29.719013 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11701/11701/10657) [9,13,30]/[13,42] r=1 lpr=11701 pi=9952-11700/26 luod=0'0 crt=6789'26114 active+remapped] exit Started 6.807533 0 0.000000 -6241> 2014-09-18 18:08:29.719023 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11701/11701/10657) [9,13,30]/[13,42] r=1 lpr=11701 pi=9952-11700/26 luod=0'0 crt=6789'26114 active+remapped] enter Reset -6240> 2014-09-18 18:08:29.719042 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x10d19a80 con 0xcbecec0 -6239> 2014-09-18 18:08:29.718842 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 924 ==== pg_notify(2.78f(155) epoch 11708) v5 ==== 9335+0+0 (4194529238 0 0) 0xe0c9660 con 0x1395f8c0 -6238> 2014-09-18 18:08:29.719064 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_query(2.179 epoch 11708) v3 -- ?+0 0x1247dfe0 con 0xcbecec0 -6237> 2014-09-18 18:08:29.719074 7f4c9f251700 5 -- op tracker -- , seq: 85863, time: 2014-09-18 18:08:29.718663, event: header_read, op: pg_notify(2.78f(155) epoch 11708) -6236> 2014-09-18 18:08:29.719077 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x10d18640 con 0x1395f080 -6235> 2014-09-18 18:08:29.719084 7f4c9f251700 5 -- op tracker -- , seq: 85863, time: 2014-09-18 18:08:29.718664, event: throttled, op: pg_notify(2.78f(155) epoch 11708) -6234> 2014-09-18 18:08:29.719090 7f4c9f251700 5 -- op tracker -- , seq: 85863, time: 2014-09-18 18:08:29.718826, event: all_read, op: pg_notify(2.78f(155) epoch 11708) -6233> 2014-09-18 18:08:29.719090 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- pg_query(2.179 epoch 11708) v3 -- ?+0 0x1247d2c0 con 0x1395f080 -6232> 2014-09-18 18:08:29.719097 7f4c9f251700 5 -- op tracker -- , seq: 85863, time: 2014-09-18 18:08:29.719068, event: dispatched, op: pg_notify(2.78f(155) epoch 11708) -6231> 2014-09-18 18:08:29.719104 7f4c9f251700 5 -- op tracker -- , seq: 85863, time: 2014-09-18 18:08:29.719102, event: waiting_for_osdmap, op: pg_notify(2.78f(155) epoch 11708) -6230> 2014-09-18 18:08:29.719100 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] exit Reset 0.000077 1 0.000136 -6229> 2014-09-18 18:08:29.719114 7f4c9f251700 5 -- op tracker -- , seq: 85863, time: 2014-09-18 18:08:29.719111, event: started, op: pg_notify(2.78f(155) epoch 11708) -6228> 2014-09-18 18:08:29.719114 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] enter Started -6227> 2014-09-18 18:08:29.719134 7f4c9f251700 5 -- op tracker -- , seq: 85863, time: 2014-09-18 18:08:29.719132, event: done, op: pg_notify(2.78f(155) epoch 11708) -6226> 2014-09-18 18:08:29.719125 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] enter Start -6225> 2014-09-18 18:08:29.719164 7f4c97241700 1 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] state: transitioning to Stray -6224> 2014-09-18 18:08:29.719172 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] exit Start 0.000047 0 0.000000 -6223> 2014-09-18 18:08:29.719180 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] enter Started/Stray -6222> 2014-09-18 18:08:29.719578 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1036 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0xcc02f40 con 0x14358260 -6221> 2014-09-18 18:08:29.719599 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6220> 2014-09-18 18:08:29.719606 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1037 ==== pg_notify(2.719(17) epoch 11708) v5 ==== 1601+0+0 (1293383990 0 0) 0xf7303a0 con 0x14358260 -6219> 2014-09-18 18:08:29.719618 7f4c9f251700 5 -- op tracker -- , seq: 85864, time: 2014-09-18 18:08:29.719399, event: header_read, op: pg_notify(2.719(17) epoch 11708) -6218> 2014-09-18 18:08:29.719650 7f4c9f251700 5 -- op tracker -- , seq: 85864, time: 2014-09-18 18:08:29.719399, event: throttled, op: pg_notify(2.719(17) epoch 11708) -6217> 2014-09-18 18:08:29.719656 7f4c9f251700 5 -- op tracker -- , seq: 85864, time: 2014-09-18 18:08:29.719444, event: all_read, op: pg_notify(2.719(17) epoch 11708) -6216> 2014-09-18 18:08:29.719662 7f4c9f251700 5 -- op tracker -- , seq: 85864, time: 2014-09-18 18:08:29.719614, event: dispatched, op: pg_notify(2.719(17) epoch 11708) -6215> 2014-09-18 18:08:29.719670 7f4c9f251700 5 -- op tracker -- , seq: 85864, time: 2014-09-18 18:08:29.719668, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11708) -6214> 2014-09-18 18:08:29.719679 7f4c9f251700 5 -- op tracker -- , seq: 85864, time: 2014-09-18 18:08:29.719676, event: started, op: pg_notify(2.719(17) epoch 11708) -6213> 2014-09-18 18:08:29.719718 7f4c9f251700 5 -- op tracker -- , seq: 85864, time: 2014-09-18 18:08:29.719715, event: done, op: pg_notify(2.719(17) epoch 11708) -6212> 2014-09-18 18:08:29.719713 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11483/11483/9945) [42,6,30]/[42,30] r=0 lpr=11483 pi=10729-11482/9 bft=6 crt=6788'24342 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitRemoteBackfillReserved 148.716289 328 0.001672 -6211> 2014-09-18 18:08:29.719757 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11483/11483/9945) [42,6,30]/[42,30] r=0 lpr=11483 pi=10729-11482/9 bft=6 crt=6788'24342 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 383.620736 0 0.000000 -6210> 2014-09-18 18:08:29.719791 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1723 ==== pg_notify(2.178 epoch 11708) v5 ==== 722+0+0 (412618369 0 0) 0x1247ed00 con 0x4900000 -6209> 2014-09-18 18:08:29.719805 7f4c9f251700 5 -- op tracker -- , seq: 85865, time: 2014-09-18 18:08:29.719727, event: header_read, op: pg_notify(2.178 epoch 11708) -6208> 2014-09-18 18:08:29.719814 7f4c9f251700 5 -- op tracker -- , seq: 85865, time: 2014-09-18 18:08:29.719728, event: throttled, op: pg_notify(2.178 epoch 11708) -6207> 2014-09-18 18:08:29.719822 7f4c9f251700 5 -- op tracker -- , seq: 85865, time: 2014-09-18 18:08:29.719756, event: all_read, op: pg_notify(2.178 epoch 11708) -6206> 2014-09-18 18:08:29.719828 7f4c9f251700 5 -- op tracker -- , seq: 85865, time: 2014-09-18 18:08:29.719800, event: dispatched, op: pg_notify(2.178 epoch 11708) -6205> 2014-09-18 18:08:29.719835 7f4c9f251700 5 -- op tracker -- , seq: 85865, time: 2014-09-18 18:08:29.719833, event: waiting_for_osdmap, op: pg_notify(2.178 epoch 11708) -6204> 2014-09-18 18:08:29.719844 7f4c9f251700 5 -- op tracker -- , seq: 85865, time: 2014-09-18 18:08:29.719841, event: started, op: pg_notify(2.178 epoch 11708) -6203> 2014-09-18 18:08:29.719861 7f4c9f251700 5 -- op tracker -- , seq: 85865, time: 2014-09-18 18:08:29.719859, event: done, op: pg_notify(2.178 epoch 11708) -6202> 2014-09-18 18:08:29.719792 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11483/11483/9945) [42,6,30]/[42,30] r=0 lpr=11483 pi=10729-11482/9 bft=6 crt=6788'24342 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 385.737845 0 0.000000 -6201> 2014-09-18 18:08:29.719902 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11483/11483/9945) [42,6,30]/[42,30] r=0 lpr=11483 pi=10729-11482/9 bft=6 crt=6788'24342 lcod 0'0 mlcod 0'0 active+remapped] exit Started 385.737994 0 0.000000 -6200> 2014-09-18 18:08:29.719923 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11483/11483/9945) [42,6,30]/[42,30] r=0 lpr=11483 pi=10729-11482/9 bft=6 crt=6788'24342 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -6199> 2014-09-18 18:08:29.719981 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000057 1 0.000265 -6198> 2014-09-18 18:08:29.720016 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 784 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x2d7fbb00 con 0xe9a27e0 -6197> 2014-09-18 18:08:29.720030 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6196> 2014-09-18 18:08:29.720018 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped] enter Started -6195> 2014-09-18 18:08:29.720051 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped] enter Start -6194> 2014-09-18 18:08:29.720066 7f4c97241700 1 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -6193> 2014-09-18 18:08:29.720106 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 299 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x24d1a480 con 0x140f95a0 -6192> 2014-09-18 18:08:29.720106 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000054 0 0.000000 -6191> 2014-09-18 18:08:29.720120 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6190> 2014-09-18 18:08:29.720119 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -6189> 2014-09-18 18:08:29.720131 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -6188> 2014-09-18 18:08:29.720144 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -6187> 2014-09-18 18:08:29.720159 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 300 ==== pg_query(2.756 epoch 11708) v3 ==== 144+0+0 (3363559458 0 0) 0xf811de0 con 0x140f95a0 -6186> 2014-09-18 18:08:29.720172 7f4c9f251700 5 -- op tracker -- , seq: 85866, time: 2014-09-18 18:08:29.720115, event: header_read, op: pg_query(2.756 epoch 11708) -6185> 2014-09-18 18:08:29.720181 7f4c9f251700 5 -- op tracker -- , seq: 85866, time: 2014-09-18 18:08:29.720116, event: throttled, op: pg_query(2.756 epoch 11708) -6184> 2014-09-18 18:08:29.720187 7f4c9f251700 5 -- op tracker -- , seq: 85866, time: 2014-09-18 18:08:29.720144, event: all_read, op: pg_query(2.756 epoch 11708) -6183> 2014-09-18 18:08:29.720193 7f4c9f251700 5 -- op tracker -- , seq: 85866, time: 2014-09-18 18:08:29.720167, event: dispatched, op: pg_query(2.756 epoch 11708) -6182> 2014-09-18 18:08:29.720200 7f4c9f251700 5 -- op tracker -- , seq: 85866, time: 2014-09-18 18:08:29.720198, event: waiting_for_osdmap, op: pg_query(2.756 epoch 11708) -6181> 2014-09-18 18:08:29.720208 7f4c9f251700 5 -- op tracker -- , seq: 85866, time: 2014-09-18 18:08:29.720206, event: started, op: pg_query(2.756 epoch 11708) -6180> 2014-09-18 18:08:29.720242 7f4c9f251700 5 -- op tracker -- , seq: 85866, time: 2014-09-18 18:08:29.720240, event: done, op: pg_query(2.756 epoch 11708) -6179> 2014-09-18 18:08:29.720253 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 785 ==== pg_notify(2.2d8(173) epoch 11708) v5 ==== 10757+0+0 (443447201 0 0) 0x1421e5a0 con 0xe9a27e0 -6178> 2014-09-18 18:08:29.720267 7f4c9f251700 5 -- op tracker -- , seq: 85867, time: 2014-09-18 18:08:29.720014, event: header_read, op: pg_notify(2.2d8(173) epoch 11708) -6177> 2014-09-18 18:08:29.720275 7f4c9f251700 5 -- op tracker -- , seq: 85867, time: 2014-09-18 18:08:29.720015, event: throttled, op: pg_notify(2.2d8(173) epoch 11708) -6176> 2014-09-18 18:08:29.720283 7f4c9f251700 5 -- op tracker -- , seq: 85867, time: 2014-09-18 18:08:29.720198, event: all_read, op: pg_notify(2.2d8(173) epoch 11708) -6175> 2014-09-18 18:08:29.720290 7f4c9f251700 5 -- op tracker -- , seq: 85867, time: 2014-09-18 18:08:29.720262, event: dispatched, op: pg_notify(2.2d8(173) epoch 11708) -6174> 2014-09-18 18:08:29.720297 7f4c9f251700 5 -- op tracker -- , seq: 85867, time: 2014-09-18 18:08:29.720295, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11708) -6173> 2014-09-18 18:08:29.720306 7f4c9f251700 5 -- op tracker -- , seq: 85867, time: 2014-09-18 18:08:29.720304, event: started, op: pg_notify(2.2d8(173) epoch 11708) -6172> 2014-09-18 18:08:29.720309 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 27 ==== osd_map(11704..11708 src has 1..11708) v3 ==== 5186+0+0 (3949358766 0 0) 0xf954400 con 0x111771e0 -6171> 2014-09-18 18:08:29.720315 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11483/11483/11483) [32,3,46]/[32,46] r=-1 lpr=11483 pi=10661-11482/10 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started/Stray 385.737911 675 0.007962 -6170> 2014-09-18 18:08:29.720325 7f4c9f251700 5 -- op tracker -- , seq: 85867, time: 2014-09-18 18:08:29.720323, event: done, op: pg_notify(2.2d8(173) epoch 11708) -6169> 2014-09-18 18:08:29.720330 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11483/11483/11483) [32,3,46]/[32,46] r=-1 lpr=11483 pi=10661-11482/10 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started 385.737976 0 0.000000 -6168> 2014-09-18 18:08:29.720338 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11483/11483/11483) [32,3,46]/[32,46] r=-1 lpr=11483 pi=10661-11482/10 crt=0'0 lcod 0'0 remapped NOTIFY] enter Reset -6167> 2014-09-18 18:08:29.720345 7f4ca0a54700 3 osd.42 11708 handle_osd_map epochs [11704,11708], i have 11708, src has [1,11708] -6166> 2014-09-18 18:08:29.720465 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] exit Reset 0.000127 1 0.000056 -6165> 2014-09-18 18:08:29.720481 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started -6164> 2014-09-18 18:08:29.720491 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Start -6163> 2014-09-18 18:08:29.720500 7f4c97241700 1 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -6162> 2014-09-18 18:08:29.720516 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] exit Start 0.000025 0 0.000000 -6161> 2014-09-18 18:08:29.720530 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11341/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/Stray -6160> 2014-09-18 18:08:29.720562 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1262 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x14408d80 con 0xe8db760 -6159> 2014-09-18 18:08:29.720584 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6158> 2014-09-18 18:08:29.720701 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1263 ==== pg_notify(2.17e(17) epoch 11708) v5 ==== 1593+0+0 (2773586974 0 0) 0xf0ddc00 con 0xe8db760 -6157> 2014-09-18 18:08:29.720719 7f4c9f251700 5 -- op tracker -- , seq: 85868, time: 2014-09-18 18:08:29.720565, event: header_read, op: pg_notify(2.17e(17) epoch 11708) -6156> 2014-09-18 18:08:29.720732 7f4c9f251700 5 -- op tracker -- , seq: 85868, time: 2014-09-18 18:08:29.720566, event: throttled, op: pg_notify(2.17e(17) epoch 11708) -6155> 2014-09-18 18:08:29.720742 7f4c9f251700 5 -- op tracker -- , seq: 85868, time: 2014-09-18 18:08:29.720675, event: all_read, op: pg_notify(2.17e(17) epoch 11708) -6154> 2014-09-18 18:08:29.720746 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.756(27) epoch 11708) v5 -- ?+0 0xf72f2c0 con 0x140f95a0 -6153> 2014-09-18 18:08:29.720752 7f4c9f251700 5 -- op tracker -- , seq: 85868, time: 2014-09-18 18:08:29.720714, event: dispatched, op: pg_notify(2.17e(17) epoch 11708) -6152> 2014-09-18 18:08:29.720760 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(11) epoch 11708) v5 -- ?+0 0x146deb40 con 0x1395ef20 -6151> 2014-09-18 18:08:29.720761 7f4c9f251700 5 -- op tracker -- , seq: 85868, time: 2014-09-18 18:08:29.720758, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11708) -6150> 2014-09-18 18:08:29.720776 7f4c9f251700 5 -- op tracker -- , seq: 85868, time: 2014-09-18 18:08:29.720774, event: started, op: pg_notify(2.17e(17) epoch 11708) -6149> 2014-09-18 18:08:29.720778 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_query(2.6f8 epoch 11708) v3 -- ?+0 0x2b940940 con 0x1395edc0 -6148> 2014-09-18 18:08:29.720797 7f4c9f251700 5 -- op tracker -- , seq: 85868, time: 2014-09-18 18:08:29.720795, event: done, op: pg_notify(2.17e(17) epoch 11708) -6147> 2014-09-18 18:08:29.720871 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_query(2.6f8 epoch 11708) v3 -- ?+0 0x2b93e1e0 con 0xcbecec0 -6146> 2014-09-18 18:08:29.721189 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11504/11504/11258) [27,6,42]/[42] r=0 lpr=11504 pi=5116-11503/26 bft=6,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 138.712556 307 0.001275 -6145> 2014-09-18 18:08:29.721208 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11504/11504/11258) [27,6,42]/[42] r=0 lpr=11504 pi=5116-11503/26 bft=6,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active 360.584521 0 0.000000 -6144> 2014-09-18 18:08:29.721219 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11504/11504/11258) [27,6,42]/[42] r=0 lpr=11504 pi=5116-11503/26 bft=6,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 364.073883 0 0.000000 -6143> 2014-09-18 18:08:29.721228 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11504/11504/11258) [27,6,42]/[42] r=0 lpr=11504 pi=5116-11503/26 bft=6,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+remapped] exit Started 364.073975 0 0.000000 -6142> 2014-09-18 18:08:29.721235 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11504/11504/11258) [27,6,42]/[42] r=0 lpr=11504 pi=5116-11503/26 bft=6,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -6141> 2014-09-18 18:08:29.721334 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000099 1 0.000145 -6140> 2014-09-18 18:08:29.721352 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped] enter Started -6139> 2014-09-18 18:08:29.721362 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped] enter Start -6138> 2014-09-18 18:08:29.721371 7f4c97241700 1 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -6137> 2014-09-18 18:08:29.721383 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000021 0 0.000000 -6136> 2014-09-18 18:08:29.721434 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1194 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x1182ab40 con 0x1088b440 -6135> 2014-09-18 18:08:29.721449 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6134> 2014-09-18 18:08:29.721498 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2351 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0xe2681c0 con 0xdd8c940 -6133> 2014-09-18 18:08:29.721512 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6132> 2014-09-18 18:08:29.721594 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.872157 4 0.000528 -6131> 2014-09-18 18:08:29.721394 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -6130> 2014-09-18 18:08:29.721657 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -6129> 2014-09-18 18:08:29.721681 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -6128> 2014-09-18 18:08:29.721656 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.008526 0 0.000000 -6127> 2014-09-18 18:08:29.721699 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Started/Primary 1.008578 0 0.000000 -6126> 2014-09-18 18:08:29.721747 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2352 ==== pg_notify(2.2c7(139) epoch 11708) v5 ==== 8651+0+0 (3367366939 0 0) 0xf2b7860 con 0xdd8c940 -6125> 2014-09-18 18:08:29.721747 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Started 1.008698 0 0.000000 -6124> 2014-09-18 18:08:29.721762 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11707/11707/11258) [27,42]/[42,35] r=0 lpr=11707 pi=5697-11706/154 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Reset -6123> 2014-09-18 18:08:29.721764 7f4c9f251700 5 -- op tracker -- , seq: 85869, time: 2014-09-18 18:08:29.721558, event: header_read, op: pg_notify(2.2c7(139) epoch 11708) -6122> 2014-09-18 18:08:29.721776 7f4c9f251700 5 -- op tracker -- , seq: 85869, time: 2014-09-18 18:08:29.721559, event: throttled, op: pg_notify(2.2c7(139) epoch 11708) -6121> 2014-09-18 18:08:29.721788 7f4c9f251700 5 -- op tracker -- , seq: 85869, time: 2014-09-18 18:08:29.721728, event: all_read, op: pg_notify(2.2c7(139) epoch 11708) -6120> 2014-09-18 18:08:29.721808 7f4c9f251700 5 -- op tracker -- , seq: 85869, time: 2014-09-18 18:08:29.721759, event: dispatched, op: pg_notify(2.2c7(139) epoch 11708) -6119> 2014-09-18 18:08:29.721811 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000049 1 0.000215 -6118> 2014-09-18 18:08:29.721827 7f4c9f251700 5 -- op tracker -- , seq: 85869, time: 2014-09-18 18:08:29.721824, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11708) -6117> 2014-09-18 18:08:29.721834 7f4c9f251700 5 -- op tracker -- , seq: 85869, time: 2014-09-18 18:08:29.721833, event: started, op: pg_notify(2.2c7(139) epoch 11708) -6116> 2014-09-18 18:08:29.721827 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started -6115> 2014-09-18 18:08:29.721838 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Start -6114> 2014-09-18 18:08:29.721848 7f4c9f251700 5 -- op tracker -- , seq: 85869, time: 2014-09-18 18:08:29.721847, event: done, op: pg_notify(2.2c7(139) epoch 11708) -6113> 2014-09-18 18:08:29.721848 7f4c96a40700 1 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -6112> 2014-09-18 18:08:29.721861 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000022 0 0.000000 -6111> 2014-09-18 18:08:29.721874 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -6110> 2014-09-18 18:08:29.721885 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -6109> 2014-09-18 18:08:29.721920 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1195 ==== pg_notify(2.2d8(173) epoch 11708) v5 ==== 10757+0+0 (443447201 0 0) 0x1390ab40 con 0x1088b440 -6108> 2014-09-18 18:08:29.721932 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -6107> 2014-09-18 18:08:29.721940 7f4c9f251700 5 -- op tracker -- , seq: 85870, time: 2014-09-18 18:08:29.721472, event: header_read, op: pg_notify(2.2d8(173) epoch 11708) -6106> 2014-09-18 18:08:29.721950 7f4c9f251700 5 -- op tracker -- , seq: 85870, time: 2014-09-18 18:08:29.721473, event: throttled, op: pg_notify(2.2d8(173) epoch 11708) -6105> 2014-09-18 18:08:29.721956 7f4c9f251700 5 -- op tracker -- , seq: 85870, time: 2014-09-18 18:08:29.721793, event: all_read, op: pg_notify(2.2d8(173) epoch 11708) -6104> 2014-09-18 18:08:29.721963 7f4c9f251700 5 -- op tracker -- , seq: 85870, time: 2014-09-18 18:08:29.721935, event: dispatched, op: pg_notify(2.2d8(173) epoch 11708) -6103> 2014-09-18 18:08:29.721966 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_query(2.11b epoch 11708) v3 -- ?+0 0x1393c940 con 0x1395ec60 -6102> 2014-09-18 18:08:29.721970 7f4c9f251700 5 -- op tracker -- , seq: 85870, time: 2014-09-18 18:08:29.721968, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11708) -6101> 2014-09-18 18:08:29.721986 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0xe937440 con 0x1395f8c0 -6100> 2014-09-18 18:08:29.721995 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.11b epoch 11708) v3 -- ?+0 0x1393d2a0 con 0x1395f8c0 -6099> 2014-09-18 18:08:29.722012 7f4c9f251700 5 -- op tracker -- , seq: 85870, time: 2014-09-18 18:08:29.722000, event: started, op: pg_notify(2.2d8(173) epoch 11708) -6098> 2014-09-18 18:08:29.722037 7f4c9f251700 5 -- op tracker -- , seq: 85870, time: 2014-09-18 18:08:29.722034, event: done, op: pg_notify(2.2d8(173) epoch 11708) -6097> 2014-09-18 18:08:29.722071 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3630 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x10d19a80 con 0xcbecec0 -6096> 2014-09-18 18:08:29.722086 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6095> 2014-09-18 18:08:29.722113 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2353 ==== pg_notify(2.457(21) epoch 11708) v5 ==== 1875+0+0 (2899030829 0 0) 0xf2b8ee0 con 0xdd8c940 -6094> 2014-09-18 18:08:29.722127 7f4c9f251700 5 -- op tracker -- , seq: 85871, time: 2014-09-18 18:08:29.721949, event: header_read, op: pg_notify(2.457(21) epoch 11708) -6093> 2014-09-18 18:08:29.722147 7f4c9f251700 5 -- op tracker -- , seq: 85871, time: 2014-09-18 18:08:29.721950, event: throttled, op: pg_notify(2.457(21) epoch 11708) -6092> 2014-09-18 18:08:29.722155 7f4c9f251700 5 -- op tracker -- , seq: 85871, time: 2014-09-18 18:08:29.722017, event: all_read, op: pg_notify(2.457(21) epoch 11708) -6091> 2014-09-18 18:08:29.722160 7f4c9f251700 5 -- op tracker -- , seq: 85871, time: 2014-09-18 18:08:29.722123, event: dispatched, op: pg_notify(2.457(21) epoch 11708) -6090> 2014-09-18 18:08:29.722165 7f4c9f251700 5 -- op tracker -- , seq: 85871, time: 2014-09-18 18:08:29.722163, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11708) -6089> 2014-09-18 18:08:29.722172 7f4c9f251700 5 -- op tracker -- , seq: 85871, time: 2014-09-18 18:08:29.722170, event: started, op: pg_notify(2.457(21) epoch 11708) -6088> 2014-09-18 18:08:29.722199 7f4c9f251700 5 -- op tracker -- , seq: 85871, time: 2014-09-18 18:08:29.722196, event: done, op: pg_notify(2.457(21) epoch 11708) -6087> 2014-09-18 18:08:29.722214 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3631 ==== pg_notify(2.179 epoch 11708) v5 ==== 722+0+0 (4234603849 0 0) 0x1247dfe0 con 0xcbecec0 -6086> 2014-09-18 18:08:29.722225 7f4c9f251700 5 -- op tracker -- , seq: 85872, time: 2014-09-18 18:08:29.721939, event: header_read, op: pg_notify(2.179 epoch 11708) -6085> 2014-09-18 18:08:29.722232 7f4c9f251700 5 -- op tracker -- , seq: 85872, time: 2014-09-18 18:08:29.721940, event: throttled, op: pg_notify(2.179 epoch 11708) -6084> 2014-09-18 18:08:29.722237 7f4c9f251700 5 -- op tracker -- , seq: 85872, time: 2014-09-18 18:08:29.721977, event: all_read, op: pg_notify(2.179 epoch 11708) -6083> 2014-09-18 18:08:29.722152 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11598/11599/11572) [9,42,57]/[9,42] r=1 lpr=11599 pi=11491-11598/3 luod=0'0 crt=6745'24728 lcod 0'0 active+remapped] exit Started/ReplicaActive/RepNotRecovering 147.043374 324 0.000710 -6082> 2014-09-18 18:08:29.722243 7f4c9f251700 5 -- op tracker -- , seq: 85872, time: 2014-09-18 18:08:29.722222, event: dispatched, op: pg_notify(2.179 epoch 11708) -6081> 2014-09-18 18:08:29.722250 7f4c9f251700 5 -- op tracker -- , seq: 85872, time: 2014-09-18 18:08:29.722248, event: waiting_for_osdmap, op: pg_notify(2.179 epoch 11708) -6080> 2014-09-18 18:08:29.722256 7f4c9f251700 5 -- op tracker -- , seq: 85872, time: 2014-09-18 18:08:29.722254, event: started, op: pg_notify(2.179 epoch 11708) -6079> 2014-09-18 18:08:29.722247 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11598/11599/11572) [9,42,57]/[9,42] r=1 lpr=11599 pi=11491-11598/3 luod=0'0 crt=6745'24728 lcod 0'0 active+remapped] exit Started/ReplicaActive 147.043480 0 0.000000 -6078> 2014-09-18 18:08:29.722267 7f4c9f251700 5 -- op tracker -- , seq: 85872, time: 2014-09-18 18:08:29.722266, event: done, op: pg_notify(2.179 epoch 11708) -6077> 2014-09-18 18:08:29.722264 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11598/11599/11572) [9,42,57]/[9,42] r=1 lpr=11599 pi=11491-11598/3 luod=0'0 crt=6745'24728 lcod 0'0 active+remapped] exit Started 148.046003 0 0.000000 -6076> 2014-09-18 18:08:29.722274 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 527 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x10d18d00 con 0x1395ef20 -6075> 2014-09-18 18:08:29.722281 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -6074> 2014-09-18 18:08:29.722277 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11598/11599/11572) [9,42,57]/[9,42] r=1 lpr=11599 pi=11491-11598/3 luod=0'0 crt=6745'24728 lcod 0'0 active+remapped] enter Reset -6073> 2014-09-18 18:08:29.722285 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3632 ==== pg_notify(2.4f6(12) epoch 11708) v5 ==== 1382+0+0 (3527153425 0 0) 0xe1ab0e0 con 0xcbecec0 -6072> 2014-09-18 18:08:29.722291 7f4c9f251700 5 -- op tracker -- , seq: 85873, time: 2014-09-18 18:08:29.721995, event: header_read, op: pg_notify(2.4f6(12) epoch 11708) -6071> 2014-09-18 18:08:29.722296 7f4c9f251700 5 -- op tracker -- , seq: 85873, time: 2014-09-18 18:08:29.721996, event: throttled, op: pg_notify(2.4f6(12) epoch 11708) -6070> 2014-09-18 18:08:29.722300 7f4c9f251700 5 -- op tracker -- , seq: 85873, time: 2014-09-18 18:08:29.722045, event: all_read, op: pg_notify(2.4f6(12) epoch 11708) -6069> 2014-09-18 18:08:29.722304 7f4c9f251700 5 -- op tracker -- , seq: 85873, time: 2014-09-18 18:08:29.722289, event: dispatched, op: pg_notify(2.4f6(12) epoch 11708) -6068> 2014-09-18 18:08:29.722308 7f4c9f251700 5 -- op tracker -- , seq: 85873, time: 2014-09-18 18:08:29.722307, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11708) -6067> 2014-09-18 18:08:29.722313 7f4c9f251700 5 -- op tracker -- , seq: 85873, time: 2014-09-18 18:08:29.722312, event: started, op: pg_notify(2.4f6(12) epoch 11708) -6066> 2014-09-18 18:08:29.722324 7f4c9f251700 5 -- op tracker -- , seq: 85873, time: 2014-09-18 18:08:29.722323, event: done, op: pg_notify(2.4f6(12) epoch 11708) -6065> 2014-09-18 18:08:29.722331 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 528 ==== pg_query(2.352 epoch 11708) v3 ==== 144+0+0 (4042539466 0 0) 0x146deb40 con 0x1395ef20 -6064> 2014-09-18 18:08:29.722338 7f4c9f251700 5 -- op tracker -- , seq: 85874, time: 2014-09-18 18:08:29.722124, event: header_read, op: pg_query(2.352 epoch 11708) -6063> 2014-09-18 18:08:29.722342 7f4c9f251700 5 -- op tracker -- , seq: 85874, time: 2014-09-18 18:08:29.722126, event: throttled, op: pg_query(2.352 epoch 11708) -6062> 2014-09-18 18:08:29.722287 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11695/11695/9945) [42,57,9]/[42,6] r=0 lpr=11695 pi=11585-11694/3 bft=9,57 crt=6868'25546 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 7.698050 22 0.000118 -6061> 2014-09-18 18:08:29.722346 7f4c9f251700 5 -- op tracker -- , seq: 85874, time: 2014-09-18 18:08:29.722164, event: all_read, op: pg_query(2.352 epoch 11708) -6060> 2014-09-18 18:08:29.722350 7f4c9f251700 5 -- op tracker -- , seq: 85874, time: 2014-09-18 18:08:29.722336, event: dispatched, op: pg_query(2.352 epoch 11708) -6059> 2014-09-18 18:08:29.722355 7f4c9f251700 5 -- op tracker -- , seq: 85874, time: 2014-09-18 18:08:29.722353, event: waiting_for_osdmap, op: pg_query(2.352 epoch 11708) -6058> 2014-09-18 18:08:29.722349 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11695/11695/9945) [42,57,9]/[42,6] r=0 lpr=11695 pi=11585-11694/3 bft=9,57 crt=6868'25546 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 12.730504 0 0.000000 -6057> 2014-09-18 18:08:29.722360 7f4c9f251700 5 -- op tracker -- , seq: 85874, time: 2014-09-18 18:08:29.722358, event: started, op: pg_query(2.352 epoch 11708) -6056> 2014-09-18 18:08:29.722369 7f4c9f251700 5 -- op tracker -- , seq: 85874, time: 2014-09-18 18:08:29.722367, event: done, op: pg_query(2.352 epoch 11708) -6055> 2014-09-18 18:08:29.722362 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11695/11695/9945) [42,57,9]/[42,6] r=0 lpr=11695 pi=11585-11694/3 bft=9,57 crt=6868'25546 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 13.878525 0 0.000000 -6054> 2014-09-18 18:08:29.722377 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3633 ==== pg_notify(2.6f8 epoch 11708) v5 ==== 722+0+0 (3711463301 0 0) 0x2b93e1e0 con 0xcbecec0 -6053> 2014-09-18 18:08:29.722387 7f4c9f251700 5 -- op tracker -- , seq: 85875, time: 2014-09-18 18:08:29.722105, event: header_read, op: pg_notify(2.6f8 epoch 11708) -6052> 2014-09-18 18:08:29.722393 7f4c9f251700 5 -- op tracker -- , seq: 85875, time: 2014-09-18 18:08:29.722105, event: throttled, op: pg_notify(2.6f8 epoch 11708) -6051> 2014-09-18 18:08:29.722397 7f4c9f251700 5 -- op tracker -- , seq: 85875, time: 2014-09-18 18:08:29.722135, event: all_read, op: pg_notify(2.6f8 epoch 11708) -6050> 2014-09-18 18:08:29.722400 7f4c9f251700 5 -- op tracker -- , seq: 85875, time: 2014-09-18 18:08:29.722385, event: dispatched, op: pg_notify(2.6f8 epoch 11708) -6049> 2014-09-18 18:08:29.722405 7f4c9f251700 5 -- op tracker -- , seq: 85875, time: 2014-09-18 18:08:29.722403, event: waiting_for_osdmap, op: pg_notify(2.6f8 epoch 11708) -6048> 2014-09-18 18:08:29.722410 7f4c9f251700 5 -- op tracker -- , seq: 85875, time: 2014-09-18 18:08:29.722408, event: started, op: pg_notify(2.6f8 epoch 11708) -6047> 2014-09-18 18:08:29.722420 7f4c9f251700 5 -- op tracker -- , seq: 85875, time: 2014-09-18 18:08:29.722418, event: done, op: pg_notify(2.6f8 epoch 11708) -6046> 2014-09-18 18:08:29.722376 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11695/11695/9945) [42,57,9]/[42,6] r=0 lpr=11695 pi=11585-11694/3 bft=9,57 crt=6868'25546 lcod 0'0 mlcod 0'0 active+remapped] exit Started 13.878580 0 0.000000 -6045> 2014-09-18 18:08:29.722436 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11695/11695/9945) [42,57,9]/[42,6] r=0 lpr=11695 pi=11585-11694/3 bft=9,57 crt=6868'25546 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -6044> 2014-09-18 18:08:29.722509 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000073 1 0.000260 -6043> 2014-09-18 18:08:29.722528 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped] enter Started -6042> 2014-09-18 18:08:29.722535 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped] enter Start -6041> 2014-09-18 18:08:29.722549 7f4c96a40700 1 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -6040> 2014-09-18 18:08:29.722556 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000022 0 0.000000 -6039> 2014-09-18 18:08:29.722563 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -6038> 2014-09-18 18:08:29.722570 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -6037> 2014-09-18 18:08:29.722581 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -6036> 2014-09-18 18:08:29.722728 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] exit Reset 0.000451 1 0.000514 -6035> 2014-09-18 18:08:29.722760 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] enter Started -6034> 2014-09-18 18:08:29.722770 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] enter Start -6033> 2014-09-18 18:08:29.722780 7f4c97241700 1 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -6032> 2014-09-18 18:08:29.722792 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] exit Start 0.000021 0 0.000000 -6031> 2014-09-18 18:08:29.722790 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11504/11504/11332) [47,27,42]/[35,42] r=1 lpr=11504 pi=10768-11503/10 luod=0'0 crt=6775'24651 lcod 0'0 active+remapped] exit Started/ReplicaActive/RepNotRecovering 362.208778 609 0.001535 -6030> 2014-09-18 18:08:29.722804 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] enter Started/Stray -6029> 2014-09-18 18:08:29.722815 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11504/11504/11332) [47,27,42]/[35,42] r=1 lpr=11504 pi=10768-11503/10 luod=0'0 crt=6775'24651 lcod 0'0 active+remapped] exit Started/ReplicaActive 362.208836 0 0.000000 -6028> 2014-09-18 18:08:29.722824 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11504/11504/11332) [47,27,42]/[35,42] r=1 lpr=11504 pi=10768-11503/10 luod=0'0 crt=6775'24651 lcod 0'0 active+remapped] exit Started 364.077663 0 0.000000 -6027> 2014-09-18 18:08:29.722836 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11504/11504/11332) [47,27,42]/[35,42] r=1 lpr=11504 pi=10768-11503/10 luod=0'0 crt=6775'24651 lcod 0'0 active+remapped] enter Reset -6026> 2014-09-18 18:08:29.722973 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] exit Reset 0.000137 1 0.000093 -6025> 2014-09-18 18:08:29.722995 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Started -6024> 2014-09-18 18:08:29.723004 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Start -6023> 2014-09-18 18:08:29.723065 7f4c96a40700 1 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -6022> 2014-09-18 18:08:29.723083 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] exit Start 0.000079 0 0.000000 -6021> 2014-09-18 18:08:29.723082 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 0.406198 1 0.000081 -6020> 2014-09-18 18:08:29.723099 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Started/Stray -6019> 2014-09-18 18:08:29.723104 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 1.393271 0 0.000000 -6018> 2014-09-18 18:08:29.723123 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 6.808691 0 0.000000 -6017> 2014-09-18 18:08:29.723193 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+remapped] exit Started 6.808811 0 0.000000 -6016> 2014-09-18 18:08:29.723226 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11701/11701/9945) [42,6,27]/[42] r=0 lpr=11701 pi=9945-11700/11 bft=6,27 crt=6840'31228 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -6015> 2014-09-18 18:08:29.723332 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(11) epoch 11708) v5 -- ?+0 0x12139fe0 con 0xcbecec0 -6014> 2014-09-18 18:08:29.723371 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6854/39317 -- osd_map(11704..11708 src has 1..11708) v3 -- ?+0 0x1533dcc0 con 0x12b3aec0 -6013> 2014-09-18 18:08:29.723382 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6854/39317 -- pg_query(2.78f epoch 11708) v3 -- ?+0 0x12139a40 con 0x12b3aec0 -6012> 2014-09-18 18:08:29.723430 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6819/22324 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x1533a6c0 con 0x1f102840 -6011> 2014-09-18 18:08:29.723456 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6819/22324 -- pg_query(2.519 epoch 11708) v3 -- ?+0 0x1368cf00 con 0x1f102840 -6010> 2014-09-18 18:08:29.723455 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000229 1 0.000210 -6009> 2014-09-18 18:08:29.723475 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] enter Started -6008> 2014-09-18 18:08:29.723483 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] enter Start -6007> 2014-09-18 18:08:29.723490 7f4c97241700 1 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -6006> 2014-09-18 18:08:29.723484 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x1533da80 con 0x1088aec0 -6005> 2014-09-18 18:08:29.723500 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000017 0 0.000000 -6004> 2014-09-18 18:08:29.723509 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -6003> 2014-09-18 18:08:29.723512 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_query(2.519 epoch 11708) v3 -- ?+0 0x1368cd20 con 0x1088aec0 -6002> 2014-09-18 18:08:29.723518 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -6001> 2014-09-18 18:08:29.723526 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -6000> 2014-09-18 18:08:29.723587 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.78f epoch 11708) v3 -- ?+0 0x2b987860 con 0x1395f8c0 -5999> 2014-09-18 18:08:29.723623 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_query(2.78f epoch 11708) v3 -- ?+0 0x2b9863c0 con 0xcbecec0 -5998> 2014-09-18 18:08:29.723642 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- pg_query(2.519 epoch 11708) v3 -- ?+0 0x2b987e00 con 0x1395f080 -5997> 2014-09-18 18:08:29.723722 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.1bb( v 6791'24086 (4155'21085,6791'24086] local-les=11407 n=551 ec=188 les/c 11407/9946 11406/11406/9945) [42,27,0]/[42] r=0 lpr=11406 pi=9945-11405/9 bft=0,27 crt=6752'24077 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 139.933053 514 0.002899 -5996> 2014-09-18 18:08:29.723745 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.1bb( v 6791'24086 (4155'21085,6791'24086] local-les=11407 n=551 ec=188 les/c 11407/9946 11406/11406/9945) [42,27,0]/[42] r=0 lpr=11406 pi=9945-11405/9 bft=0,27 crt=6752'24077 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved -5995> 2014-09-18 18:08:29.723772 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- MBackfillReserve REQUEST pgid: 2.1bb, query_epoch: 11708, prio: 202 v3 -- ?+0 0xf788580 con 0x10a878c0 -5994> 2014-09-18 18:08:29.724043 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.756(27) epoch 11708) v5 -- ?+0 0x12ba23a0 con 0x140f95a0 -5993> 2014-09-18 18:08:29.724058 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(11) epoch 11708) v5 -- ?+0 0xfccab20 con 0x1395ef20 -5992> 2014-09-18 18:08:29.724078 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.e0( v 6854'28782 (4157'25783,6854'28782] local-les=11407 n=509 ec=188 les/c 11407/9946 11406/11406/10777) [27,42,0]/[42] r=0 lpr=11406 pi=5117-11405/79 bft=0,27 crt=6854'28778 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 139.946582 413 0.003388 -5991> 2014-09-18 18:08:29.724108 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.e0( v 6854'28782 (4157'25783,6854'28782] local-les=11407 n=509 ec=188 les/c 11407/9946 11406/11406/10777) [27,42,0]/[42] r=0 lpr=11406 pi=5117-11405/79 bft=0,27 crt=6854'28778 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved -5990> 2014-09-18 18:08:29.724120 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- MBackfillReserve REQUEST pgid: 2.e0, query_epoch: 11708, prio: 202 v3 -- ?+0 0x21e12580 con 0x10a878c0 -5989> 2014-09-18 18:08:29.724164 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.67b(4) epoch 11708) v5 -- ?+0 0xea00940 con 0x1395ec60 -5988> 2014-09-18 18:08:29.724307 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x1440a400 con 0x23a74f20 -5987> 2014-09-18 18:08:29.724346 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11708) v5 -- ?+0 0x24deb840 con 0x23a74f20 -5986> 2014-09-18 18:08:29.724382 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.410 epoch 11708) v3 -- ?+0 0x24dea3a0 con 0x1395f8c0 -5985> 2014-09-18 18:08:29.724430 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x19df2d00 con 0x14358260 -5984> 2014-09-18 18:08:29.724447 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- pg_query(2.410 epoch 11708) v3 -- ?+0 0x24de8780 con 0x14358260 -5983> 2014-09-18 18:08:29.724874 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1038 ==== pg_notify(2.410 epoch 11708) v5 ==== 722+0+0 (2946361851 0 0) 0x24de8780 con 0x14358260 -5982> 2014-09-18 18:08:29.724891 7f4c9f251700 5 -- op tracker -- , seq: 85876, time: 2014-09-18 18:08:29.724835, event: header_read, op: pg_notify(2.410 epoch 11708) -5981> 2014-09-18 18:08:29.724901 7f4c9f251700 5 -- op tracker -- , seq: 85876, time: 2014-09-18 18:08:29.724836, event: throttled, op: pg_notify(2.410 epoch 11708) -5980> 2014-09-18 18:08:29.724908 7f4c9f251700 5 -- op tracker -- , seq: 85876, time: 2014-09-18 18:08:29.724859, event: all_read, op: pg_notify(2.410 epoch 11708) -5979> 2014-09-18 18:08:29.724914 7f4c9f251700 5 -- op tracker -- , seq: 85876, time: 2014-09-18 18:08:29.724887, event: dispatched, op: pg_notify(2.410 epoch 11708) -5978> 2014-09-18 18:08:29.724920 7f4c9f251700 5 -- op tracker -- , seq: 85876, time: 2014-09-18 18:08:29.724919, event: waiting_for_osdmap, op: pg_notify(2.410 epoch 11708) -5977> 2014-09-18 18:08:29.724928 7f4c9f251700 5 -- op tracker -- , seq: 85876, time: 2014-09-18 18:08:29.724926, event: started, op: pg_notify(2.410 epoch 11708) -5976> 2014-09-18 18:08:29.724943 7f4c9f251700 5 -- op tracker -- , seq: 85876, time: 2014-09-18 18:08:29.724942, event: done, op: pg_notify(2.410 epoch 11708) -5975> 2014-09-18 18:08:29.725295 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2009 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0xef578c0 con 0xe2d9a20 -5974> 2014-09-18 18:08:29.725313 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5973> 2014-09-18 18:08:29.725367 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2010 ==== pg_notify(2.26d(9) epoch 11708) v5 ==== 1257+0+0 (1158262362 0 0) 0xec6b660 con 0xe2d9a20 -5972> 2014-09-18 18:08:29.725381 7f4c9f251700 5 -- op tracker -- , seq: 85877, time: 2014-09-18 18:08:29.725307, event: header_read, op: pg_notify(2.26d(9) epoch 11708) -5971> 2014-09-18 18:08:29.725389 7f4c9f251700 5 -- op tracker -- , seq: 85877, time: 2014-09-18 18:08:29.725307, event: throttled, op: pg_notify(2.26d(9) epoch 11708) -5970> 2014-09-18 18:08:29.725396 7f4c9f251700 5 -- op tracker -- , seq: 85877, time: 2014-09-18 18:08:29.725351, event: all_read, op: pg_notify(2.26d(9) epoch 11708) -5969> 2014-09-18 18:08:29.725401 7f4c9f251700 5 -- op tracker -- , seq: 85877, time: 2014-09-18 18:08:29.725377, event: dispatched, op: pg_notify(2.26d(9) epoch 11708) -5968> 2014-09-18 18:08:29.725407 7f4c9f251700 5 -- op tracker -- , seq: 85877, time: 2014-09-18 18:08:29.725406, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11708) -5967> 2014-09-18 18:08:29.725415 7f4c9f251700 5 -- op tracker -- , seq: 85877, time: 2014-09-18 18:08:29.725414, event: started, op: pg_notify(2.26d(9) epoch 11708) -5966> 2014-09-18 18:08:29.725435 7f4c9f251700 5 -- op tracker -- , seq: 85877, time: 2014-09-18 18:08:29.725433, event: done, op: pg_notify(2.26d(9) epoch 11708) -5965> 2014-09-18 18:08:29.726147 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3634 ==== pg_query(2.654 epoch 11708) v3 ==== 144+0+0 (3657590874 0 0) 0x2b9863c0 con 0xcbecec0 -5964> 2014-09-18 18:08:29.726166 7f4c9f251700 5 -- op tracker -- , seq: 85878, time: 2014-09-18 18:08:29.726110, event: header_read, op: pg_query(2.654 epoch 11708) -5963> 2014-09-18 18:08:29.726181 7f4c9f251700 5 -- op tracker -- , seq: 85878, time: 2014-09-18 18:08:29.726110, event: throttled, op: pg_query(2.654 epoch 11708) -5962> 2014-09-18 18:08:29.726191 7f4c9f251700 5 -- op tracker -- , seq: 85878, time: 2014-09-18 18:08:29.726133, event: all_read, op: pg_query(2.654 epoch 11708) -5961> 2014-09-18 18:08:29.726201 7f4c9f251700 5 -- op tracker -- , seq: 85878, time: 2014-09-18 18:08:29.726161, event: dispatched, op: pg_query(2.654 epoch 11708) -5960> 2014-09-18 18:08:29.726230 7f4c9f251700 5 -- op tracker -- , seq: 85878, time: 2014-09-18 18:08:29.726210, event: waiting_for_osdmap, op: pg_query(2.654 epoch 11708) -5959> 2014-09-18 18:08:29.726243 7f4c9f251700 5 -- op tracker -- , seq: 85878, time: 2014-09-18 18:08:29.726241, event: started, op: pg_query(2.654 epoch 11708) -5958> 2014-09-18 18:08:29.726265 7f4c9f251700 5 -- op tracker -- , seq: 85878, time: 2014-09-18 18:08:29.726263, event: done, op: pg_query(2.654 epoch 11708) -5957> 2014-09-18 18:08:29.726285 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(11) epoch 11708) v5 -- ?+0 0xf93c3a0 con 0xcbecec0 -5956> 2014-09-18 18:08:29.726783 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 529 ==== pg_notify(2.178(3) epoch 11708) v5 ==== 907+0+0 (855225080 0 0) 0xfccab20 con 0x1395ef20 -5955> 2014-09-18 18:08:29.726807 7f4c9f251700 5 -- op tracker -- , seq: 85879, time: 2014-09-18 18:08:29.726729, event: header_read, op: pg_notify(2.178(3) epoch 11708) -5954> 2014-09-18 18:08:29.726821 7f4c9f251700 5 -- op tracker -- , seq: 85879, time: 2014-09-18 18:08:29.726729, event: throttled, op: pg_notify(2.178(3) epoch 11708) -5953> 2014-09-18 18:08:29.726832 7f4c9f251700 5 -- op tracker -- , seq: 85879, time: 2014-09-18 18:08:29.726757, event: all_read, op: pg_notify(2.178(3) epoch 11708) -5952> 2014-09-18 18:08:29.726840 7f4c9f251700 5 -- op tracker -- , seq: 85879, time: 2014-09-18 18:08:29.726801, event: dispatched, op: pg_notify(2.178(3) epoch 11708) -5951> 2014-09-18 18:08:29.726856 7f4c9f251700 5 -- op tracker -- , seq: 85879, time: 2014-09-18 18:08:29.726849, event: waiting_for_osdmap, op: pg_notify(2.178(3) epoch 11708) -5950> 2014-09-18 18:08:29.726864 7f4c9f251700 5 -- op tracker -- , seq: 85879, time: 2014-09-18 18:08:29.726862, event: started, op: pg_notify(2.178(3) epoch 11708) -5949> 2014-09-18 18:08:29.726882 7f4c9f251700 5 -- op tracker -- , seq: 85879, time: 2014-09-18 18:08:29.726880, event: done, op: pg_notify(2.178(3) epoch 11708) -5948> 2014-09-18 18:08:29.726898 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2594 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x19df0480 con 0x4900b00 -5947> 2014-09-18 18:08:29.726912 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5946> 2014-09-18 18:08:29.726950 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.59 10.242.42.178:6840/4508 1 ==== pg_notify(2.4e8(7) epoch 11708) v5 ==== 1139+0+0 (1881673174 0 0) 0x24dea760 con 0x23af51e0 -5945> 2014-09-18 18:08:29.726976 7f4c9f251700 5 -- op tracker -- , seq: 85880, time: 2014-09-18 18:08:29.726848, event: header_read, op: pg_notify(2.4e8(7) epoch 11708) -5944> 2014-09-18 18:08:29.726988 7f4c9f251700 5 -- op tracker -- , seq: 85880, time: 2014-09-18 18:08:29.726850, event: throttled, op: pg_notify(2.4e8(7) epoch 11708) -5943> 2014-09-18 18:08:29.726995 7f4c9f251700 5 -- op tracker -- , seq: 85880, time: 2014-09-18 18:08:29.726917, event: all_read, op: pg_notify(2.4e8(7) epoch 11708) -5942> 2014-09-18 18:08:29.727001 7f4c9f251700 5 -- op tracker -- , seq: 85880, time: 2014-09-18 18:08:29.726970, event: dispatched, op: pg_notify(2.4e8(7) epoch 11708) -5941> 2014-09-18 18:08:29.727008 7f4c9f251700 5 -- op tracker -- , seq: 85880, time: 2014-09-18 18:08:29.727006, event: waiting_for_osdmap, op: pg_notify(2.4e8(7) epoch 11708) -5940> 2014-09-18 18:08:29.727015 7f4c9f251700 5 -- op tracker -- , seq: 85880, time: 2014-09-18 18:08:29.727014, event: started, op: pg_notify(2.4e8(7) epoch 11708) -5939> 2014-09-18 18:08:29.727080 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8(unlocked)] enter Initial -5938> 2014-09-18 18:08:29.727133 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=0 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] exit Initial 0.000053 0 0.000000 -5937> 2014-09-18 18:08:29.727146 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=0 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] enter Reset -5936> 2014-09-18 18:08:29.727156 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] exit Reset 0.000010 1 0.000022 -5935> 2014-09-18 18:08:29.727165 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] enter Started -5934> 2014-09-18 18:08:29.727173 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] enter Start -5933> 2014-09-18 18:08:29.727180 7f4c9f251700 1 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] state: transitioning to Primary -5932> 2014-09-18 18:08:29.727188 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] exit Start 0.000015 0 0.000000 -5931> 2014-09-18 18:08:29.727197 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] enter Started/Primary -5930> 2014-09-18 18:08:29.727205 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -5929> 2014-09-18 18:08:29.727212 7f4c9f251700 5 osd.42 pg_epoch: 11708 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -5928> 2014-09-18 18:08:29.727243 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6823/20623 -- osd_map(11708..11708 src has 1..11708) v3 -- ?+0 0x19df0480 con 0x1f103e40 -5927> 2014-09-18 18:08:29.727257 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6823/20623 -- pg_query(2.4e8 epoch 11708) v3 -- ?+0 0xfccab20 con 0x1f103e40 -5926> 2014-09-18 18:08:29.727297 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_query(2.4e8 epoch 11708) v3 -- ?+0 0x2b9863c0 con 0x1088aec0 -5925> 2014-09-18 18:08:29.727310 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- pg_query(2.4e8 epoch 11708) v3 -- ?+0 0xec6b660 con 0x1395f080 -5924> 2014-09-18 18:08:29.727475 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6840/4508 -- osd_map(11609..11708 src has 1..11708) v3 -- ?+0 0xef578c0 con 0x23af51e0 -5923> 2014-09-18 18:08:29.727487 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6840/4508 -- pg_query(2.4e8 epoch 11708) v3 -- ?+0 0x24de8780 con 0x23af51e0 -5922> 2014-09-18 18:08:29.727699 7f4c9f251700 5 -- op tracker -- , seq: 85880, time: 2014-09-18 18:08:29.727693, event: done, op: pg_notify(2.4e8 epoch 11708) -5921> 2014-09-18 18:08:29.727719 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2595 ==== pg_notify(2.71c(12) epoch 11708) v5 ==== 1424+0+0 (1907660018 0 0) 0xf7a8760 con 0x4900b00 -5920> 2014-09-18 18:08:29.727731 7f4c9f251700 5 -- op tracker -- , seq: 85881, time: 2014-09-18 18:08:29.726892, event: header_read, op: pg_notify(2.71c(12) epoch 11708) -5919> 2014-09-18 18:08:29.727739 7f4c9f251700 5 -- op tracker -- , seq: 85881, time: 2014-09-18 18:08:29.726893, event: throttled, op: pg_notify(2.71c(12) epoch 11708) -5918> 2014-09-18 18:08:29.727746 7f4c9f251700 5 -- op tracker -- , seq: 85881, time: 2014-09-18 18:08:29.726935, event: all_read, op: pg_notify(2.71c(12) epoch 11708) -5917> 2014-09-18 18:08:29.727753 7f4c9f251700 5 -- op tracker -- , seq: 85881, time: 2014-09-18 18:08:29.727728, event: dispatched, op: pg_notify(2.71c(12) epoch 11708) -5916> 2014-09-18 18:08:29.727759 7f4c9f251700 5 -- op tracker -- , seq: 85881, time: 2014-09-18 18:08:29.727757, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11708) -5915> 2014-09-18 18:08:29.727767 7f4c9f251700 5 -- op tracker -- , seq: 85881, time: 2014-09-18 18:08:29.727766, event: started, op: pg_notify(2.71c(12) epoch 11708) -5914> 2014-09-18 18:08:29.727788 7f4c9f251700 5 -- op tracker -- , seq: 85881, time: 2014-09-18 18:08:29.727786, event: done, op: pg_notify(2.71c(12) epoch 11708) -5913> 2014-09-18 18:08:29.727798 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3290 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x12bf1b00 con 0x4900580 -5912> 2014-09-18 18:08:29.727809 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5911> 2014-09-18 18:08:29.727915 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3291 ==== pg_notify(2.4f6(184) epoch 11708) v5 ==== 11296+0+0 (1282095497 0 0) 0xf4e72c0 con 0x4900580 -5910> 2014-09-18 18:08:29.727927 7f4c9f251700 5 -- op tracker -- , seq: 85882, time: 2014-09-18 18:08:29.727746, event: header_read, op: pg_notify(2.4f6(184) epoch 11708) -5909> 2014-09-18 18:08:29.727933 7f4c9f251700 5 -- op tracker -- , seq: 85882, time: 2014-09-18 18:08:29.727747, event: throttled, op: pg_notify(2.4f6(184) epoch 11708) -5908> 2014-09-18 18:08:29.727937 7f4c9f251700 5 -- op tracker -- , seq: 85882, time: 2014-09-18 18:08:29.727902, event: all_read, op: pg_notify(2.4f6(184) epoch 11708) -5907> 2014-09-18 18:08:29.727941 7f4c9f251700 5 -- op tracker -- , seq: 85882, time: 2014-09-18 18:08:29.727925, event: dispatched, op: pg_notify(2.4f6(184) epoch 11708) -5906> 2014-09-18 18:08:29.727948 7f4c9f251700 5 -- op tracker -- , seq: 85882, time: 2014-09-18 18:08:29.727946, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11708) -5905> 2014-09-18 18:08:29.727956 7f4c9f251700 5 -- op tracker -- , seq: 85882, time: 2014-09-18 18:08:29.727954, event: started, op: pg_notify(2.4f6(184) epoch 11708) -5904> 2014-09-18 18:08:29.728028 7f4c9f251700 5 -- op tracker -- , seq: 85882, time: 2014-09-18 18:08:29.728026, event: done, op: pg_notify(2.4f6(184) epoch 11708) -5903> 2014-09-18 18:08:29.728123 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3635 ==== pg_notify(2.3b6(10) epoch 11708) v5 ==== 1260+0+0 (1246706749 0 0) 0xf93c3a0 con 0xcbecec0 -5902> 2014-09-18 18:08:29.728143 7f4c9f251700 5 -- op tracker -- , seq: 85883, time: 2014-09-18 18:08:29.728081, event: header_read, op: pg_notify(2.3b6(10) epoch 11708) -5901> 2014-09-18 18:08:29.728154 7f4c9f251700 5 -- op tracker -- , seq: 85883, time: 2014-09-18 18:08:29.728082, event: throttled, op: pg_notify(2.3b6(10) epoch 11708) -5900> 2014-09-18 18:08:29.728163 7f4c9f251700 5 -- op tracker -- , seq: 85883, time: 2014-09-18 18:08:29.728110, event: all_read, op: pg_notify(2.3b6(10) epoch 11708) -5899> 2014-09-18 18:08:29.728174 7f4c9f251700 5 -- op tracker -- , seq: 85883, time: 2014-09-18 18:08:29.728138, event: dispatched, op: pg_notify(2.3b6(10) epoch 11708) -5898> 2014-09-18 18:08:29.728185 7f4c9f251700 5 -- op tracker -- , seq: 85883, time: 2014-09-18 18:08:29.728183, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11708) -5897> 2014-09-18 18:08:29.728194 7f4c9f251700 5 -- op tracker -- , seq: 85883, time: 2014-09-18 18:08:29.728192, event: started, op: pg_notify(2.3b6(10) epoch 11708) -5896> 2014-09-18 18:08:29.728218 7f4c9f251700 5 -- op tracker -- , seq: 85883, time: 2014-09-18 18:08:29.728216, event: done, op: pg_notify(2.3b6(10) epoch 11708) -5895> 2014-09-18 18:08:29.728367 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.59 10.242.42.178:6840/4508 2 ==== pg_notify(2.4e8(7) epoch 11708) v5 ==== 1139+0+0 (1881673174 0 0) 0x24de8780 con 0x23af51e0 -5894> 2014-09-18 18:08:29.728382 7f4c9f251700 5 -- op tracker -- , seq: 85884, time: 2014-09-18 18:08:29.728323, event: header_read, op: pg_notify(2.4e8(7) epoch 11708) -5893> 2014-09-18 18:08:29.728390 7f4c9f251700 5 -- op tracker -- , seq: 85884, time: 2014-09-18 18:08:29.728324, event: throttled, op: pg_notify(2.4e8(7) epoch 11708) -5892> 2014-09-18 18:08:29.728395 7f4c9f251700 5 -- op tracker -- , seq: 85884, time: 2014-09-18 18:08:29.728353, event: all_read, op: pg_notify(2.4e8(7) epoch 11708) -5891> 2014-09-18 18:08:29.728401 7f4c9f251700 5 -- op tracker -- , seq: 85884, time: 2014-09-18 18:08:29.728378, event: dispatched, op: pg_notify(2.4e8(7) epoch 11708) -5890> 2014-09-18 18:08:29.728405 7f4c9f251700 5 -- op tracker -- , seq: 85884, time: 2014-09-18 18:08:29.728404, event: waiting_for_osdmap, op: pg_notify(2.4e8(7) epoch 11708) -5889> 2014-09-18 18:08:29.728413 7f4c9f251700 5 -- op tracker -- , seq: 85884, time: 2014-09-18 18:08:29.728411, event: started, op: pg_notify(2.4e8(7) epoch 11708) -5888> 2014-09-18 18:08:29.728430 7f4c9f251700 5 -- op tracker -- , seq: 85884, time: 2014-09-18 18:08:29.728428, event: done, op: pg_notify(2.4e8(7) epoch 11708) -5887> 2014-09-18 18:08:29.729026 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3292 ==== pg_notify(2.17e(17) epoch 11708) v5 ==== 1593+0+0 (2773586974 0 0) 0xf1e7680 con 0x4900580 -5886> 2014-09-18 18:08:29.729042 7f4c9f251700 5 -- op tracker -- , seq: 85885, time: 2014-09-18 18:08:29.728980, event: header_read, op: pg_notify(2.17e(17) epoch 11708) -5885> 2014-09-18 18:08:29.729050 7f4c9f251700 5 -- op tracker -- , seq: 85885, time: 2014-09-18 18:08:29.728981, event: throttled, op: pg_notify(2.17e(17) epoch 11708) -5884> 2014-09-18 18:08:29.729055 7f4c9f251700 5 -- op tracker -- , seq: 85885, time: 2014-09-18 18:08:29.729012, event: all_read, op: pg_notify(2.17e(17) epoch 11708) -5883> 2014-09-18 18:08:29.729061 7f4c9f251700 5 -- op tracker -- , seq: 85885, time: 2014-09-18 18:08:29.729038, event: dispatched, op: pg_notify(2.17e(17) epoch 11708) -5882> 2014-09-18 18:08:29.729066 7f4c9f251700 5 -- op tracker -- , seq: 85885, time: 2014-09-18 18:08:29.729064, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11708) -5881> 2014-09-18 18:08:29.729073 7f4c9f251700 5 -- op tracker -- , seq: 85885, time: 2014-09-18 18:08:29.729072, event: started, op: pg_notify(2.17e(17) epoch 11708) -5880> 2014-09-18 18:08:29.729091 7f4c9f251700 5 -- op tracker -- , seq: 85885, time: 2014-09-18 18:08:29.729089, event: done, op: pg_notify(2.17e(17) epoch 11708) -5879> 2014-09-18 18:08:29.729112 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 925 ==== pg_notify(2.410(12) epoch 11708) v5 ==== 1338+0+0 (1346775082 0 0) 0x24dea3a0 con 0x1395f8c0 -5878> 2014-09-18 18:08:29.729123 7f4c9f251700 5 -- op tracker -- , seq: 85886, time: 2014-09-18 18:08:29.729061, event: header_read, op: pg_notify(2.410(12) epoch 11708) -5877> 2014-09-18 18:08:29.729130 7f4c9f251700 5 -- op tracker -- , seq: 85886, time: 2014-09-18 18:08:29.729062, event: throttled, op: pg_notify(2.410(12) epoch 11708) -5876> 2014-09-18 18:08:29.729135 7f4c9f251700 5 -- op tracker -- , seq: 85886, time: 2014-09-18 18:08:29.729099, event: all_read, op: pg_notify(2.410(12) epoch 11708) -5875> 2014-09-18 18:08:29.729141 7f4c9f251700 5 -- op tracker -- , seq: 85886, time: 2014-09-18 18:08:29.729120, event: dispatched, op: pg_notify(2.410(12) epoch 11708) -5874> 2014-09-18 18:08:29.729145 7f4c9f251700 5 -- op tracker -- , seq: 85886, time: 2014-09-18 18:08:29.729144, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11708) -5873> 2014-09-18 18:08:29.729153 7f4c9f251700 5 -- op tracker -- , seq: 85886, time: 2014-09-18 18:08:29.729151, event: started, op: pg_notify(2.410(12) epoch 11708) -5872> 2014-09-18 18:08:29.729169 7f4c9f251700 5 -- op tracker -- , seq: 85886, time: 2014-09-18 18:08:29.729167, event: done, op: pg_notify(2.410(12) epoch 11708) -5871> 2014-09-18 18:08:29.729182 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.005656 3 0.000263 -5870> 2014-09-18 18:08:29.729204 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -5869> 2014-09-18 18:08:29.729282 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000079 0 0.000000 -5868> 2014-09-18 18:08:29.729342 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -5867> 2014-09-18 18:08:29.729387 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000045 0 0.000000 -5866> 2014-09-18 18:08:29.729398 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -5865> 2014-09-18 18:08:29.729419 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 106 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x24d1a900 con 0x1395edc0 -5864> 2014-09-18 18:08:29.729435 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5863> 2014-09-18 18:08:29.729591 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 107 ==== pg_notify(2.6f8(141) epoch 11708) v5 ==== 8801+0+0 (2372154062 0 0) 0x2b940940 con 0x1395edc0 -5862> 2014-09-18 18:08:29.729607 7f4c9f251700 5 -- op tracker -- , seq: 85887, time: 2014-09-18 18:08:29.729480, event: header_read, op: pg_notify(2.6f8(141) epoch 11708) -5861> 2014-09-18 18:08:29.729617 7f4c9f251700 5 -- op tracker -- , seq: 85887, time: 2014-09-18 18:08:29.729481, event: throttled, op: pg_notify(2.6f8(141) epoch 11708) -5860> 2014-09-18 18:08:29.729623 7f4c9f251700 5 -- op tracker -- , seq: 85887, time: 2014-09-18 18:08:29.729575, event: all_read, op: pg_notify(2.6f8(141) epoch 11708) -5859> 2014-09-18 18:08:29.729629 7f4c9f251700 5 -- op tracker -- , seq: 85887, time: 2014-09-18 18:08:29.729603, event: dispatched, op: pg_notify(2.6f8(141) epoch 11708) -5858> 2014-09-18 18:08:29.729635 7f4c9f251700 5 -- op tracker -- , seq: 85887, time: 2014-09-18 18:08:29.729634, event: waiting_for_osdmap, op: pg_notify(2.6f8(141) epoch 11708) -5857> 2014-09-18 18:08:29.729644 7f4c9f251700 5 -- op tracker -- , seq: 85887, time: 2014-09-18 18:08:29.729642, event: started, op: pg_notify(2.6f8(141) epoch 11708) -5856> 2014-09-18 18:08:29.729661 7f4c9f251700 5 -- op tracker -- , seq: 85887, time: 2014-09-18 18:08:29.729660, event: done, op: pg_notify(2.6f8(141) epoch 11708) -5855> 2014-09-18 18:08:29.729674 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.009530 3 0.000247 -5854> 2014-09-18 18:08:29.729701 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -5853> 2014-09-18 18:08:29.729730 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000029 0 0.000000 -5852> 2014-09-18 18:08:29.729777 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -5851> 2014-09-18 18:08:29.729822 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3293 ==== pg_notify(2.1bb(9) epoch 11708) v5 ==== 1189+0+0 (1072905183 0 0) 0xf1e6d20 con 0x4900580 -5850> 2014-09-18 18:08:29.729841 7f4c9f251700 5 -- op tracker -- , seq: 85888, time: 2014-09-18 18:08:29.729778, event: header_read, op: pg_notify(2.1bb(9) epoch 11708) -5849> 2014-09-18 18:08:29.729846 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000068 0 0.000000 -5848> 2014-09-18 18:08:29.729854 7f4c9f251700 5 -- op tracker -- , seq: 85888, time: 2014-09-18 18:08:29.729779, event: throttled, op: pg_notify(2.1bb(9) epoch 11708) -5847> 2014-09-18 18:08:29.729856 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -5846> 2014-09-18 18:08:29.729861 7f4c9f251700 5 -- op tracker -- , seq: 85888, time: 2014-09-18 18:08:29.729809, event: all_read, op: pg_notify(2.1bb(9) epoch 11708) -5845> 2014-09-18 18:08:29.729875 7f4c9f251700 5 -- op tracker -- , seq: 85888, time: 2014-09-18 18:08:29.729836, event: dispatched, op: pg_notify(2.1bb(9) epoch 11708) -5844> 2014-09-18 18:08:29.729882 7f4c9f251700 5 -- op tracker -- , seq: 85888, time: 2014-09-18 18:08:29.729881, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11708) -5843> 2014-09-18 18:08:29.729890 7f4c9f251700 5 -- op tracker -- , seq: 85888, time: 2014-09-18 18:08:29.729888, event: started, op: pg_notify(2.1bb(9) epoch 11708) -5842> 2014-09-18 18:08:29.729907 7f4c9f251700 5 -- op tracker -- , seq: 85888, time: 2014-09-18 18:08:29.729906, event: done, op: pg_notify(2.1bb(9) epoch 11708) -5841> 2014-09-18 18:08:29.730105 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 530 ==== pg_notify(2.4d0(10) epoch 11708) v5 ==== 1292+0+0 (1661662371 0 0) 0x1247c780 con 0x1395ef20 -5840> 2014-09-18 18:08:29.730116 7f4c9f251700 5 -- op tracker -- , seq: 85889, time: 2014-09-18 18:08:29.730065, event: header_read, op: pg_notify(2.4d0(10) epoch 11708) -5839> 2014-09-18 18:08:29.730123 7f4c9f251700 5 -- op tracker -- , seq: 85889, time: 2014-09-18 18:08:29.730066, event: throttled, op: pg_notify(2.4d0(10) epoch 11708) -5838> 2014-09-18 18:08:29.730129 7f4c9f251700 5 -- op tracker -- , seq: 85889, time: 2014-09-18 18:08:29.730094, event: all_read, op: pg_notify(2.4d0(10) epoch 11708) -5837> 2014-09-18 18:08:29.730135 7f4c9f251700 5 -- op tracker -- , seq: 85889, time: 2014-09-18 18:08:29.730114, event: dispatched, op: pg_notify(2.4d0(10) epoch 11708) -5836> 2014-09-18 18:08:29.730141 7f4c9f251700 5 -- op tracker -- , seq: 85889, time: 2014-09-18 18:08:29.730140, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11708) -5835> 2014-09-18 18:08:29.730149 7f4c9f251700 5 -- op tracker -- , seq: 85889, time: 2014-09-18 18:08:29.730147, event: started, op: pg_notify(2.4d0(10) epoch 11708) -5834> 2014-09-18 18:08:29.730164 7f4c9f251700 5 -- op tracker -- , seq: 85889, time: 2014-09-18 18:08:29.730163, event: done, op: pg_notify(2.4d0(10) epoch 11708) -5833> 2014-09-18 18:08:29.730463 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3636 ==== pg_notify(2.517(11) epoch 11708) v5 ==== 1343+0+0 (3679590507 0 0) 0x12139fe0 con 0xcbecec0 -5832> 2014-09-18 18:08:29.730491 7f4c9f251700 5 -- op tracker -- , seq: 85890, time: 2014-09-18 18:08:29.730421, event: header_read, op: pg_notify(2.517(11) epoch 11708) -5831> 2014-09-18 18:08:29.730499 7f4c9f251700 5 -- op tracker -- , seq: 85890, time: 2014-09-18 18:08:29.730423, event: throttled, op: pg_notify(2.517(11) epoch 11708) -5830> 2014-09-18 18:08:29.730504 7f4c9f251700 5 -- op tracker -- , seq: 85890, time: 2014-09-18 18:08:29.730452, event: all_read, op: pg_notify(2.517(11) epoch 11708) -5829> 2014-09-18 18:08:29.730510 7f4c9f251700 5 -- op tracker -- , seq: 85890, time: 2014-09-18 18:08:29.730488, event: dispatched, op: pg_notify(2.517(11) epoch 11708) -5828> 2014-09-18 18:08:29.730517 7f4c9f251700 5 -- op tracker -- , seq: 85890, time: 2014-09-18 18:08:29.730515, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11708) -5827> 2014-09-18 18:08:29.730524 7f4c9f251700 5 -- op tracker -- , seq: 85890, time: 2014-09-18 18:08:29.730522, event: started, op: pg_notify(2.517(11) epoch 11708) -5826> 2014-09-18 18:08:29.730625 7f4c9f251700 5 -- op tracker -- , seq: 85890, time: 2014-09-18 18:08:29.730622, event: done, op: pg_notify(2.517(11) epoch 11708) -5825> 2014-09-18 18:08:29.731930 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 108 ==== pg_notify(2.6f8(141) epoch 11708) v5 ==== 8801+0+0 (2372154062 0 0) 0x21cfa1e0 con 0x1395edc0 -5824> 2014-09-18 18:08:29.731946 7f4c9f251700 5 -- op tracker -- , seq: 85891, time: 2014-09-18 18:08:29.731817, event: header_read, op: pg_notify(2.6f8(141) epoch 11708) -5823> 2014-09-18 18:08:29.731955 7f4c9f251700 5 -- op tracker -- , seq: 85891, time: 2014-09-18 18:08:29.731818, event: throttled, op: pg_notify(2.6f8(141) epoch 11708) -5822> 2014-09-18 18:08:29.731961 7f4c9f251700 5 -- op tracker -- , seq: 85891, time: 2014-09-18 18:08:29.731918, event: all_read, op: pg_notify(2.6f8(141) epoch 11708) -5821> 2014-09-18 18:08:29.731967 7f4c9f251700 5 -- op tracker -- , seq: 85891, time: 2014-09-18 18:08:29.731942, event: dispatched, op: pg_notify(2.6f8(141) epoch 11708) -5820> 2014-09-18 18:08:29.731974 7f4c9f251700 5 -- op tracker -- , seq: 85891, time: 2014-09-18 18:08:29.731972, event: waiting_for_osdmap, op: pg_notify(2.6f8(141) epoch 11708) -5819> 2014-09-18 18:08:29.731981 7f4c9f251700 5 -- op tracker -- , seq: 85891, time: 2014-09-18 18:08:29.731980, event: started, op: pg_notify(2.6f8(141) epoch 11708) -5818> 2014-09-18 18:08:29.732050 7f4c9f251700 5 -- op tracker -- , seq: 85891, time: 2014-09-18 18:08:29.732048, event: done, op: pg_notify(2.6f8(141) epoch 11708) -5817> 2014-09-18 18:08:29.732295 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 531 ==== pg_notify(2.178(3) epoch 11708) v5 ==== 907+0+0 (855225080 0 0) 0xe7961c0 con 0x1395ef20 -5816> 2014-09-18 18:08:29.732309 7f4c9f251700 5 -- op tracker -- , seq: 85892, time: 2014-09-18 18:08:29.732259, event: header_read, op: pg_notify(2.178(3) epoch 11708) -5815> 2014-09-18 18:08:29.732317 7f4c9f251700 5 -- op tracker -- , seq: 85892, time: 2014-09-18 18:08:29.732260, event: throttled, op: pg_notify(2.178(3) epoch 11708) -5814> 2014-09-18 18:08:29.732323 7f4c9f251700 5 -- op tracker -- , seq: 85892, time: 2014-09-18 18:08:29.732284, event: all_read, op: pg_notify(2.178(3) epoch 11708) -5813> 2014-09-18 18:08:29.732329 7f4c9f251700 5 -- op tracker -- , seq: 85892, time: 2014-09-18 18:08:29.732306, event: dispatched, op: pg_notify(2.178(3) epoch 11708) -5812> 2014-09-18 18:08:29.732335 7f4c9f251700 5 -- op tracker -- , seq: 85892, time: 2014-09-18 18:08:29.732333, event: waiting_for_osdmap, op: pg_notify(2.178(3) epoch 11708) -5811> 2014-09-18 18:08:29.732342 7f4c9f251700 5 -- op tracker -- , seq: 85892, time: 2014-09-18 18:08:29.732341, event: started, op: pg_notify(2.178(3) epoch 11708) -5810> 2014-09-18 18:08:29.732358 7f4c9f251700 5 -- op tracker -- , seq: 85892, time: 2014-09-18 18:08:29.732357, event: done, op: pg_notify(2.178(3) epoch 11708) -5809> 2014-09-18 18:08:29.732596 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 1 ==== pg_notify(2.120(15) epoch 11708) v5 ==== 1535+0+0 (1631416788 0 0) 0x12139a40 con 0x12b3aec0 -5808> 2014-09-18 18:08:29.732612 7f4c9f251700 5 -- op tracker -- , seq: 85893, time: 2014-09-18 18:08:29.732549, event: header_read, op: pg_notify(2.120(15) epoch 11708) -5807> 2014-09-18 18:08:29.732620 7f4c9f251700 5 -- op tracker -- , seq: 85893, time: 2014-09-18 18:08:29.732549, event: throttled, op: pg_notify(2.120(15) epoch 11708) -5806> 2014-09-18 18:08:29.732627 7f4c9f251700 5 -- op tracker -- , seq: 85893, time: 2014-09-18 18:08:29.732582, event: all_read, op: pg_notify(2.120(15) epoch 11708) -5805> 2014-09-18 18:08:29.732633 7f4c9f251700 5 -- op tracker -- , seq: 85893, time: 2014-09-18 18:08:29.732608, event: dispatched, op: pg_notify(2.120(15) epoch 11708) -5804> 2014-09-18 18:08:29.732639 7f4c9f251700 5 -- op tracker -- , seq: 85893, time: 2014-09-18 18:08:29.732637, event: waiting_for_osdmap, op: pg_notify(2.120(15) epoch 11708) -5803> 2014-09-18 18:08:29.732647 7f4c9f251700 5 -- op tracker -- , seq: 85893, time: 2014-09-18 18:08:29.732645, event: started, op: pg_notify(2.120(15) epoch 11708) -5802> 2014-09-18 18:08:29.732664 7f4c9f251700 5 -- op tracker -- , seq: 85893, time: 2014-09-18 18:08:29.732662, event: done, op: pg_notify(2.120(15) epoch 11708) -5801> 2014-09-18 18:08:29.732985 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3637 ==== pg_notify(2.2c7(139) epoch 11708) v5 ==== 8651+0+0 (151129840 0 0) 0xe1acd00 con 0xcbecec0 -5800> 2014-09-18 18:08:29.733000 7f4c9f251700 5 -- op tracker -- , seq: 85894, time: 2014-09-18 18:08:29.732794, event: header_read, op: pg_notify(2.2c7(139) epoch 11708) -5799> 2014-09-18 18:08:29.733009 7f4c9f251700 5 -- op tracker -- , seq: 85894, time: 2014-09-18 18:08:29.732795, event: throttled, op: pg_notify(2.2c7(139) epoch 11708) -5798> 2014-09-18 18:08:29.733015 7f4c9f251700 5 -- op tracker -- , seq: 85894, time: 2014-09-18 18:08:29.732970, event: all_read, op: pg_notify(2.2c7(139) epoch 11708) -5797> 2014-09-18 18:08:29.733021 7f4c9f251700 5 -- op tracker -- , seq: 85894, time: 2014-09-18 18:08:29.732996, event: dispatched, op: pg_notify(2.2c7(139) epoch 11708) -5796> 2014-09-18 18:08:29.733029 7f4c9f251700 5 -- op tracker -- , seq: 85894, time: 2014-09-18 18:08:29.733027, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11708) -5795> 2014-09-18 18:08:29.733037 7f4c9f251700 5 -- op tracker -- , seq: 85894, time: 2014-09-18 18:08:29.733035, event: started, op: pg_notify(2.2c7(139) epoch 11708) -5794> 2014-09-18 18:08:29.733112 7f4c9f251700 5 -- op tracker -- , seq: 85894, time: 2014-09-18 18:08:29.733110, event: done, op: pg_notify(2.2c7(139) epoch 11708) -5793> 2014-09-18 18:08:29.734422 7f4caae41700 5 osd.42 11708 tick -5792> 2014-09-18 18:08:29.734615 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 41 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x24d1a480 con 0x1395f340 -5791> 2014-09-18 18:08:29.734630 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5790> 2014-09-18 18:08:29.734668 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 42 ==== pg_notify(2.1d8(9) epoch 11708) v5 ==== 1261+0+0 (2077326071 0 0) 0xf1ec3a0 con 0x1395f340 -5789> 2014-09-18 18:08:29.734678 7f4c9f251700 5 -- op tracker -- , seq: 85895, time: 2014-09-18 18:08:29.734625, event: header_read, op: pg_notify(2.1d8(9) epoch 11708) -5788> 2014-09-18 18:08:29.734686 7f4c9f251700 5 -- op tracker -- , seq: 85895, time: 2014-09-18 18:08:29.734626, event: throttled, op: pg_notify(2.1d8(9) epoch 11708) -5787> 2014-09-18 18:08:29.734693 7f4c9f251700 5 -- op tracker -- , seq: 85895, time: 2014-09-18 18:08:29.734658, event: all_read, op: pg_notify(2.1d8(9) epoch 11708) -5786> 2014-09-18 18:08:29.734699 7f4c9f251700 5 -- op tracker -- , seq: 85895, time: 2014-09-18 18:08:29.734675, event: dispatched, op: pg_notify(2.1d8(9) epoch 11708) -5785> 2014-09-18 18:08:29.734705 7f4c9f251700 5 -- op tracker -- , seq: 85895, time: 2014-09-18 18:08:29.734704, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11708) -5784> 2014-09-18 18:08:29.734713 7f4c9f251700 5 -- op tracker -- , seq: 85895, time: 2014-09-18 18:08:29.734711, event: started, op: pg_notify(2.1d8(9) epoch 11708) -5783> 2014-09-18 18:08:29.734730 7f4c9f251700 5 -- op tracker -- , seq: 85895, time: 2014-09-18 18:08:29.734729, event: done, op: pg_notify(2.1d8(9) epoch 11708) -5782> 2014-09-18 18:08:29.737481 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 926 ==== pg_notify(2.11b(27) epoch 11708) v5 ==== 2247+0+0 (3185279835 0 0) 0x2b987860 con 0x1395f8c0 -5781> 2014-09-18 18:08:29.737503 7f4c9f251700 5 -- op tracker -- , seq: 85896, time: 2014-09-18 18:08:29.737416, event: header_read, op: pg_notify(2.11b(27) epoch 11708) -5780> 2014-09-18 18:08:29.737512 7f4c9f251700 5 -- op tracker -- , seq: 85896, time: 2014-09-18 18:08:29.737417, event: throttled, op: pg_notify(2.11b(27) epoch 11708) -5779> 2014-09-18 18:08:29.737518 7f4c9f251700 5 -- op tracker -- , seq: 85896, time: 2014-09-18 18:08:29.737466, event: all_read, op: pg_notify(2.11b(27) epoch 11708) -5778> 2014-09-18 18:08:29.737524 7f4c9f251700 5 -- op tracker -- , seq: 85896, time: 2014-09-18 18:08:29.737498, event: dispatched, op: pg_notify(2.11b(27) epoch 11708) -5777> 2014-09-18 18:08:29.737538 7f4c9f251700 5 -- op tracker -- , seq: 85896, time: 2014-09-18 18:08:29.737536, event: waiting_for_osdmap, op: pg_notify(2.11b(27) epoch 11708) -5776> 2014-09-18 18:08:29.737546 7f4c9f251700 5 -- op tracker -- , seq: 85896, time: 2014-09-18 18:08:29.737544, event: started, op: pg_notify(2.11b(27) epoch 11708) -5775> 2014-09-18 18:08:29.737564 7f4c9f251700 5 -- op tracker -- , seq: 85896, time: 2014-09-18 18:08:29.737562, event: done, op: pg_notify(2.11b(27) epoch 11708) -5774> 2014-09-18 18:08:29.739969 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 927 ==== pg_notify(2.11b(27) epoch 11708) v5 ==== 2247+0+0 (3185279835 0 0) 0x1393d2a0 con 0x1395f8c0 -5773> 2014-09-18 18:08:29.739993 7f4c9f251700 5 -- op tracker -- , seq: 85897, time: 2014-09-18 18:08:29.739877, event: header_read, op: pg_notify(2.11b(27) epoch 11708) -5772> 2014-09-18 18:08:29.740002 7f4c9f251700 5 -- op tracker -- , seq: 85897, time: 2014-09-18 18:08:29.739877, event: throttled, op: pg_notify(2.11b(27) epoch 11708) -5771> 2014-09-18 18:08:29.740006 7f4c9f251700 5 -- op tracker -- , seq: 85897, time: 2014-09-18 18:08:29.739956, event: all_read, op: pg_notify(2.11b(27) epoch 11708) -5770> 2014-09-18 18:08:29.740012 7f4c9f251700 5 -- op tracker -- , seq: 85897, time: 2014-09-18 18:08:29.739990, event: dispatched, op: pg_notify(2.11b(27) epoch 11708) -5769> 2014-09-18 18:08:29.740021 7f4c9f251700 5 -- op tracker -- , seq: 85897, time: 2014-09-18 18:08:29.740020, event: waiting_for_osdmap, op: pg_notify(2.11b(27) epoch 11708) -5768> 2014-09-18 18:08:29.740029 7f4c9f251700 5 -- op tracker -- , seq: 85897, time: 2014-09-18 18:08:29.740028, event: started, op: pg_notify(2.11b(27) epoch 11708) -5767> 2014-09-18 18:08:29.740052 7f4c9f251700 5 -- op tracker -- , seq: 85897, time: 2014-09-18 18:08:29.740050, event: done, op: pg_notify(2.11b(27) epoch 11708) -5766> 2014-09-18 18:08:29.740592 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 928 ==== pg_notify(2.410(12) epoch 11708) v5 ==== 1338+0+0 (1346775082 0 0) 0xe9574a0 con 0x1395f8c0 -5765> 2014-09-18 18:08:29.740603 7f4c9f251700 5 -- op tracker -- , seq: 85898, time: 2014-09-18 18:08:29.740536, event: header_read, op: pg_notify(2.410(12) epoch 11708) -5764> 2014-09-18 18:08:29.740611 7f4c9f251700 5 -- op tracker -- , seq: 85898, time: 2014-09-18 18:08:29.740537, event: throttled, op: pg_notify(2.410(12) epoch 11708) -5763> 2014-09-18 18:08:29.740622 7f4c9f251700 5 -- op tracker -- , seq: 85898, time: 2014-09-18 18:08:29.740579, event: all_read, op: pg_notify(2.410(12) epoch 11708) -5762> 2014-09-18 18:08:29.740629 7f4c9f251700 5 -- op tracker -- , seq: 85898, time: 2014-09-18 18:08:29.740600, event: dispatched, op: pg_notify(2.410(12) epoch 11708) -5761> 2014-09-18 18:08:29.740636 7f4c9f251700 5 -- op tracker -- , seq: 85898, time: 2014-09-18 18:08:29.740634, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11708) -5760> 2014-09-18 18:08:29.740643 7f4c9f251700 5 -- op tracker -- , seq: 85898, time: 2014-09-18 18:08:29.740641, event: started, op: pg_notify(2.410(12) epoch 11708) -5759> 2014-09-18 18:08:29.740659 7f4c9f251700 5 -- op tracker -- , seq: 85898, time: 2014-09-18 18:08:29.740658, event: done, op: pg_notify(2.410(12) epoch 11708) -5758> 2014-09-18 18:08:29.741080 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 929 ==== pg_notify(2.78f(155) epoch 11708) v5 ==== 9335+0+0 (4194529238 0 0) 0xf4865a0 con 0x1395f8c0 -5757> 2014-09-18 18:08:29.741094 7f4c9f251700 5 -- op tracker -- , seq: 85899, time: 2014-09-18 18:08:29.740947, event: header_read, op: pg_notify(2.78f(155) epoch 11708) -5756> 2014-09-18 18:08:29.741103 7f4c9f251700 5 -- op tracker -- , seq: 85899, time: 2014-09-18 18:08:29.740948, event: throttled, op: pg_notify(2.78f(155) epoch 11708) -5755> 2014-09-18 18:08:29.741109 7f4c9f251700 5 -- op tracker -- , seq: 85899, time: 2014-09-18 18:08:29.741069, event: all_read, op: pg_notify(2.78f(155) epoch 11708) -5754> 2014-09-18 18:08:29.741122 7f4c9f251700 5 -- op tracker -- , seq: 85899, time: 2014-09-18 18:08:29.741091, event: dispatched, op: pg_notify(2.78f(155) epoch 11708) -5753> 2014-09-18 18:08:29.741129 7f4c9f251700 5 -- op tracker -- , seq: 85899, time: 2014-09-18 18:08:29.741127, event: waiting_for_osdmap, op: pg_notify(2.78f(155) epoch 11708) -5752> 2014-09-18 18:08:29.741136 7f4c9f251700 5 -- op tracker -- , seq: 85899, time: 2014-09-18 18:08:29.741135, event: started, op: pg_notify(2.78f(155) epoch 11708) -5751> 2014-09-18 18:08:29.741153 7f4c9f251700 5 -- op tracker -- , seq: 85899, time: 2014-09-18 18:08:29.741152, event: done, op: pg_notify(2.78f(155) epoch 11708) -5750> 2014-09-18 18:08:29.749013 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 115 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x10d18640 con 0x1395f080 -5749> 2014-09-18 18:08:29.749029 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5748> 2014-09-18 18:08:29.749114 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 116 ==== pg_notify(2.179(81) epoch 11708) v5 ==== 5289+0+0 (3791867560 0 0) 0xec6b660 con 0x1395f080 -5747> 2014-09-18 18:08:29.749125 7f4c9f251700 5 -- op tracker -- , seq: 85900, time: 2014-09-18 18:08:29.749003, event: header_read, op: pg_notify(2.179(81) epoch 11708) -5746> 2014-09-18 18:08:29.749288 7f4c9f251700 5 -- op tracker -- , seq: 85900, time: 2014-09-18 18:08:29.749003, event: throttled, op: pg_notify(2.179(81) epoch 11708) -5745> 2014-09-18 18:08:29.749297 7f4c9f251700 5 -- op tracker -- , seq: 85900, time: 2014-09-18 18:08:29.749103, event: all_read, op: pg_notify(2.179(81) epoch 11708) -5744> 2014-09-18 18:08:29.749305 7f4c9f251700 5 -- op tracker -- , seq: 85900, time: 2014-09-18 18:08:29.749122, event: dispatched, op: pg_notify(2.179(81) epoch 11708) -5743> 2014-09-18 18:08:29.749311 7f4c9f251700 5 -- op tracker -- , seq: 85900, time: 2014-09-18 18:08:29.749310, event: waiting_for_osdmap, op: pg_notify(2.179(81) epoch 11708) -5742> 2014-09-18 18:08:29.749319 7f4c9f251700 5 -- op tracker -- , seq: 85900, time: 2014-09-18 18:08:29.749318, event: started, op: pg_notify(2.179(81) epoch 11708) -5741> 2014-09-18 18:08:29.749337 7f4c9f251700 5 -- op tracker -- , seq: 85900, time: 2014-09-18 18:08:29.749336, event: done, op: pg_notify(2.179(81) epoch 11708) -5740> 2014-09-18 18:08:29.749357 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.031556 3 0.000148 -5739> 2014-09-18 18:08:29.749406 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -5738> 2014-09-18 18:08:29.749448 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- pg_query(2.179 epoch 11708) v3 -- ?+0 0xf195a20 con 0x1395f080 -5737> 2014-09-18 18:08:29.750852 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 2 ==== pg_notify(2.40a(162) epoch 11708) v5 ==== 9736+0+0 (2980019402 0 0) 0xf4865a0 con 0x12b3aec0 -5736> 2014-09-18 18:08:29.750866 7f4c9f251700 5 -- op tracker -- , seq: 85901, time: 2014-09-18 18:08:29.750723, event: header_read, op: pg_notify(2.40a(162) epoch 11708) -5735> 2014-09-18 18:08:29.750874 7f4c9f251700 5 -- op tracker -- , seq: 85901, time: 2014-09-18 18:08:29.750724, event: throttled, op: pg_notify(2.40a(162) epoch 11708) -5734> 2014-09-18 18:08:29.750880 7f4c9f251700 5 -- op tracker -- , seq: 85901, time: 2014-09-18 18:08:29.750838, event: all_read, op: pg_notify(2.40a(162) epoch 11708) -5733> 2014-09-18 18:08:29.750887 7f4c9f251700 5 -- op tracker -- , seq: 85901, time: 2014-09-18 18:08:29.750863, event: dispatched, op: pg_notify(2.40a(162) epoch 11708) -5732> 2014-09-18 18:08:29.750893 7f4c9f251700 5 -- op tracker -- , seq: 85901, time: 2014-09-18 18:08:29.750891, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11708) -5731> 2014-09-18 18:08:29.750901 7f4c9f251700 5 -- op tracker -- , seq: 85901, time: 2014-09-18 18:08:29.750899, event: started, op: pg_notify(2.40a(162) epoch 11708) -5730> 2014-09-18 18:08:29.750919 7f4c9f251700 5 -- op tracker -- , seq: 85901, time: 2014-09-18 18:08:29.750917, event: done, op: pg_notify(2.40a(162) epoch 11708) -5729> 2014-09-18 18:08:29.751971 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 117 ==== pg_notify(2.519(4) epoch 11708) v5 ==== 962+0+0 (3723035416 0 0) 0xf195a20 con 0x1395f080 -5728> 2014-09-18 18:08:29.751983 7f4c9f251700 5 -- op tracker -- , seq: 85902, time: 2014-09-18 18:08:29.751920, event: header_read, op: pg_notify(2.519(4) epoch 11708) -5727> 2014-09-18 18:08:29.751991 7f4c9f251700 5 -- op tracker -- , seq: 85902, time: 2014-09-18 18:08:29.751921, event: throttled, op: pg_notify(2.519(4) epoch 11708) -5726> 2014-09-18 18:08:29.751997 7f4c9f251700 5 -- op tracker -- , seq: 85902, time: 2014-09-18 18:08:29.751958, event: all_read, op: pg_notify(2.519(4) epoch 11708) -5725> 2014-09-18 18:08:29.752003 7f4c9f251700 5 -- op tracker -- , seq: 85902, time: 2014-09-18 18:08:29.751980, event: dispatched, op: pg_notify(2.519(4) epoch 11708) -5724> 2014-09-18 18:08:29.752009 7f4c9f251700 5 -- op tracker -- , seq: 85902, time: 2014-09-18 18:08:29.752007, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11708) -5723> 2014-09-18 18:08:29.752016 7f4c9f251700 5 -- op tracker -- , seq: 85902, time: 2014-09-18 18:08:29.752014, event: started, op: pg_notify(2.519(4) epoch 11708) -5722> 2014-09-18 18:08:29.752032 7f4c9f251700 5 -- op tracker -- , seq: 85902, time: 2014-09-18 18:08:29.752031, event: done, op: pg_notify(2.519(4) epoch 11708) -5721> 2014-09-18 18:08:29.753757 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 118 ==== pg_notify(2.4e8(37) epoch 11708) v5 ==== 2849+0+0 (2544103907 0 0) 0x2b987e00 con 0x1395f080 -5720> 2014-09-18 18:08:29.753771 7f4c9f251700 5 -- op tracker -- , seq: 85903, time: 2014-09-18 18:08:29.753686, event: header_read, op: pg_notify(2.4e8(37) epoch 11708) -5719> 2014-09-18 18:08:29.753779 7f4c9f251700 5 -- op tracker -- , seq: 85903, time: 2014-09-18 18:08:29.753687, event: throttled, op: pg_notify(2.4e8(37) epoch 11708) -5718> 2014-09-18 18:08:29.753784 7f4c9f251700 5 -- op tracker -- , seq: 85903, time: 2014-09-18 18:08:29.753744, event: all_read, op: pg_notify(2.4e8(37) epoch 11708) -5717> 2014-09-18 18:08:29.753790 7f4c9f251700 5 -- op tracker -- , seq: 85903, time: 2014-09-18 18:08:29.753767, event: dispatched, op: pg_notify(2.4e8(37) epoch 11708) -5716> 2014-09-18 18:08:29.753796 7f4c9f251700 5 -- op tracker -- , seq: 85903, time: 2014-09-18 18:08:29.753795, event: waiting_for_osdmap, op: pg_notify(2.4e8(37) epoch 11708) -5715> 2014-09-18 18:08:29.753804 7f4c9f251700 5 -- op tracker -- , seq: 85903, time: 2014-09-18 18:08:29.753802, event: started, op: pg_notify(2.4e8(37) epoch 11708) -5714> 2014-09-18 18:08:29.753820 7f4c9f251700 5 -- op tracker -- , seq: 85903, time: 2014-09-18 18:08:29.753818, event: done, op: pg_notify(2.4e8(37) epoch 11708) -5713> 2014-09-18 18:08:29.754361 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 119 ==== pg_notify(2.519(4) epoch 11708) v5 ==== 962+0+0 (3723035416 0 0) 0x1247d2c0 con 0x1395f080 -5712> 2014-09-18 18:08:29.754374 7f4c9f251700 5 -- op tracker -- , seq: 85904, time: 2014-09-18 18:08:29.754312, event: header_read, op: pg_notify(2.519(4) epoch 11708) -5711> 2014-09-18 18:08:29.754383 7f4c9f251700 5 -- op tracker -- , seq: 85904, time: 2014-09-18 18:08:29.754313, event: throttled, op: pg_notify(2.519(4) epoch 11708) -5710> 2014-09-18 18:08:29.754392 7f4c9f251700 5 -- op tracker -- , seq: 85904, time: 2014-09-18 18:08:29.754348, event: all_read, op: pg_notify(2.519(4) epoch 11708) -5709> 2014-09-18 18:08:29.754402 7f4c9f251700 5 -- op tracker -- , seq: 85904, time: 2014-09-18 18:08:29.754371, event: dispatched, op: pg_notify(2.519(4) epoch 11708) -5708> 2014-09-18 18:08:29.754410 7f4c9f251700 5 -- op tracker -- , seq: 85904, time: 2014-09-18 18:08:29.754409, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11708) -5707> 2014-09-18 18:08:29.754417 7f4c9f251700 5 -- op tracker -- , seq: 85904, time: 2014-09-18 18:08:29.754416, event: started, op: pg_notify(2.519(4) epoch 11708) -5706> 2014-09-18 18:08:29.754456 7f4c9f251700 5 -- op tracker -- , seq: 85904, time: 2014-09-18 18:08:29.754454, event: done, op: pg_notify(2.519(4) epoch 11708) -5705> 2014-09-18 18:08:29.754687 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 120 ==== pg_notify(2.179(81) epoch 11708) v5 ==== 5289+0+0 (3791867560 0 0) 0xf4ef0e0 con 0x1395f080 -5704> 2014-09-18 18:08:29.754700 7f4c9f251700 5 -- op tracker -- , seq: 85905, time: 2014-09-18 18:08:29.754580, event: header_read, op: pg_notify(2.179(81) epoch 11708) -5703> 2014-09-18 18:08:29.754708 7f4c9f251700 5 -- op tracker -- , seq: 85905, time: 2014-09-18 18:08:29.754580, event: throttled, op: pg_notify(2.179(81) epoch 11708) -5702> 2014-09-18 18:08:29.754714 7f4c9f251700 5 -- op tracker -- , seq: 85905, time: 2014-09-18 18:08:29.754674, event: all_read, op: pg_notify(2.179(81) epoch 11708) -5701> 2014-09-18 18:08:29.754720 7f4c9f251700 5 -- op tracker -- , seq: 85905, time: 2014-09-18 18:08:29.754697, event: dispatched, op: pg_notify(2.179(81) epoch 11708) -5700> 2014-09-18 18:08:29.754726 7f4c9f251700 5 -- op tracker -- , seq: 85905, time: 2014-09-18 18:08:29.754725, event: waiting_for_osdmap, op: pg_notify(2.179(81) epoch 11708) -5699> 2014-09-18 18:08:29.754734 7f4c9f251700 5 -- op tracker -- , seq: 85905, time: 2014-09-18 18:08:29.754732, event: started, op: pg_notify(2.179(81) epoch 11708) -5698> 2014-09-18 18:08:29.754749 7f4c9f251700 5 -- op tracker -- , seq: 85905, time: 2014-09-18 18:08:29.754748, event: done, op: pg_notify(2.179(81) epoch 11708) -5697> 2014-09-18 18:08:29.754959 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 121 ==== pg_notify(2.4e8(37) epoch 11708) v5 ==== 2849+0+0 (2544103907 0 0) 0x2b928760 con 0x1395f080 -5696> 2014-09-18 18:08:29.754970 7f4c9f251700 5 -- op tracker -- , seq: 85906, time: 2014-09-18 18:08:29.754908, event: header_read, op: pg_notify(2.4e8(37) epoch 11708) -5695> 2014-09-18 18:08:29.754977 7f4c9f251700 5 -- op tracker -- , seq: 85906, time: 2014-09-18 18:08:29.754909, event: throttled, op: pg_notify(2.4e8(37) epoch 11708) -5694> 2014-09-18 18:08:29.754983 7f4c9f251700 5 -- op tracker -- , seq: 85906, time: 2014-09-18 18:08:29.754948, event: all_read, op: pg_notify(2.4e8(37) epoch 11708) -5693> 2014-09-18 18:08:29.754988 7f4c9f251700 5 -- op tracker -- , seq: 85906, time: 2014-09-18 18:08:29.754967, event: dispatched, op: pg_notify(2.4e8(37) epoch 11708) -5692> 2014-09-18 18:08:29.755002 7f4c9f251700 5 -- op tracker -- , seq: 85906, time: 2014-09-18 18:08:29.755001, event: waiting_for_osdmap, op: pg_notify(2.4e8(37) epoch 11708) -5691> 2014-09-18 18:08:29.755009 7f4c9f251700 5 -- op tracker -- , seq: 85906, time: 2014-09-18 18:08:29.755008, event: started, op: pg_notify(2.4e8(37) epoch 11708) -5690> 2014-09-18 18:08:29.755024 7f4c9f251700 5 -- op tracker -- , seq: 85906, time: 2014-09-18 18:08:29.755023, event: done, op: pg_notify(2.4e8(37) epoch 11708) -5689> 2014-09-18 18:08:29.755622 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 122 ==== pg_log(2.179 epoch 11708 log log((6752'29654,6752'29654], crt=6752'29654) query_epoch 11708) v4 ==== 798+0+0 (3696773611 0 0) 0x10bea000 con 0x1395f080 -5688> 2014-09-18 18:08:29.755634 7f4c9f251700 5 -- op tracker -- , seq: 85907, time: 2014-09-18 18:08:29.755585, event: header_read, op: pg_log(2.179 epoch 11708 log log((6752'29654,6752'29654], crt=6752'29654) query_epoch 11708) -5687> 2014-09-18 18:08:29.755642 7f4c9f251700 5 -- op tracker -- , seq: 85907, time: 2014-09-18 18:08:29.755586, event: throttled, op: pg_log(2.179 epoch 11708 log log((6752'29654,6752'29654], crt=6752'29654) query_epoch 11708) -5686> 2014-09-18 18:08:29.755649 7f4c9f251700 5 -- op tracker -- , seq: 85907, time: 2014-09-18 18:08:29.755611, event: all_read, op: pg_log(2.179 epoch 11708 log log((6752'29654,6752'29654], crt=6752'29654) query_epoch 11708) -5685> 2014-09-18 18:08:29.755655 7f4c9f251700 5 -- op tracker -- , seq: 85907, time: 2014-09-18 18:08:29.755631, event: dispatched, op: pg_log(2.179 epoch 11708 log log((6752'29654,6752'29654], crt=6752'29654) query_epoch 11708) -5684> 2014-09-18 18:08:29.755662 7f4c9f251700 5 -- op tracker -- , seq: 85907, time: 2014-09-18 18:08:29.755660, event: waiting_for_osdmap, op: pg_log(2.179 epoch 11708 log log((6752'29654,6752'29654], crt=6752'29654) query_epoch 11708) -5683> 2014-09-18 18:08:29.755670 7f4c9f251700 5 -- op tracker -- , seq: 85907, time: 2014-09-18 18:08:29.755668, event: started, op: pg_log(2.179 epoch 11708 log log((6752'29654,6752'29654], crt=6752'29654) query_epoch 11708) -5682> 2014-09-18 18:08:29.755684 7f4c9f251700 5 -- op tracker -- , seq: 85907, time: 2014-09-18 18:08:29.755682, event: done, op: pg_log(2.179 epoch 11708 log log((6752'29654,6752'29654], crt=6752'29654) query_epoch 11708) -5681> 2014-09-18 18:08:29.755698 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.006292 2 0.000098 -5680> 2014-09-18 18:08:29.755844 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -5679> 2014-09-18 18:08:29.755866 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000022 0 0.000000 -5678> 2014-09-18 18:08:29.755885 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -5677> 2014-09-18 18:08:29.776311 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 3 ==== pg_notify(2.1a2(20) epoch 11708) v5 ==== 1798+0+0 (2804368626 0 0) 0xf097c20 con 0x12b3aec0 -5676> 2014-09-18 18:08:29.776339 7f4c9f251700 5 -- op tracker -- , seq: 85908, time: 2014-09-18 18:08:29.776236, event: header_read, op: pg_notify(2.1a2(20) epoch 11708) -5675> 2014-09-18 18:08:29.776347 7f4c9f251700 5 -- op tracker -- , seq: 85908, time: 2014-09-18 18:08:29.776237, event: throttled, op: pg_notify(2.1a2(20) epoch 11708) -5674> 2014-09-18 18:08:29.776358 7f4c9f251700 5 -- op tracker -- , seq: 85908, time: 2014-09-18 18:08:29.776297, event: all_read, op: pg_notify(2.1a2(20) epoch 11708) -5673> 2014-09-18 18:08:29.776363 7f4c9f251700 5 -- op tracker -- , seq: 85908, time: 2014-09-18 18:08:29.776336, event: dispatched, op: pg_notify(2.1a2(20) epoch 11708) -5672> 2014-09-18 18:08:29.776372 7f4c9f251700 5 -- op tracker -- , seq: 85908, time: 2014-09-18 18:08:29.776370, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11708) -5671> 2014-09-18 18:08:29.776384 7f4c9f251700 5 -- op tracker -- , seq: 85908, time: 2014-09-18 18:08:29.776382, event: started, op: pg_notify(2.1a2(20) epoch 11708) -5670> 2014-09-18 18:08:29.776405 7f4c9f251700 5 -- op tracker -- , seq: 85908, time: 2014-09-18 18:08:29.776403, event: done, op: pg_notify(2.1a2(20) epoch 11708) -5669> 2014-09-18 18:08:29.780604 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 4 ==== pg_notify(2.78f(155),2.4c6(15) epoch 11708) v5 ==== 10834+0+0 (981379163 0 0) 0x146deb40 con 0x12b3aec0 -5668> 2014-09-18 18:08:29.780633 7f4c9f251700 5 -- op tracker -- , seq: 85909, time: 2014-09-18 18:08:29.780441, event: header_read, op: pg_notify(2.78f(155),2.4c6(15) epoch 11708) -5667> 2014-09-18 18:08:29.780644 7f4c9f251700 5 -- op tracker -- , seq: 85909, time: 2014-09-18 18:08:29.780442, event: throttled, op: pg_notify(2.78f(155),2.4c6(15) epoch 11708) -5666> 2014-09-18 18:08:29.780650 7f4c9f251700 5 -- op tracker -- , seq: 85909, time: 2014-09-18 18:08:29.780590, event: all_read, op: pg_notify(2.78f(155),2.4c6(15) epoch 11708) -5665> 2014-09-18 18:08:29.780657 7f4c9f251700 5 -- op tracker -- , seq: 85909, time: 2014-09-18 18:08:29.780620, event: dispatched, op: pg_notify(2.78f(155),2.4c6(15) epoch 11708) -5664> 2014-09-18 18:08:29.780668 7f4c9f251700 5 -- op tracker -- , seq: 85909, time: 2014-09-18 18:08:29.780667, event: waiting_for_osdmap, op: pg_notify(2.78f(155),2.4c6(15) epoch 11708) -5663> 2014-09-18 18:08:29.780677 7f4c9f251700 5 -- op tracker -- , seq: 85909, time: 2014-09-18 18:08:29.780675, event: started, op: pg_notify(2.78f(155),2.4c6(15) epoch 11708) -5662> 2014-09-18 18:08:29.780711 7f4c9f251700 5 -- op tracker -- , seq: 85909, time: 2014-09-18 18:08:29.780710, event: done, op: pg_notify(2.78f(155),2.4c6(15) epoch 11708) -5661> 2014-09-18 18:08:29.781919 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 5 ==== pg_notify(2.78f(155) epoch 11708) v5 ==== 9335+0+0 (4194529238 0 0) 0xf786d20 con 0x12b3aec0 -5660> 2014-09-18 18:08:29.781935 7f4c9f251700 5 -- op tracker -- , seq: 85910, time: 2014-09-18 18:08:29.781672, event: header_read, op: pg_notify(2.78f(155) epoch 11708) -5659> 2014-09-18 18:08:29.781944 7f4c9f251700 5 -- op tracker -- , seq: 85910, time: 2014-09-18 18:08:29.781673, event: throttled, op: pg_notify(2.78f(155) epoch 11708) -5658> 2014-09-18 18:08:29.781950 7f4c9f251700 5 -- op tracker -- , seq: 85910, time: 2014-09-18 18:08:29.781872, event: all_read, op: pg_notify(2.78f(155) epoch 11708) -5657> 2014-09-18 18:08:29.781956 7f4c9f251700 5 -- op tracker -- , seq: 85910, time: 2014-09-18 18:08:29.781932, event: dispatched, op: pg_notify(2.78f(155) epoch 11708) -5656> 2014-09-18 18:08:29.781962 7f4c9f251700 5 -- op tracker -- , seq: 85910, time: 2014-09-18 18:08:29.781960, event: waiting_for_osdmap, op: pg_notify(2.78f(155) epoch 11708) -5655> 2014-09-18 18:08:29.781969 7f4c9f251700 5 -- op tracker -- , seq: 85910, time: 2014-09-18 18:08:29.781968, event: started, op: pg_notify(2.78f(155) epoch 11708) -5654> 2014-09-18 18:08:29.781992 7f4c9f251700 5 -- op tracker -- , seq: 85910, time: 2014-09-18 18:08:29.781991, event: done, op: pg_notify(2.78f(155) epoch 11708) -5653> 2014-09-18 18:08:29.890188 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3179 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0xe2ef600 con 0x1395ec60 -5652> 2014-09-18 18:08:29.890218 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5651> 2014-09-18 18:08:29.890229 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3180 ==== pg_query(2.13 epoch 11708) v3 ==== 144+0+0 (3676936502 0 0) 0xea00940 con 0x1395ec60 -5650> 2014-09-18 18:08:29.890249 7f4c9f251700 5 -- op tracker -- , seq: 85911, time: 2014-09-18 18:08:29.890190, event: header_read, op: pg_query(2.13 epoch 11708) -5649> 2014-09-18 18:08:29.890261 7f4c9f251700 5 -- op tracker -- , seq: 85911, time: 2014-09-18 18:08:29.890190, event: throttled, op: pg_query(2.13 epoch 11708) -5648> 2014-09-18 18:08:29.890272 7f4c9f251700 5 -- op tracker -- , seq: 85911, time: 2014-09-18 18:08:29.890206, event: all_read, op: pg_query(2.13 epoch 11708) -5647> 2014-09-18 18:08:29.890280 7f4c9f251700 5 -- op tracker -- , seq: 85911, time: 2014-09-18 18:08:29.890236, event: dispatched, op: pg_query(2.13 epoch 11708) -5646> 2014-09-18 18:08:29.890288 7f4c9f251700 5 -- op tracker -- , seq: 85911, time: 2014-09-18 18:08:29.890286, event: waiting_for_osdmap, op: pg_query(2.13 epoch 11708) -5645> 2014-09-18 18:08:29.890296 7f4c9f251700 5 -- op tracker -- , seq: 85911, time: 2014-09-18 18:08:29.890295, event: started, op: pg_query(2.13 epoch 11708) -5644> 2014-09-18 18:08:29.890326 7f4c9f251700 5 -- op tracker -- , seq: 85911, time: 2014-09-18 18:08:29.890325, event: done, op: pg_query(2.13 epoch 11708) -5643> 2014-09-18 18:08:29.890709 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.13(6) epoch 11708) v5 -- ?+0 0xe1abc20 con 0x1395ec60 -5642> 2014-09-18 18:08:29.894301 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3181 ==== pg_notify(2.71(19) epoch 11708) v5 ==== 1795+0+0 (3537286815 0 0) 0xe1abc20 con 0x1395ec60 -5641> 2014-09-18 18:08:29.894321 7f4c9f251700 5 -- op tracker -- , seq: 85912, time: 2014-09-18 18:08:29.894251, event: header_read, op: pg_notify(2.71(19) epoch 11708) -5640> 2014-09-18 18:08:29.894331 7f4c9f251700 5 -- op tracker -- , seq: 85912, time: 2014-09-18 18:08:29.894252, event: throttled, op: pg_notify(2.71(19) epoch 11708) -5639> 2014-09-18 18:08:29.894338 7f4c9f251700 5 -- op tracker -- , seq: 85912, time: 2014-09-18 18:08:29.894288, event: all_read, op: pg_notify(2.71(19) epoch 11708) -5638> 2014-09-18 18:08:29.894344 7f4c9f251700 5 -- op tracker -- , seq: 85912, time: 2014-09-18 18:08:29.894316, event: dispatched, op: pg_notify(2.71(19) epoch 11708) -5637> 2014-09-18 18:08:29.894350 7f4c9f251700 5 -- op tracker -- , seq: 85912, time: 2014-09-18 18:08:29.894349, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11708) -5636> 2014-09-18 18:08:29.894358 7f4c9f251700 5 -- op tracker -- , seq: 85912, time: 2014-09-18 18:08:29.894357, event: started, op: pg_notify(2.71(19) epoch 11708) -5635> 2014-09-18 18:08:29.894379 7f4c9f251700 5 -- op tracker -- , seq: 85912, time: 2014-09-18 18:08:29.894377, event: done, op: pg_notify(2.71(19) epoch 11708) -5634> 2014-09-18 18:08:29.897285 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3182 ==== pg_query(2.67b epoch 11708) v3 ==== 144+0+0 (1804403469 0 0) 0x1393c940 con 0x1395ec60 -5633> 2014-09-18 18:08:29.897300 7f4c9f251700 5 -- op tracker -- , seq: 85913, time: 2014-09-18 18:08:29.897250, event: header_read, op: pg_query(2.67b epoch 11708) -5632> 2014-09-18 18:08:29.897308 7f4c9f251700 5 -- op tracker -- , seq: 85913, time: 2014-09-18 18:08:29.897252, event: throttled, op: pg_query(2.67b epoch 11708) -5631> 2014-09-18 18:08:29.897314 7f4c9f251700 5 -- op tracker -- , seq: 85913, time: 2014-09-18 18:08:29.897273, event: all_read, op: pg_query(2.67b epoch 11708) -5630> 2014-09-18 18:08:29.897320 7f4c9f251700 5 -- op tracker -- , seq: 85913, time: 2014-09-18 18:08:29.897296, event: dispatched, op: pg_query(2.67b epoch 11708) -5629> 2014-09-18 18:08:29.897326 7f4c9f251700 5 -- op tracker -- , seq: 85913, time: 2014-09-18 18:08:29.897325, event: waiting_for_osdmap, op: pg_query(2.67b epoch 11708) -5628> 2014-09-18 18:08:29.897334 7f4c9f251700 5 -- op tracker -- , seq: 85913, time: 2014-09-18 18:08:29.897333, event: started, op: pg_query(2.67b epoch 11708) -5627> 2014-09-18 18:08:29.897349 7f4c9f251700 5 -- op tracker -- , seq: 85913, time: 2014-09-18 18:08:29.897347, event: done, op: pg_query(2.67b epoch 11708) -5626> 2014-09-18 18:08:29.897530 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.67b(4) epoch 11708) v5 -- ?+0 0xe1ab860 con 0x1395ec60 -5625> 2014-09-18 18:08:29.976361 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3183 ==== pg_notify(2.519(4) epoch 11708) v5 ==== 962+0+0 (3723035416 0 0) 0xe1ab860 con 0x1395ec60 -5624> 2014-09-18 18:08:29.976392 7f4c9f251700 5 -- op tracker -- , seq: 85914, time: 2014-09-18 18:08:29.976271, event: header_read, op: pg_notify(2.519(4) epoch 11708) -5623> 2014-09-18 18:08:29.976406 7f4c9f251700 5 -- op tracker -- , seq: 85914, time: 2014-09-18 18:08:29.976274, event: throttled, op: pg_notify(2.519(4) epoch 11708) -5622> 2014-09-18 18:08:29.976414 7f4c9f251700 5 -- op tracker -- , seq: 85914, time: 2014-09-18 18:08:29.976314, event: all_read, op: pg_notify(2.519(4) epoch 11708) -5621> 2014-09-18 18:08:29.976421 7f4c9f251700 5 -- op tracker -- , seq: 85914, time: 2014-09-18 18:08:29.976385, event: dispatched, op: pg_notify(2.519(4) epoch 11708) -5620> 2014-09-18 18:08:29.976435 7f4c9f251700 5 -- op tracker -- , seq: 85914, time: 2014-09-18 18:08:29.976433, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11708) -5619> 2014-09-18 18:08:29.976451 7f4c9f251700 5 -- op tracker -- , seq: 85914, time: 2014-09-18 18:08:29.976448, event: started, op: pg_notify(2.519(4) epoch 11708) -5618> 2014-09-18 18:08:29.976481 7f4c9f251700 5 -- op tracker -- , seq: 85914, time: 2014-09-18 18:08:29.976479, event: done, op: pg_notify(2.519(4) epoch 11708) -5617> 2014-09-18 18:08:29.981344 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3184 ==== pg_notify(2.410(12) epoch 11708) v5 ==== 1376+0+0 (611409295 0 0) 0x1247d0e0 con 0x1395ec60 -5616> 2014-09-18 18:08:29.981365 7f4c9f251700 5 -- op tracker -- , seq: 85915, time: 2014-09-18 18:08:29.981269, event: header_read, op: pg_notify(2.410(12) epoch 11708) -5615> 2014-09-18 18:08:29.981381 7f4c9f251700 5 -- op tracker -- , seq: 85915, time: 2014-09-18 18:08:29.981270, event: throttled, op: pg_notify(2.410(12) epoch 11708) -5614> 2014-09-18 18:08:29.981392 7f4c9f251700 5 -- op tracker -- , seq: 85915, time: 2014-09-18 18:08:29.981309, event: all_read, op: pg_notify(2.410(12) epoch 11708) -5613> 2014-09-18 18:08:29.981403 7f4c9f251700 5 -- op tracker -- , seq: 85915, time: 2014-09-18 18:08:29.981359, event: dispatched, op: pg_notify(2.410(12) epoch 11708) -5612> 2014-09-18 18:08:29.981413 7f4c9f251700 5 -- op tracker -- , seq: 85915, time: 2014-09-18 18:08:29.981411, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11708) -5611> 2014-09-18 18:08:29.981426 7f4c9f251700 5 -- op tracker -- , seq: 85915, time: 2014-09-18 18:08:29.981423, event: started, op: pg_notify(2.410(12) epoch 11708) -5610> 2014-09-18 18:08:29.981452 7f4c9f251700 5 -- op tracker -- , seq: 85915, time: 2014-09-18 18:08:29.981450, event: done, op: pg_notify(2.410(12) epoch 11708) -5609> 2014-09-18 18:08:29.982482 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3185 ==== pg_notify(2.178(24),2.186(12) epoch 11708) v5 ==== 3360+0+0 (3840702546 0 0) 0xe9590c0 con 0x1395ec60 -5608> 2014-09-18 18:08:29.982503 7f4c9f251700 5 -- op tracker -- , seq: 85916, time: 2014-09-18 18:08:29.982387, event: header_read, op: pg_notify(2.178(24),2.186(12) epoch 11708) -5607> 2014-09-18 18:08:29.982518 7f4c9f251700 5 -- op tracker -- , seq: 85916, time: 2014-09-18 18:08:29.982388, event: throttled, op: pg_notify(2.178(24),2.186(12) epoch 11708) -5606> 2014-09-18 18:08:29.982528 7f4c9f251700 5 -- op tracker -- , seq: 85916, time: 2014-09-18 18:08:29.982444, event: all_read, op: pg_notify(2.178(24),2.186(12) epoch 11708) -5605> 2014-09-18 18:08:29.982546 7f4c9f251700 5 -- op tracker -- , seq: 85916, time: 2014-09-18 18:08:29.982497, event: dispatched, op: pg_notify(2.178(24),2.186(12) epoch 11708) -5604> 2014-09-18 18:08:29.982558 7f4c9f251700 5 -- op tracker -- , seq: 85916, time: 2014-09-18 18:08:29.982555, event: waiting_for_osdmap, op: pg_notify(2.178(24),2.186(12) epoch 11708) -5603> 2014-09-18 18:08:29.982571 7f4c9f251700 5 -- op tracker -- , seq: 85916, time: 2014-09-18 18:08:29.982568, event: started, op: pg_notify(2.178(24),2.186(12) epoch 11708) -5602> 2014-09-18 18:08:29.982604 7f4c9f251700 5 -- op tracker -- , seq: 85916, time: 2014-09-18 18:08:29.982602, event: done, op: pg_notify(2.178(24),2.186(12) epoch 11708) -5601> 2014-09-18 18:08:29.982611 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.265639 5 0.000108 -5600> 2014-09-18 18:08:29.982633 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -5599> 2014-09-18 18:08:29.982668 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000035 0 0.000000 -5598> 2014-09-18 18:08:29.982746 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -5597> 2014-09-18 18:08:29.982760 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000014 0 0.000000 -5596> 2014-09-18 18:08:29.982771 7f4c97241700 5 osd.42 pg_epoch: 11708 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -5595> 2014-09-18 18:08:30.018893 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.32 10.242.42.176:0/1099 112 ==== osd_ping(ping e11708 stamp 2014-09-18 23:37:34.766925) v2 ==== 47+0+0 (3459757377 0 0) 0x2b931a20 con 0x12abd080 -5594> 2014-09-18 18:08:30.018921 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/1099 -- osd_ping(ping_reply e11708 stamp 2014-09-18 23:37:34.766925) v2 -- ?+0 0xfccb480 con 0x12abd080 -5593> 2014-09-18 18:08:30.018931 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.32 10.242.42.176:0/1099 112 ==== osd_ping(ping e11708 stamp 2014-09-18 23:37:34.766925) v2 ==== 47+0+0 (3459757377 0 0) 0xe1a5480 con 0x110bb2e0 -5592> 2014-09-18 18:08:30.018952 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/1099 -- osd_ping(ping_reply e11708 stamp 2014-09-18 23:37:34.766925) v2 -- ?+0 0x11b11840 con 0x110bb2e0 -5591> 2014-09-18 18:08:30.048616 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3186 ==== pg_notify(2.74e(11) epoch 11708) v5 ==== 1303+0+0 (4290574251 0 0) 0xf80e000 con 0x1395ec60 -5590> 2014-09-18 18:08:30.048652 7f4c9f251700 5 -- op tracker -- , seq: 85917, time: 2014-09-18 18:08:30.048493, event: header_read, op: pg_notify(2.74e(11) epoch 11708) -5589> 2014-09-18 18:08:30.048669 7f4c9f251700 5 -- op tracker -- , seq: 85917, time: 2014-09-18 18:08:30.048496, event: throttled, op: pg_notify(2.74e(11) epoch 11708) -5588> 2014-09-18 18:08:30.048679 7f4c9f251700 5 -- op tracker -- , seq: 85917, time: 2014-09-18 18:08:30.048567, event: all_read, op: pg_notify(2.74e(11) epoch 11708) -5587> 2014-09-18 18:08:30.048690 7f4c9f251700 5 -- op tracker -- , seq: 85917, time: 2014-09-18 18:08:30.048642, event: dispatched, op: pg_notify(2.74e(11) epoch 11708) -5586> 2014-09-18 18:08:30.048701 7f4c9f251700 5 -- op tracker -- , seq: 85917, time: 2014-09-18 18:08:30.048698, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11708) -5585> 2014-09-18 18:08:30.048714 7f4c9f251700 5 -- op tracker -- , seq: 85917, time: 2014-09-18 18:08:30.048712, event: started, op: pg_notify(2.74e(11) epoch 11708) -5584> 2014-09-18 18:08:30.048745 7f4c9f251700 5 -- op tracker -- , seq: 85917, time: 2014-09-18 18:08:30.048743, event: done, op: pg_notify(2.74e(11) epoch 11708) -5583> 2014-09-18 18:08:30.164504 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 930 ==== pg_info(1 pgs e11708:2.34f) v4 ==== 738+0+0 (3974861194 0 0) 0xf4861e0 con 0x1395f8c0 -5582> 2014-09-18 18:08:30.164533 7f4c9f251700 5 -- op tracker -- , seq: 85918, time: 2014-09-18 18:08:30.164344, event: header_read, op: pg_info(1 pgs e11708:2.34f) -5581> 2014-09-18 18:08:30.164546 7f4c9f251700 5 -- op tracker -- , seq: 85918, time: 2014-09-18 18:08:30.164346, event: throttled, op: pg_info(1 pgs e11708:2.34f) -5580> 2014-09-18 18:08:30.164554 7f4c9f251700 5 -- op tracker -- , seq: 85918, time: 2014-09-18 18:08:30.164400, event: all_read, op: pg_info(1 pgs e11708:2.34f) -5579> 2014-09-18 18:08:30.164561 7f4c9f251700 5 -- op tracker -- , seq: 85918, time: 2014-09-18 18:08:30.164526, event: dispatched, op: pg_info(1 pgs e11708:2.34f) -5578> 2014-09-18 18:08:30.164569 7f4c9f251700 5 -- op tracker -- , seq: 85918, time: 2014-09-18 18:08:30.164567, event: waiting_for_osdmap, op: pg_info(1 pgs e11708:2.34f) -5577> 2014-09-18 18:08:30.164579 7f4c9f251700 5 -- op tracker -- , seq: 85918, time: 2014-09-18 18:08:30.164577, event: started, op: pg_info(1 pgs e11708:2.34f) -5576> 2014-09-18 18:08:30.164602 7f4c9f251700 5 -- op tracker -- , seq: 85918, time: 2014-09-18 18:08:30.164600, event: done, op: pg_info(1 pgs e11708:2.34f) -5575> 2014-09-18 18:08:30.184157 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 684 ==== pg_notify(2.4f6(15) epoch 11707) v5 ==== 1543+0+0 (1813678845 0 0) 0xfccab20 con 0x1f103e40 -5574> 2014-09-18 18:08:30.184178 7f4c9f251700 5 -- op tracker -- , seq: 85919, time: 2014-09-18 18:08:30.184053, event: header_read, op: pg_notify(2.4f6(15) epoch 11707) -5573> 2014-09-18 18:08:30.184194 7f4c9f251700 5 -- op tracker -- , seq: 85919, time: 2014-09-18 18:08:30.184055, event: throttled, op: pg_notify(2.4f6(15) epoch 11707) -5572> 2014-09-18 18:08:30.184206 7f4c9f251700 5 -- op tracker -- , seq: 85919, time: 2014-09-18 18:08:30.184111, event: all_read, op: pg_notify(2.4f6(15) epoch 11707) -5571> 2014-09-18 18:08:30.184217 7f4c9f251700 5 -- op tracker -- , seq: 85919, time: 2014-09-18 18:08:30.184173, event: dispatched, op: pg_notify(2.4f6(15) epoch 11707) -5570> 2014-09-18 18:08:30.184229 7f4c9f251700 5 -- op tracker -- , seq: 85919, time: 2014-09-18 18:08:30.184227, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11707) -5569> 2014-09-18 18:08:30.184241 7f4c9f251700 5 -- op tracker -- , seq: 85919, time: 2014-09-18 18:08:30.184239, event: started, op: pg_notify(2.4f6(15) epoch 11707) -5568> 2014-09-18 18:08:30.184262 7f4c9f251700 5 -- op tracker -- , seq: 85919, time: 2014-09-18 18:08:30.184261, event: done, op: pg_notify(2.4f6(15) epoch 11707) -5567> 2014-09-18 18:08:30.193923 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 685 ==== pg_notify(2.799(10),2.4c6(15) epoch 11707) v5 ==== 2779+0+0 (271343236 0 0) 0x21e10960 con 0x1f103e40 -5566> 2014-09-18 18:08:30.193943 7f4c9f251700 5 -- op tracker -- , seq: 85920, time: 2014-09-18 18:08:30.193761, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11707) -5565> 2014-09-18 18:08:30.193952 7f4c9f251700 5 -- op tracker -- , seq: 85920, time: 2014-09-18 18:08:30.193762, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11707) -5564> 2014-09-18 18:08:30.193957 7f4c9f251700 5 -- op tracker -- , seq: 85920, time: 2014-09-18 18:08:30.193820, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11707) -5563> 2014-09-18 18:08:30.193972 7f4c9f251700 5 -- op tracker -- , seq: 85920, time: 2014-09-18 18:08:30.193939, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11707) -5562> 2014-09-18 18:08:30.193986 7f4c9f251700 5 -- op tracker -- , seq: 85920, time: 2014-09-18 18:08:30.193984, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11707) -5561> 2014-09-18 18:08:30.193998 7f4c9f251700 5 -- op tracker -- , seq: 85920, time: 2014-09-18 18:08:30.193996, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11707) -5560> 2014-09-18 18:08:30.194043 7f4c9f251700 5 -- op tracker -- , seq: 85920, time: 2014-09-18 18:08:30.194041, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11707) -5559> 2014-09-18 18:08:30.195206 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1277 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x1533a6c0 con 0x1f102840 -5558> 2014-09-18 18:08:30.195226 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5557> 2014-09-18 18:08:30.195253 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1278 ==== pg_notify(2.4f6(15) epoch 11708) v5 ==== 1543+0+0 (345275160 0 0) 0x1368cf00 con 0x1f102840 -5556> 2014-09-18 18:08:30.195264 7f4c9f251700 5 -- op tracker -- , seq: 85921, time: 2014-09-18 18:08:30.195197, event: header_read, op: pg_notify(2.4f6(15) epoch 11708) -5555> 2014-09-18 18:08:30.195274 7f4c9f251700 5 -- op tracker -- , seq: 85921, time: 2014-09-18 18:08:30.195197, event: throttled, op: pg_notify(2.4f6(15) epoch 11708) -5554> 2014-09-18 18:08:30.195280 7f4c9f251700 5 -- op tracker -- , seq: 85921, time: 2014-09-18 18:08:30.195242, event: all_read, op: pg_notify(2.4f6(15) epoch 11708) -5553> 2014-09-18 18:08:30.195286 7f4c9f251700 5 -- op tracker -- , seq: 85921, time: 2014-09-18 18:08:30.195261, event: dispatched, op: pg_notify(2.4f6(15) epoch 11708) -5552> 2014-09-18 18:08:30.195292 7f4c9f251700 5 -- op tracker -- , seq: 85921, time: 2014-09-18 18:08:30.195291, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11708) -5551> 2014-09-18 18:08:30.195300 7f4c9f251700 5 -- op tracker -- , seq: 85921, time: 2014-09-18 18:08:30.195299, event: started, op: pg_notify(2.4f6(15) epoch 11708) -5550> 2014-09-18 18:08:30.195317 7f4c9f251700 5 -- op tracker -- , seq: 85921, time: 2014-09-18 18:08:30.195316, event: done, op: pg_notify(2.4f6(15) epoch 11708) -5549> 2014-09-18 18:08:30.199898 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 686 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x19df0480 con 0x1f103e40 -5548> 2014-09-18 18:08:30.199920 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5547> 2014-09-18 18:08:30.199937 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3187 ==== pg_notify(2.4b1(13) epoch 11708) v5 ==== 1441+0+0 (3249423885 0 0) 0xf80f0e0 con 0x1395ec60 -5546> 2014-09-18 18:08:30.199954 7f4c9f251700 5 -- op tracker -- , seq: 85922, time: 2014-09-18 18:08:30.199857, event: header_read, op: pg_notify(2.4b1(13) epoch 11708) -5545> 2014-09-18 18:08:30.199969 7f4c9f251700 5 -- op tracker -- , seq: 85922, time: 2014-09-18 18:08:30.199858, event: throttled, op: pg_notify(2.4b1(13) epoch 11708) -5544> 2014-09-18 18:08:30.199979 7f4c9f251700 5 -- op tracker -- , seq: 85922, time: 2014-09-18 18:08:30.199901, event: all_read, op: pg_notify(2.4b1(13) epoch 11708) -5543> 2014-09-18 18:08:30.199987 7f4c9f251700 5 -- op tracker -- , seq: 85922, time: 2014-09-18 18:08:30.199949, event: dispatched, op: pg_notify(2.4b1(13) epoch 11708) -5542> 2014-09-18 18:08:30.199999 7f4c9f251700 5 -- op tracker -- , seq: 85922, time: 2014-09-18 18:08:30.199996, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11708) -5541> 2014-09-18 18:08:30.200012 7f4c9f251700 5 -- op tracker -- , seq: 85922, time: 2014-09-18 18:08:30.200009, event: started, op: pg_notify(2.4b1(13) epoch 11708) -5540> 2014-09-18 18:08:30.200039 7f4c9f251700 5 -- op tracker -- , seq: 85922, time: 2014-09-18 18:08:30.200036, event: done, op: pg_notify(2.4b1(13) epoch 11708) -5539> 2014-09-18 18:08:30.200052 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 687 ==== pg_notify(2.4f6(15) epoch 11708) v5 ==== 1543+0+0 (3774854094 0 0) 0x21e101e0 con 0x1f103e40 -5538> 2014-09-18 18:08:30.200062 7f4c9f251700 5 -- op tracker -- , seq: 85923, time: 2014-09-18 18:08:30.199898, event: header_read, op: pg_notify(2.4f6(15) epoch 11708) -5537> 2014-09-18 18:08:30.200069 7f4c9f251700 5 -- op tracker -- , seq: 85923, time: 2014-09-18 18:08:30.199898, event: throttled, op: pg_notify(2.4f6(15) epoch 11708) -5536> 2014-09-18 18:08:30.200075 7f4c9f251700 5 -- op tracker -- , seq: 85923, time: 2014-09-18 18:08:30.199933, event: all_read, op: pg_notify(2.4f6(15) epoch 11708) -5535> 2014-09-18 18:08:30.200082 7f4c9f251700 5 -- op tracker -- , seq: 85923, time: 2014-09-18 18:08:30.200060, event: dispatched, op: pg_notify(2.4f6(15) epoch 11708) -5534> 2014-09-18 18:08:30.200090 7f4c9f251700 5 -- op tracker -- , seq: 85923, time: 2014-09-18 18:08:30.200088, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11708) -5533> 2014-09-18 18:08:30.200099 7f4c9f251700 5 -- op tracker -- , seq: 85923, time: 2014-09-18 18:08:30.200098, event: started, op: pg_notify(2.4f6(15) epoch 11708) -5532> 2014-09-18 18:08:30.200116 7f4c9f251700 5 -- op tracker -- , seq: 85923, time: 2014-09-18 18:08:30.200115, event: done, op: pg_notify(2.4f6(15) epoch 11708) -5531> 2014-09-18 18:08:30.200191 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3188 ==== pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11708) v5 ==== 9048+0+0 (711904185 0 0) 0xf811a20 con 0x1395ec60 -5530> 2014-09-18 18:08:30.200204 7f4c9f251700 5 -- op tracker -- , seq: 85924, time: 2014-09-18 18:08:30.200059, event: header_read, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11708) -5529> 2014-09-18 18:08:30.200248 7f4c9f251700 5 -- op tracker -- , seq: 85924, time: 2014-09-18 18:08:30.200059, event: throttled, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11708) -5528> 2014-09-18 18:08:30.200254 7f4c9f251700 5 -- op tracker -- , seq: 85924, time: 2014-09-18 18:08:30.200178, event: all_read, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11708) -5527> 2014-09-18 18:08:30.200261 7f4c9f251700 5 -- op tracker -- , seq: 85924, time: 2014-09-18 18:08:30.200200, event: dispatched, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11708) -5526> 2014-09-18 18:08:30.200268 7f4c9f251700 5 -- op tracker -- , seq: 85924, time: 2014-09-18 18:08:30.200266, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11708) -5525> 2014-09-18 18:08:30.200276 7f4c9f251700 5 -- op tracker -- , seq: 85924, time: 2014-09-18 18:08:30.200274, event: started, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11708) -5524> 2014-09-18 18:08:30.200317 7f4c9f251700 5 -- op tracker -- , seq: 85924, time: 2014-09-18 18:08:30.200315, event: done, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11708) -5523> 2014-09-18 18:08:30.201424 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 688 ==== pg_notify(2.4e8 epoch 11708) v5 ==== 722+0+0 (195919924 0 0) 0x21e12ee0 con 0x1f103e40 -5522> 2014-09-18 18:08:30.201443 7f4c9f251700 5 -- op tracker -- , seq: 85925, time: 2014-09-18 18:08:30.201378, event: header_read, op: pg_notify(2.4e8 epoch 11708) -5521> 2014-09-18 18:08:30.201454 7f4c9f251700 5 -- op tracker -- , seq: 85925, time: 2014-09-18 18:08:30.201380, event: throttled, op: pg_notify(2.4e8 epoch 11708) -5520> 2014-09-18 18:08:30.201460 7f4c9f251700 5 -- op tracker -- , seq: 85925, time: 2014-09-18 18:08:30.201406, event: all_read, op: pg_notify(2.4e8 epoch 11708) -5519> 2014-09-18 18:08:30.201466 7f4c9f251700 5 -- op tracker -- , seq: 85925, time: 2014-09-18 18:08:30.201439, event: dispatched, op: pg_notify(2.4e8 epoch 11708) -5518> 2014-09-18 18:08:30.201472 7f4c9f251700 5 -- op tracker -- , seq: 85925, time: 2014-09-18 18:08:30.201471, event: waiting_for_osdmap, op: pg_notify(2.4e8 epoch 11708) -5517> 2014-09-18 18:08:30.201480 7f4c9f251700 5 -- op tracker -- , seq: 85925, time: 2014-09-18 18:08:30.201479, event: started, op: pg_notify(2.4e8 epoch 11708) -5516> 2014-09-18 18:08:30.201497 7f4c9f251700 5 -- op tracker -- , seq: 85925, time: 2014-09-18 18:08:30.201496, event: done, op: pg_notify(2.4e8 epoch 11708) -5515> 2014-09-18 18:08:30.202724 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 689 ==== pg_notify(2.799(10),2.4c6(15) epoch 11708) v5 ==== 2779+0+0 (3753907126 0 0) 0x21e11fe0 con 0x1f103e40 -5514> 2014-09-18 18:08:30.202744 7f4c9f251700 5 -- op tracker -- , seq: 85926, time: 2014-09-18 18:08:30.202634, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11708) -5513> 2014-09-18 18:08:30.202755 7f4c9f251700 5 -- op tracker -- , seq: 85926, time: 2014-09-18 18:08:30.202635, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11708) -5512> 2014-09-18 18:08:30.202761 7f4c9f251700 5 -- op tracker -- , seq: 85926, time: 2014-09-18 18:08:30.202688, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11708) -5511> 2014-09-18 18:08:30.202768 7f4c9f251700 5 -- op tracker -- , seq: 85926, time: 2014-09-18 18:08:30.202739, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11708) -5510> 2014-09-18 18:08:30.202774 7f4c9f251700 5 -- op tracker -- , seq: 85926, time: 2014-09-18 18:08:30.202773, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11708) -5509> 2014-09-18 18:08:30.202783 7f4c9f251700 5 -- op tracker -- , seq: 85926, time: 2014-09-18 18:08:30.202781, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11708) -5508> 2014-09-18 18:08:30.202866 7f4c9f251700 5 -- op tracker -- , seq: 85926, time: 2014-09-18 18:08:30.202863, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11708) -5507> 2014-09-18 18:08:30.244799 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 339 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x1e285cc0 con 0x12562260 -5506> 2014-09-18 18:08:30.244837 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5505> 2014-09-18 18:08:30.244863 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 340 ==== pg_notify(2.719(17) epoch 11708) v5 ==== 1601+0+0 (41124429 0 0) 0x11738b40 con 0x12562260 -5504> 2014-09-18 18:08:30.244875 7f4c9f251700 5 -- op tracker -- , seq: 85927, time: 2014-09-18 18:08:30.244794, event: header_read, op: pg_notify(2.719(17) epoch 11708) -5503> 2014-09-18 18:08:30.244885 7f4c9f251700 5 -- op tracker -- , seq: 85927, time: 2014-09-18 18:08:30.244795, event: throttled, op: pg_notify(2.719(17) epoch 11708) -5502> 2014-09-18 18:08:30.244893 7f4c9f251700 5 -- op tracker -- , seq: 85927, time: 2014-09-18 18:08:30.244848, event: all_read, op: pg_notify(2.719(17) epoch 11708) -5501> 2014-09-18 18:08:30.244901 7f4c9f251700 5 -- op tracker -- , seq: 85927, time: 2014-09-18 18:08:30.244871, event: dispatched, op: pg_notify(2.719(17) epoch 11708) -5500> 2014-09-18 18:08:30.244909 7f4c9f251700 5 -- op tracker -- , seq: 85927, time: 2014-09-18 18:08:30.244907, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11708) -5499> 2014-09-18 18:08:30.244917 7f4c9f251700 5 -- op tracker -- , seq: 85927, time: 2014-09-18 18:08:30.244916, event: started, op: pg_notify(2.719(17) epoch 11708) -5498> 2014-09-18 18:08:30.244940 7f4c9f251700 5 -- op tracker -- , seq: 85927, time: 2014-09-18 18:08:30.244938, event: done, op: pg_notify(2.719(17) epoch 11708) -5497> 2014-09-18 18:08:30.274982 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3189 ==== pg_query(0.12 epoch 11708) v3 ==== 144+0+0 (3249968381 0 0) 0x2b9401c0 con 0x1395ec60 -5496> 2014-09-18 18:08:30.275011 7f4c9f251700 5 -- op tracker -- , seq: 85928, time: 2014-09-18 18:08:30.274910, event: header_read, op: pg_query(0.12 epoch 11708) -5495> 2014-09-18 18:08:30.275024 7f4c9f251700 5 -- op tracker -- , seq: 85928, time: 2014-09-18 18:08:30.274913, event: throttled, op: pg_query(0.12 epoch 11708) -5494> 2014-09-18 18:08:30.275032 7f4c9f251700 5 -- op tracker -- , seq: 85928, time: 2014-09-18 18:08:30.274960, event: all_read, op: pg_query(0.12 epoch 11708) -5493> 2014-09-18 18:08:30.275040 7f4c9f251700 5 -- op tracker -- , seq: 85928, time: 2014-09-18 18:08:30.275004, event: dispatched, op: pg_query(0.12 epoch 11708) -5492> 2014-09-18 18:08:30.275047 7f4c9f251700 5 -- op tracker -- , seq: 85928, time: 2014-09-18 18:08:30.275045, event: waiting_for_osdmap, op: pg_query(0.12 epoch 11708) -5491> 2014-09-18 18:08:30.275057 7f4c9f251700 5 -- op tracker -- , seq: 85928, time: 2014-09-18 18:08:30.275055, event: started, op: pg_query(0.12 epoch 11708) -5490> 2014-09-18 18:08:30.275086 7f4c9f251700 5 -- op tracker -- , seq: 85928, time: 2014-09-18 18:08:30.275085, event: done, op: pg_query(0.12 epoch 11708) -5489> 2014-09-18 18:08:30.275267 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(0.12(27) epoch 11708) v5 -- ?+0 0xe9583a0 con 0x1395ec60 -5488> 2014-09-18 18:08:30.297017 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.11 10.242.42.168:0/17298 126 ==== osd_ping(ping e11708 stamp 2014-09-18 18:08:30.163844) v2 ==== 47+0+0 (383511069 0 0) 0xf09dde0 con 0x101cd180 -5487> 2014-09-18 18:08:30.297051 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11708 stamp 2014-09-18 18:08:30.163844) v2 -- ?+0 0x2b931a20 con 0x101cd180 -5486> 2014-09-18 18:08:30.297071 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.11 10.242.42.168:0/17298 126 ==== osd_ping(ping e11708 stamp 2014-09-18 18:08:30.163844) v2 ==== 47+0+0 (383511069 0 0) 0xe76f480 con 0xcc49e40 -5485> 2014-09-18 18:08:30.297093 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11708 stamp 2014-09-18 18:08:30.163844) v2 -- ?+0 0xe1a5480 con 0xcc49e40 -5484> 2014-09-18 18:08:30.419508 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.52 10.242.42.168:0/16848 129 ==== osd_ping(ping e11708 stamp 2014-09-18 18:08:30.286176) v2 ==== 47+0+0 (185732256 0 0) 0xf1061e0 con 0x101cc520 -5483> 2014-09-18 18:08:30.419542 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.52 10.242.42.168:0/16848 129 ==== osd_ping(ping e11708 stamp 2014-09-18 18:08:30.286176) v2 ==== 47+0+0 (185732256 0 0) 0x12114d00 con 0xcc49b80 -5482> 2014-09-18 18:08:30.419580 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16848 -- osd_ping(ping_reply e11708 stamp 2014-09-18 18:08:30.286176) v2 -- ?+0 0xf09dde0 con 0x101cc520 -5481> 2014-09-18 18:08:30.419614 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16848 -- osd_ping(ping_reply e11708 stamp 2014-09-18 18:08:30.286176) v2 -- ?+0 0xe76f480 con 0xcc49b80 -5480> 2014-09-18 18:08:30.431568 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3190 ==== pg_notify(2.178(24) epoch 11708) v5 ==== 1986+0+0 (4182116833 0 0) 0xe9583a0 con 0x1395ec60 -5479> 2014-09-18 18:08:30.431590 7f4c9f251700 5 -- op tracker -- , seq: 85929, time: 2014-09-18 18:08:30.431500, event: header_read, op: pg_notify(2.178(24) epoch 11708) -5478> 2014-09-18 18:08:30.431601 7f4c9f251700 5 -- op tracker -- , seq: 85929, time: 2014-09-18 18:08:30.431501, event: throttled, op: pg_notify(2.178(24) epoch 11708) -5477> 2014-09-18 18:08:30.431610 7f4c9f251700 5 -- op tracker -- , seq: 85929, time: 2014-09-18 18:08:30.431550, event: all_read, op: pg_notify(2.178(24) epoch 11708) -5476> 2014-09-18 18:08:30.431617 7f4c9f251700 5 -- op tracker -- , seq: 85929, time: 2014-09-18 18:08:30.431584, event: dispatched, op: pg_notify(2.178(24) epoch 11708) -5475> 2014-09-18 18:08:30.431626 7f4c9f251700 5 -- op tracker -- , seq: 85929, time: 2014-09-18 18:08:30.431624, event: waiting_for_osdmap, op: pg_notify(2.178(24) epoch 11708) -5474> 2014-09-18 18:08:30.431635 7f4c9f251700 5 -- op tracker -- , seq: 85929, time: 2014-09-18 18:08:30.431634, event: started, op: pg_notify(2.178(24) epoch 11708) -5473> 2014-09-18 18:08:30.431658 7f4c9f251700 5 -- op tracker -- , seq: 85929, time: 2014-09-18 18:08:30.431656, event: done, op: pg_notify(2.178(24) epoch 11708) -5472> 2014-09-18 18:08:30.456674 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.55 10.242.42.172:0/19404 1104 ==== osd_ping(ping e11708 stamp 2014-09-18 23:37:19.560967) v2 ==== 47+0+0 (2875638346 0 0) 0xf2b2960 con 0x4b7ab00 -5471> 2014-09-18 18:08:30.456674 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.55 10.242.42.172:0/19404 1104 ==== osd_ping(ping e11708 stamp 2014-09-18 23:37:19.560967) v2 ==== 47+0+0 (2875638346 0 0) 0x1368c3c0 con 0xeddec60 -5470> 2014-09-18 18:08:30.456721 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/19404 -- osd_ping(ping_reply e11708 stamp 2014-09-18 23:37:19.560967) v2 -- ?+0 0x12114d00 con 0x4b7ab00 -5469> 2014-09-18 18:08:30.456749 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/19404 -- osd_ping(ping_reply e11708 stamp 2014-09-18 23:37:19.560967) v2 -- ?+0 0xf1061e0 con 0xeddec60 -5468> 2014-09-18 18:08:30.538832 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.14 10.242.42.168:0/16237 141 ==== osd_ping(ping e11708 stamp 2014-09-18 18:08:30.405584) v2 ==== 47+0+0 (2880516873 0 0) 0xe0bba40 con 0x101cc3c0 -5467> 2014-09-18 18:08:30.538879 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11708 stamp 2014-09-18 18:08:30.405584) v2 -- ?+0 0x1368c3c0 con 0x101cc3c0 -5466> 2014-09-18 18:08:30.538903 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.14 10.242.42.168:0/16237 141 ==== osd_ping(ping e11708 stamp 2014-09-18 18:08:30.405584) v2 ==== 47+0+0 (2880516873 0 0) 0xf7a6960 con 0xcc49a20 -5465> 2014-09-18 18:08:30.538932 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11708 stamp 2014-09-18 18:08:30.405584) v2 -- ?+0 0xf2b2960 con 0xcc49a20 -5464> 2014-09-18 18:08:30.574035 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 956 ==== pg_notify(2.517(11) epoch 11706) v5 ==== 1335+0+0 (2029288756 0 0) 0x21e12580 con 0x10a878c0 -5463> 2014-09-18 18:08:30.574061 7f4c9f251700 5 -- op tracker -- , seq: 85930, time: 2014-09-18 18:08:30.573961, event: header_read, op: pg_notify(2.517(11) epoch 11706) -5462> 2014-09-18 18:08:30.574071 7f4c9f251700 5 -- op tracker -- , seq: 85930, time: 2014-09-18 18:08:30.573964, event: throttled, op: pg_notify(2.517(11) epoch 11706) -5461> 2014-09-18 18:08:30.574076 7f4c9f251700 5 -- op tracker -- , seq: 85930, time: 2014-09-18 18:08:30.574018, event: all_read, op: pg_notify(2.517(11) epoch 11706) -5460> 2014-09-18 18:08:30.574080 7f4c9f251700 5 -- op tracker -- , seq: 85930, time: 2014-09-18 18:08:30.574055, event: dispatched, op: pg_notify(2.517(11) epoch 11706) -5459> 2014-09-18 18:08:30.574084 7f4c9f251700 5 -- op tracker -- , seq: 85930, time: 2014-09-18 18:08:30.574083, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11706) -5458> 2014-09-18 18:08:30.574091 7f4c9f251700 5 -- op tracker -- , seq: 85930, time: 2014-09-18 18:08:30.574090, event: started, op: pg_notify(2.517(11) epoch 11706) -5457> 2014-09-18 18:08:30.574114 7f4c9f251700 5 -- op tracker -- , seq: 85930, time: 2014-09-18 18:08:30.574113, event: done, op: pg_notify(2.517(11) epoch 11706) -5456> 2014-09-18 18:08:30.574283 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 957 ==== pg_notify(2.4c6(15) epoch 11706) v5 ==== 1511+0+0 (1084701602 0 0) 0xf788580 con 0x10a878c0 -5455> 2014-09-18 18:08:30.574296 7f4c9f251700 5 -- op tracker -- , seq: 85931, time: 2014-09-18 18:08:30.574211, event: header_read, op: pg_notify(2.4c6(15) epoch 11706) -5454> 2014-09-18 18:08:30.574304 7f4c9f251700 5 -- op tracker -- , seq: 85931, time: 2014-09-18 18:08:30.574212, event: throttled, op: pg_notify(2.4c6(15) epoch 11706) -5453> 2014-09-18 18:08:30.574310 7f4c9f251700 5 -- op tracker -- , seq: 85931, time: 2014-09-18 18:08:30.574269, event: all_read, op: pg_notify(2.4c6(15) epoch 11706) -5452> 2014-09-18 18:08:30.574316 7f4c9f251700 5 -- op tracker -- , seq: 85931, time: 2014-09-18 18:08:30.574293, event: dispatched, op: pg_notify(2.4c6(15) epoch 11706) -5451> 2014-09-18 18:08:30.574322 7f4c9f251700 5 -- op tracker -- , seq: 85931, time: 2014-09-18 18:08:30.574321, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11706) -5450> 2014-09-18 18:08:30.574330 7f4c9f251700 5 -- op tracker -- , seq: 85931, time: 2014-09-18 18:08:30.574328, event: started, op: pg_notify(2.4c6(15) epoch 11706) -5449> 2014-09-18 18:08:30.574347 7f4c9f251700 5 -- op tracker -- , seq: 85931, time: 2014-09-18 18:08:30.574346, event: done, op: pg_notify(2.4c6(15) epoch 11706) -5448> 2014-09-18 18:08:30.587435 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3191 ==== pg_notify(2.11b epoch 11708) v5 ==== 722+0+0 (1069589565 0 0) 0x24f78960 con 0x1395ec60 -5447> 2014-09-18 18:08:30.587454 7f4c9f251700 5 -- op tracker -- , seq: 85932, time: 2014-09-18 18:08:30.587370, event: header_read, op: pg_notify(2.11b epoch 11708) -5446> 2014-09-18 18:08:30.587467 7f4c9f251700 5 -- op tracker -- , seq: 85932, time: 2014-09-18 18:08:30.587372, event: throttled, op: pg_notify(2.11b epoch 11708) -5445> 2014-09-18 18:08:30.587475 7f4c9f251700 5 -- op tracker -- , seq: 85932, time: 2014-09-18 18:08:30.587416, event: all_read, op: pg_notify(2.11b epoch 11708) -5444> 2014-09-18 18:08:30.587483 7f4c9f251700 5 -- op tracker -- , seq: 85932, time: 2014-09-18 18:08:30.587450, event: dispatched, op: pg_notify(2.11b epoch 11708) -5443> 2014-09-18 18:08:30.587679 7f4c9f251700 5 -- op tracker -- , seq: 85932, time: 2014-09-18 18:08:30.587675, event: waiting_for_osdmap, op: pg_notify(2.11b epoch 11708) -5442> 2014-09-18 18:08:30.587702 7f4c9f251700 5 -- op tracker -- , seq: 85932, time: 2014-09-18 18:08:30.587698, event: started, op: pg_notify(2.11b epoch 11708) -5441> 2014-09-18 18:08:30.587747 7f4c9f251700 5 -- op tracker -- , seq: 85932, time: 2014-09-18 18:08:30.587744, event: done, op: pg_notify(2.11b epoch 11708) -5440> 2014-09-18 18:08:30.587808 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.866128 4 0.000377 -5439> 2014-09-18 18:08:30.587955 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -5438> 2014-09-18 18:08:30.588464 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000509 0 0.000000 -5437> 2014-09-18 18:08:30.588590 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -5436> 2014-09-18 18:08:30.588642 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000052 0 0.000000 -5435> 2014-09-18 18:08:30.588682 7f4c96a40700 5 osd.42 pg_epoch: 11708 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -5434> 2014-09-18 18:08:30.602391 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 958 ==== pg_notify(2.517(11) epoch 11707) v5 ==== 1335+0+0 (2045625844 0 0) 0x162512a0 con 0x10a878c0 -5433> 2014-09-18 18:08:30.602418 7f4c9f251700 5 -- op tracker -- , seq: 85933, time: 2014-09-18 18:08:30.602314, event: header_read, op: pg_notify(2.517(11) epoch 11707) -5432> 2014-09-18 18:08:30.602430 7f4c9f251700 5 -- op tracker -- , seq: 85933, time: 2014-09-18 18:08:30.602317, event: throttled, op: pg_notify(2.517(11) epoch 11707) -5431> 2014-09-18 18:08:30.602436 7f4c9f251700 5 -- op tracker -- , seq: 85933, time: 2014-09-18 18:08:30.602374, event: all_read, op: pg_notify(2.517(11) epoch 11707) -5430> 2014-09-18 18:08:30.602441 7f4c9f251700 5 -- op tracker -- , seq: 85933, time: 2014-09-18 18:08:30.602412, event: dispatched, op: pg_notify(2.517(11) epoch 11707) -5429> 2014-09-18 18:08:30.602454 7f4c9f251700 5 -- op tracker -- , seq: 85933, time: 2014-09-18 18:08:30.602452, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11707) -5428> 2014-09-18 18:08:30.602465 7f4c9f251700 5 -- op tracker -- , seq: 85933, time: 2014-09-18 18:08:30.602463, event: started, op: pg_notify(2.517(11) epoch 11707) -5427> 2014-09-18 18:08:30.602491 7f4c9f251700 5 -- op tracker -- , seq: 85933, time: 2014-09-18 18:08:30.602489, event: done, op: pg_notify(2.517(11) epoch 11707) -5426> 2014-09-18 18:08:30.602828 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 959 ==== pg_notify(2.4c6(15) epoch 11707) v5 ==== 1511+0+0 (3573698788 0 0) 0x1624e3c0 con 0x10a878c0 -5425> 2014-09-18 18:08:30.602844 7f4c9f251700 5 -- op tracker -- , seq: 85934, time: 2014-09-18 18:08:30.602782, event: header_read, op: pg_notify(2.4c6(15) epoch 11707) -5424> 2014-09-18 18:08:30.602850 7f4c9f251700 5 -- op tracker -- , seq: 85934, time: 2014-09-18 18:08:30.602783, event: throttled, op: pg_notify(2.4c6(15) epoch 11707) -5423> 2014-09-18 18:08:30.602854 7f4c9f251700 5 -- op tracker -- , seq: 85934, time: 2014-09-18 18:08:30.602815, event: all_read, op: pg_notify(2.4c6(15) epoch 11707) -5422> 2014-09-18 18:08:30.602859 7f4c9f251700 5 -- op tracker -- , seq: 85934, time: 2014-09-18 18:08:30.602840, event: dispatched, op: pg_notify(2.4c6(15) epoch 11707) -5421> 2014-09-18 18:08:30.602863 7f4c9f251700 5 -- op tracker -- , seq: 85934, time: 2014-09-18 18:08:30.602862, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11707) -5420> 2014-09-18 18:08:30.602869 7f4c9f251700 5 -- op tracker -- , seq: 85934, time: 2014-09-18 18:08:30.602867, event: started, op: pg_notify(2.4c6(15) epoch 11707) -5419> 2014-09-18 18:08:30.602888 7f4c9f251700 5 -- op tracker -- , seq: 85934, time: 2014-09-18 18:08:30.602886, event: done, op: pg_notify(2.4c6(15) epoch 11707) -5418> 2014-09-18 18:08:30.610469 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 960 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0x24dbef40 con 0x10a878c0 -5417> 2014-09-18 18:08:30.610491 7f4c9f251700 3 osd.42 11708 handle_osd_map epochs [11708,11708], i have 11708, src has [1,11708] -5416> 2014-09-18 18:08:30.610505 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 961 ==== pg_notify(2.4c6(15) epoch 11708) v5 ==== 1511+0+0 (860674853 0 0) 0x16250ee0 con 0x10a878c0 -5415> 2014-09-18 18:08:30.610516 7f4c9f251700 5 -- op tracker -- , seq: 85935, time: 2014-09-18 18:08:30.610467, event: header_read, op: pg_notify(2.4c6(15) epoch 11708) -5414> 2014-09-18 18:08:30.610525 7f4c9f251700 5 -- op tracker -- , seq: 85935, time: 2014-09-18 18:08:30.610467, event: throttled, op: pg_notify(2.4c6(15) epoch 11708) -5413> 2014-09-18 18:08:30.610531 7f4c9f251700 5 -- op tracker -- , seq: 85935, time: 2014-09-18 18:08:30.610496, event: all_read, op: pg_notify(2.4c6(15) epoch 11708) -5412> 2014-09-18 18:08:30.610544 7f4c9f251700 5 -- op tracker -- , seq: 85935, time: 2014-09-18 18:08:30.610512, event: dispatched, op: pg_notify(2.4c6(15) epoch 11708) -5411> 2014-09-18 18:08:30.610549 7f4c9f251700 5 -- op tracker -- , seq: 85935, time: 2014-09-18 18:08:30.610548, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11708) -5410> 2014-09-18 18:08:30.610563 7f4c9f251700 5 -- op tracker -- , seq: 85935, time: 2014-09-18 18:08:30.610561, event: started, op: pg_notify(2.4c6(15) epoch 11708) -5409> 2014-09-18 18:08:30.610601 7f4c9f251700 5 -- op tracker -- , seq: 85935, time: 2014-09-18 18:08:30.610596, event: done, op: pg_notify(2.4c6(15) epoch 11708) -5408> 2014-09-18 18:08:30.662696 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1279 ==== pg_notify(2.519 epoch 11708) v5 ==== 722+0+0 (2732574355 0 0) 0xf4e6d20 con 0x1f102840 -5407> 2014-09-18 18:08:30.662718 7f4c9f251700 5 -- op tracker -- , seq: 85936, time: 2014-09-18 18:08:30.662637, event: header_read, op: pg_notify(2.519 epoch 11708) -5406> 2014-09-18 18:08:30.662729 7f4c9f251700 5 -- op tracker -- , seq: 85936, time: 2014-09-18 18:08:30.662638, event: throttled, op: pg_notify(2.519 epoch 11708) -5405> 2014-09-18 18:08:30.662735 7f4c9f251700 5 -- op tracker -- , seq: 85936, time: 2014-09-18 18:08:30.662678, event: all_read, op: pg_notify(2.519 epoch 11708) -5404> 2014-09-18 18:08:30.662741 7f4c9f251700 5 -- op tracker -- , seq: 85936, time: 2014-09-18 18:08:30.662713, event: dispatched, op: pg_notify(2.519 epoch 11708) -5403> 2014-09-18 18:08:30.662747 7f4c9f251700 5 -- op tracker -- , seq: 85936, time: 2014-09-18 18:08:30.662746, event: waiting_for_osdmap, op: pg_notify(2.519 epoch 11708) -5402> 2014-09-18 18:08:30.662756 7f4c9f251700 5 -- op tracker -- , seq: 85936, time: 2014-09-18 18:08:30.662754, event: started, op: pg_notify(2.519 epoch 11708) -5401> 2014-09-18 18:08:30.662775 7f4c9f251700 5 -- op tracker -- , seq: 85936, time: 2014-09-18 18:08:30.662773, event: done, op: pg_notify(2.519 epoch 11708) -5400> 2014-09-18 18:08:30.675411 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 28 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x24d1a900 con 0x111771e0 -5399> 2014-09-18 18:08:30.675624 7f4ca0a54700 3 osd.42 11708 handle_osd_map epochs [11709,11709], i have 11708, src has [1,11709] -5398> 2014-09-18 18:08:30.676098 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 931 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0xe937440 con 0x1395f8c0 -5397> 2014-09-18 18:08:30.678731 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5396> 2014-09-18 18:08:30.678747 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 932 ==== pg_notify(2.78f(155) epoch 11709) v5 ==== 9335+0+0 (3772007661 0 0) 0xf487c20 con 0x1395f8c0 -5395> 2014-09-18 18:08:30.678765 7f4c9f251700 5 -- op tracker -- , seq: 85937, time: 2014-09-18 18:08:30.676160, event: header_read, op: pg_notify(2.78f(155) epoch 11709) -5394> 2014-09-18 18:08:30.678779 7f4c9f251700 5 -- op tracker -- , seq: 85937, time: 2014-09-18 18:08:30.676161, event: throttled, op: pg_notify(2.78f(155) epoch 11709) -5393> 2014-09-18 18:08:30.678789 7f4c9f251700 5 -- op tracker -- , seq: 85937, time: 2014-09-18 18:08:30.676514, event: all_read, op: pg_notify(2.78f(155) epoch 11709) -5392> 2014-09-18 18:08:30.678799 7f4c9f251700 5 -- op tracker -- , seq: 85937, time: 2014-09-18 18:08:30.678760, event: dispatched, op: pg_notify(2.78f(155) epoch 11709) -5391> 2014-09-18 18:08:30.678806 7f4c9f251700 5 -- op tracker -- , seq: 85937, time: 2014-09-18 18:08:30.678804, event: waiting_for_osdmap, op: pg_notify(2.78f(155) epoch 11709) -5390> 2014-09-18 18:08:30.678819 7f4c9f251700 5 -- op tracker -- , seq: 85937, time: 2014-09-18 18:08:30.678816, event: started, op: pg_notify(2.78f(155) epoch 11709) -5389> 2014-09-18 18:08:30.678841 7f4c9f251700 5 -- op tracker -- , seq: 85937, time: 2014-09-18 18:08:30.678840, event: done, op: pg_notify(2.78f(155) epoch 11709) -5388> 2014-09-18 18:08:30.679153 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 6 ==== pg_notify(2.120(15) epoch 11709) v5 ==== 1535+0+0 (2126445886 0 0) 0xf6dffe0 con 0x12b3aec0 -5387> 2014-09-18 18:08:30.679171 7f4c9f251700 5 -- op tracker -- , seq: 85938, time: 2014-09-18 18:08:30.679095, event: header_read, op: pg_notify(2.120(15) epoch 11709) -5386> 2014-09-18 18:08:30.679182 7f4c9f251700 5 -- op tracker -- , seq: 85938, time: 2014-09-18 18:08:30.679096, event: throttled, op: pg_notify(2.120(15) epoch 11709) -5385> 2014-09-18 18:08:30.679189 7f4c9f251700 5 -- op tracker -- , seq: 85938, time: 2014-09-18 18:08:30.679137, event: all_read, op: pg_notify(2.120(15) epoch 11709) -5384> 2014-09-18 18:08:30.679195 7f4c9f251700 5 -- op tracker -- , seq: 85938, time: 2014-09-18 18:08:30.679167, event: dispatched, op: pg_notify(2.120(15) epoch 11709) -5383> 2014-09-18 18:08:30.679201 7f4c9f251700 5 -- op tracker -- , seq: 85938, time: 2014-09-18 18:08:30.679200, event: waiting_for_osdmap, op: pg_notify(2.120(15) epoch 11709) -5382> 2014-09-18 18:08:30.679209 7f4c9f251700 5 -- op tracker -- , seq: 85938, time: 2014-09-18 18:08:30.679208, event: started, op: pg_notify(2.120(15) epoch 11709) -5381> 2014-09-18 18:08:30.679225 7f4c9f251700 5 -- op tracker -- , seq: 85938, time: 2014-09-18 18:08:30.679224, event: done, op: pg_notify(2.120(15) epoch 11709) -5380> 2014-09-18 18:08:30.679334 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x1440b840 con 0x1395ec60 -5379> 2014-09-18 18:08:30.679394 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.13(6),0.12(27) epoch 11709) v5 -- ?+0 0xf91d480 con 0x1395ec60 -5378> 2014-09-18 18:08:30.679403 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0xf953b00 con 0x140f95a0 -5377> 2014-09-18 18:08:30.679474 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11709) v5 -- ?+0 0xf867fe0 con 0x140f95a0 -5376> 2014-09-18 18:08:30.679535 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x2d7fd600 con 0x1395f1e0 -5375> 2014-09-18 18:08:30.679553 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11709) v5 -- ?+0 0xec681e0 con 0x1395f1e0 -5374> 2014-09-18 18:08:30.679609 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0xe339180 con 0x1395edc0 -5373> 2014-09-18 18:08:30.679628 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11709) v5 -- ?+0 0x121150c0 con 0x1395edc0 -5372> 2014-09-18 18:08:30.679958 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.697187 4 0.000197 -5371> 2014-09-18 18:08:30.680034 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.963072 0 0.000000 -5370> 2014-09-18 18:08:30.680058 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11600 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -5369> 2014-09-18 18:08:30.680342 7f4c97241700 0 log_channel(default) log [INF] : 2.178 restarting backfill on osd.20 from (0'0,0'0] MAX to 6874'18440 -5368> 2014-09-18 18:08:30.680439 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 301 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0xf953b00 con 0x140f95a0 -5367> 2014-09-18 18:08:30.680456 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5366> 2014-09-18 18:08:30.680463 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1264 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x14408480 con 0xe8db760 -5365> 2014-09-18 18:08:30.680472 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5364> 2014-09-18 18:08:30.680479 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 302 ==== pg_info(1 pgs e11709:2.756(16)) v4 ==== 1586+0+0 (1020981727 0 0) 0xf867fe0 con 0x140f95a0 -5363> 2014-09-18 18:08:30.680439 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.950582 4 0.000210 -5362> 2014-09-18 18:08:30.680501 7f4c9f251700 5 -- op tracker -- , seq: 85939, time: 2014-09-18 18:08:30.680045, event: header_read, op: pg_info(1 pgs e11709:2.756(16)) -5361> 2014-09-18 18:08:30.680511 7f4c9f251700 5 -- op tracker -- , seq: 85939, time: 2014-09-18 18:08:30.680045, event: throttled, op: pg_info(1 pgs e11709:2.756(16)) -5360> 2014-09-18 18:08:30.680516 7f4c9f251700 5 -- op tracker -- , seq: 85939, time: 2014-09-18 18:08:30.680082, event: all_read, op: pg_info(1 pgs e11709:2.756(16)) -5359> 2014-09-18 18:08:30.680508 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.960378 0 0.000000 -5358> 2014-09-18 18:08:30.680520 7f4c9f251700 5 -- op tracker -- , seq: 85939, time: 2014-09-18 18:08:30.680496, event: dispatched, op: pg_info(1 pgs e11709:2.756(16)) -5357> 2014-09-18 18:08:30.680524 7f4c9f251700 5 -- op tracker -- , seq: 85939, time: 2014-09-18 18:08:30.680523, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.756(16)) -5356> 2014-09-18 18:08:30.680530 7f4c9f251700 5 -- op tracker -- , seq: 85939, time: 2014-09-18 18:08:30.680529, event: started, op: pg_info(1 pgs e11709:2.756(16)) -5355> 2014-09-18 18:08:30.680526 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11485 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -5354> 2014-09-18 18:08:30.680543 7f4c9f251700 5 -- op tracker -- , seq: 85939, time: 2014-09-18 18:08:30.680542, event: done, op: pg_info(1 pgs e11709:2.756(16)) -5353> 2014-09-18 18:08:30.680552 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1265 ==== pg_notify(2.17e(17) epoch 11709) v5 ==== 1593+0+0 (1171030039 0 0) 0xf0daf00 con 0xe8db760 -5352> 2014-09-18 18:08:30.680560 7f4c9f251700 5 -- op tracker -- , seq: 85940, time: 2014-09-18 18:08:30.680076, event: header_read, op: pg_notify(2.17e(17) epoch 11709) -5351> 2014-09-18 18:08:30.680565 7f4c9f251700 5 -- op tracker -- , seq: 85940, time: 2014-09-18 18:08:30.680076, event: throttled, op: pg_notify(2.17e(17) epoch 11709) -5350> 2014-09-18 18:08:30.680569 7f4c9f251700 5 -- op tracker -- , seq: 85940, time: 2014-09-18 18:08:30.680128, event: all_read, op: pg_notify(2.17e(17) epoch 11709) -5349> 2014-09-18 18:08:30.680573 7f4c9f251700 5 -- op tracker -- , seq: 85940, time: 2014-09-18 18:08:30.680558, event: dispatched, op: pg_notify(2.17e(17) epoch 11709) -5348> 2014-09-18 18:08:30.680577 7f4c9f251700 5 -- op tracker -- , seq: 85940, time: 2014-09-18 18:08:30.680576, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11709) -5347> 2014-09-18 18:08:30.680582 7f4c9f251700 5 -- op tracker -- , seq: 85940, time: 2014-09-18 18:08:30.680581, event: started, op: pg_notify(2.17e(17) epoch 11709) -5346> 2014-09-18 18:08:30.680594 7f4c9f251700 5 -- op tracker -- , seq: 85940, time: 2014-09-18 18:08:30.680593, event: done, op: pg_notify(2.17e(17) epoch 11709) -5345> 2014-09-18 18:08:30.680932 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1039 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x19df2d00 con 0x14358260 -5344> 2014-09-18 18:08:30.680943 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5343> 2014-09-18 18:08:30.680998 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1040 ==== pg_notify(2.719(17) epoch 11709) v5 ==== 1601+0+0 (2738860369 0 0) 0xf730ee0 con 0x14358260 -5342> 2014-09-18 18:08:30.681007 7f4c9f251700 5 -- op tracker -- , seq: 85941, time: 2014-09-18 18:08:30.680942, event: header_read, op: pg_notify(2.719(17) epoch 11709) -5341> 2014-09-18 18:08:30.681006 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6822/33871 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x1533c640 con 0x4900000 -5340> 2014-09-18 18:08:30.681013 7f4c9f251700 5 -- op tracker -- , seq: 85941, time: 2014-09-18 18:08:30.680943, event: throttled, op: pg_notify(2.719(17) epoch 11709) -5339> 2014-09-18 18:08:30.681017 7f4c9f251700 5 -- op tracker -- , seq: 85941, time: 2014-09-18 18:08:30.680986, event: all_read, op: pg_notify(2.719(17) epoch 11709) -5338> 2014-09-18 18:08:30.681021 7f4c9f251700 5 -- op tracker -- , seq: 85941, time: 2014-09-18 18:08:30.681004, event: dispatched, op: pg_notify(2.719(17) epoch 11709) -5337> 2014-09-18 18:08:30.681021 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6822/33871 -- pg_log(2.178 epoch 11709 log log((4157'15439,6874'18440], crt=6810'18431) query_epoch 11709) v4 -- ?+0 0x21fabc00 con 0x4900000 -5336> 2014-09-18 18:08:30.681025 7f4c9f251700 5 -- op tracker -- , seq: 85941, time: 2014-09-18 18:08:30.681024, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11709) -5335> 2014-09-18 18:08:30.681038 7f4c96a40700 0 log_channel(default) log [INF] : 2.6f8 restarting backfill on osd.35 from (0'0,0'0] MAX to 6788'24342 -5334> 2014-09-18 18:08:30.681031 7f4c9f251700 5 -- op tracker -- , seq: 85941, time: 2014-09-18 18:08:30.681029, event: started, op: pg_notify(2.719(17) epoch 11709) -5333> 2014-09-18 18:08:30.681054 7f4c9f251700 5 -- op tracker -- , seq: 85941, time: 2014-09-18 18:08:30.681052, event: done, op: pg_notify(2.719(17) epoch 11709) -5332> 2014-09-18 18:08:30.681134 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11709 n=501 ec=188 les/c 11600/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -5331> 2014-09-18 18:08:30.681186 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1196 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x1182da80 con 0x1088b440 -5330> 2014-09-18 18:08:30.681201 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5329> 2014-09-18 18:08:30.681442 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 7 ==== pg_notify(2.40a(162) epoch 11709) v5 ==== 9736+0+0 (455734620 0 0) 0xf2b61e0 con 0x12b3aec0 -5328> 2014-09-18 18:08:30.681459 7f4c9f251700 5 -- op tracker -- , seq: 85942, time: 2014-09-18 18:08:30.681301, event: header_read, op: pg_notify(2.40a(162) epoch 11709) -5327> 2014-09-18 18:08:30.681471 7f4c9f251700 5 -- op tracker -- , seq: 85942, time: 2014-09-18 18:08:30.681302, event: throttled, op: pg_notify(2.40a(162) epoch 11709) -5326> 2014-09-18 18:08:30.681482 7f4c9f251700 5 -- op tracker -- , seq: 85942, time: 2014-09-18 18:08:30.681427, event: all_read, op: pg_notify(2.40a(162) epoch 11709) -5325> 2014-09-18 18:08:30.681492 7f4c9f251700 5 -- op tracker -- , seq: 85942, time: 2014-09-18 18:08:30.681454, event: dispatched, op: pg_notify(2.40a(162) epoch 11709) -5324> 2014-09-18 18:08:30.681503 7f4c9f251700 5 -- op tracker -- , seq: 85942, time: 2014-09-18 18:08:30.681500, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11709) -5323> 2014-09-18 18:08:30.681514 7f4c9f251700 5 -- op tracker -- , seq: 85942, time: 2014-09-18 18:08:30.681512, event: started, op: pg_notify(2.40a(162) epoch 11709) -5322> 2014-09-18 18:08:30.681535 7f4c9f251700 5 -- op tracker -- , seq: 85942, time: 2014-09-18 18:08:30.681533, event: done, op: pg_notify(2.40a(162) epoch 11709) -5321> 2014-09-18 18:08:30.681558 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2354 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0xe267f80 con 0xdd8c940 -5320> 2014-09-18 18:08:30.681572 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5319> 2014-09-18 18:08:30.681616 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1197 ==== pg_notify(2.2d8(173) epoch 11709) v5 ==== 10757+0+0 (1517593112 0 0) 0x1390a1e0 con 0x1088b440 -5318> 2014-09-18 18:08:30.681628 7f4c9f251700 5 -- op tracker -- , seq: 85943, time: 2014-09-18 18:08:30.681391, event: header_read, op: pg_notify(2.2d8(173) epoch 11709) -5317> 2014-09-18 18:08:30.681639 7f4c9f251700 5 -- op tracker -- , seq: 85943, time: 2014-09-18 18:08:30.681392, event: throttled, op: pg_notify(2.2d8(173) epoch 11709) -5316> 2014-09-18 18:08:30.681649 7f4c9f251700 5 -- op tracker -- , seq: 85943, time: 2014-09-18 18:08:30.681602, event: all_read, op: pg_notify(2.2d8(173) epoch 11709) -5315> 2014-09-18 18:08:30.681659 7f4c9f251700 5 -- op tracker -- , seq: 85943, time: 2014-09-18 18:08:30.681624, event: dispatched, op: pg_notify(2.2d8(173) epoch 11709) -5314> 2014-09-18 18:08:30.681669 7f4c9f251700 5 -- op tracker -- , seq: 85943, time: 2014-09-18 18:08:30.681667, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11709) -5313> 2014-09-18 18:08:30.681681 7f4c9f251700 5 -- op tracker -- , seq: 85943, time: 2014-09-18 18:08:30.681678, event: started, op: pg_notify(2.2d8(173) epoch 11709) -5312> 2014-09-18 18:08:30.681701 7f4c9f251700 5 -- op tracker -- , seq: 85943, time: 2014-09-18 18:08:30.681699, event: done, op: pg_notify(2.2d8(173) epoch 11709) -5311> 2014-09-18 18:08:30.681880 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 786 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x2d7fb680 con 0xe9a27e0 -5310> 2014-09-18 18:08:30.681904 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5309> 2014-09-18 18:08:30.682046 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 787 ==== pg_notify(2.2d8(173) epoch 11709) v5 ==== 10757+0+0 (1517593112 0 0) 0x142201c0 con 0xe9a27e0 -5308> 2014-09-18 18:08:30.682062 7f4c9f251700 5 -- op tracker -- , seq: 85944, time: 2014-09-18 18:08:30.681813, event: header_read, op: pg_notify(2.2d8(173) epoch 11709) -5307> 2014-09-18 18:08:30.682074 7f4c9f251700 5 -- op tracker -- , seq: 85944, time: 2014-09-18 18:08:30.681814, event: throttled, op: pg_notify(2.2d8(173) epoch 11709) -5306> 2014-09-18 18:08:30.682081 7f4c9f251700 5 -- op tracker -- , seq: 85944, time: 2014-09-18 18:08:30.682034, event: all_read, op: pg_notify(2.2d8(173) epoch 11709) -5305> 2014-09-18 18:08:30.682087 7f4c9f251700 5 -- op tracker -- , seq: 85944, time: 2014-09-18 18:08:30.682058, event: dispatched, op: pg_notify(2.2d8(173) epoch 11709) -5304> 2014-09-18 18:08:30.682094 7f4c9f251700 5 -- op tracker -- , seq: 85944, time: 2014-09-18 18:08:30.682092, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11709) -5303> 2014-09-18 18:08:30.682103 7f4c9f251700 5 -- op tracker -- , seq: 85944, time: 2014-09-18 18:08:30.682101, event: started, op: pg_notify(2.2d8(173) epoch 11709) -5302> 2014-09-18 18:08:30.682121 7f4c9f251700 5 -- op tracker -- , seq: 85944, time: 2014-09-18 18:08:30.682120, event: done, op: pg_notify(2.2d8(173) epoch 11709) -5301> 2014-09-18 18:08:30.682264 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 43 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x12bf1b00 con 0x1395f340 -5300> 2014-09-18 18:08:30.682282 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5299> 2014-09-18 18:08:30.682303 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 44 ==== pg_notify(2.1d8(9) epoch 11709) v5 ==== 1261+0+0 (2781272698 0 0) 0xf787860 con 0x1395f340 -5298> 2014-09-18 18:08:30.682314 7f4c9f251700 5 -- op tracker -- , seq: 85945, time: 2014-09-18 18:08:30.682262, event: header_read, op: pg_notify(2.1d8(9) epoch 11709) -5297> 2014-09-18 18:08:30.682323 7f4c9f251700 5 -- op tracker -- , seq: 85945, time: 2014-09-18 18:08:30.682262, event: throttled, op: pg_notify(2.1d8(9) epoch 11709) -5296> 2014-09-18 18:08:30.682329 7f4c9f251700 5 -- op tracker -- , seq: 85945, time: 2014-09-18 18:08:30.682293, event: all_read, op: pg_notify(2.1d8(9) epoch 11709) -5295> 2014-09-18 18:08:30.682335 7f4c9f251700 5 -- op tracker -- , seq: 85945, time: 2014-09-18 18:08:30.682310, event: dispatched, op: pg_notify(2.1d8(9) epoch 11709) -5294> 2014-09-18 18:08:30.682342 7f4c9f251700 5 -- op tracker -- , seq: 85945, time: 2014-09-18 18:08:30.682340, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11709) -5293> 2014-09-18 18:08:30.682350 7f4c9f251700 5 -- op tracker -- , seq: 85945, time: 2014-09-18 18:08:30.682348, event: started, op: pg_notify(2.1d8(9) epoch 11709) -5292> 2014-09-18 18:08:30.682367 7f4c9f251700 5 -- op tracker -- , seq: 85945, time: 2014-09-18 18:08:30.682365, event: done, op: pg_notify(2.1d8(9) epoch 11709) -5291> 2014-09-18 18:08:30.682656 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2355 ==== pg_notify(2.2c7(139) epoch 11709) v5 ==== 8651+0+0 (454176456 0 0) 0xf2b9de0 con 0xdd8c940 -5290> 2014-09-18 18:08:30.682675 7f4c9f251700 5 -- op tracker -- , seq: 85946, time: 2014-09-18 18:08:30.682474, event: header_read, op: pg_notify(2.2c7(139) epoch 11709) -5289> 2014-09-18 18:08:30.682686 7f4c9f251700 5 -- op tracker -- , seq: 85946, time: 2014-09-18 18:08:30.682475, event: throttled, op: pg_notify(2.2c7(139) epoch 11709) -5288> 2014-09-18 18:08:30.682695 7f4c9f251700 5 -- op tracker -- , seq: 85946, time: 2014-09-18 18:08:30.682640, event: all_read, op: pg_notify(2.2c7(139) epoch 11709) -5287> 2014-09-18 18:08:30.682721 7f4c9f251700 5 -- op tracker -- , seq: 85946, time: 2014-09-18 18:08:30.682671, event: dispatched, op: pg_notify(2.2c7(139) epoch 11709) -5286> 2014-09-18 18:08:30.682733 7f4c9f251700 5 -- op tracker -- , seq: 85946, time: 2014-09-18 18:08:30.682730, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11709) -5285> 2014-09-18 18:08:30.682744 7f4c9f251700 5 -- op tracker -- , seq: 85946, time: 2014-09-18 18:08:30.682742, event: started, op: pg_notify(2.2c7(139) epoch 11709) -5284> 2014-09-18 18:08:30.682765 7f4c9f251700 5 -- op tracker -- , seq: 85946, time: 2014-09-18 18:08:30.682763, event: done, op: pg_notify(2.2c7(139) epoch 11709) -5283> 2014-09-18 18:08:30.683148 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 933 ==== pg_notify(2.410(12) epoch 11709) v5 ==== 1338+0+0 (4089139301 0 0) 0xf4e81c0 con 0x1395f8c0 -5282> 2014-09-18 18:08:30.683169 7f4c9f251700 5 -- op tracker -- , seq: 85947, time: 2014-09-18 18:08:30.683051, event: header_read, op: pg_notify(2.410(12) epoch 11709) -5281> 2014-09-18 18:08:30.683180 7f4c9f251700 5 -- op tracker -- , seq: 85947, time: 2014-09-18 18:08:30.683052, event: throttled, op: pg_notify(2.410(12) epoch 11709) -5280> 2014-09-18 18:08:30.683196 7f4c9f251700 5 -- op tracker -- , seq: 85947, time: 2014-09-18 18:08:30.683129, event: all_read, op: pg_notify(2.410(12) epoch 11709) -5279> 2014-09-18 18:08:30.683206 7f4c9f251700 5 -- op tracker -- , seq: 85947, time: 2014-09-18 18:08:30.683163, event: dispatched, op: pg_notify(2.410(12) epoch 11709) -5278> 2014-09-18 18:08:30.683217 7f4c9f251700 5 -- op tracker -- , seq: 85947, time: 2014-09-18 18:08:30.683214, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11709) -5277> 2014-09-18 18:08:30.683228 7f4c9f251700 5 -- op tracker -- , seq: 85947, time: 2014-09-18 18:08:30.683226, event: started, op: pg_notify(2.410(12) epoch 11709) -5276> 2014-09-18 18:08:30.683250 7f4c9f251700 5 -- op tracker -- , seq: 85947, time: 2014-09-18 18:08:30.683248, event: done, op: pg_notify(2.410(12) epoch 11709) -5275> 2014-09-18 18:08:30.683497 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 532 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x1e282240 con 0x1395ef20 -5274> 2014-09-18 18:08:30.683514 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5273> 2014-09-18 18:08:30.683553 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 533 ==== pg_info(1 pgs e11709:2.352(11)) v4 ==== 1319+0+0 (3551247420 0 0) 0x146e0ee0 con 0x1395ef20 -5272> 2014-09-18 18:08:30.683566 7f4c9f251700 5 -- op tracker -- , seq: 85948, time: 2014-09-18 18:08:30.683503, event: header_read, op: pg_info(1 pgs e11709:2.352(11)) -5271> 2014-09-18 18:08:30.683577 7f4c9f251700 5 -- op tracker -- , seq: 85948, time: 2014-09-18 18:08:30.683504, event: throttled, op: pg_info(1 pgs e11709:2.352(11)) -5270> 2014-09-18 18:08:30.683587 7f4c9f251700 5 -- op tracker -- , seq: 85948, time: 2014-09-18 18:08:30.683542, event: all_read, op: pg_info(1 pgs e11709:2.352(11)) -5269> 2014-09-18 18:08:30.683597 7f4c9f251700 5 -- op tracker -- , seq: 85948, time: 2014-09-18 18:08:30.683561, event: dispatched, op: pg_info(1 pgs e11709:2.352(11)) -5268> 2014-09-18 18:08:30.683607 7f4c9f251700 5 -- op tracker -- , seq: 85948, time: 2014-09-18 18:08:30.683605, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.352(11)) -5267> 2014-09-18 18:08:30.683620 7f4c9f251700 5 -- op tracker -- , seq: 85948, time: 2014-09-18 18:08:30.683617, event: started, op: pg_info(1 pgs e11709:2.352(11)) -5266> 2014-09-18 18:08:30.683639 7f4c9f251700 5 -- op tracker -- , seq: 85948, time: 2014-09-18 18:08:30.683638, event: done, op: pg_info(1 pgs e11709:2.352(11)) -5265> 2014-09-18 18:08:30.683944 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0xe2ec6c0 con 0xcbecec0 -5264> 2014-09-18 18:08:30.683966 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_log(2.6f8 epoch 11709 log log((4060'20573,6788'24342], crt=6788'24342) query_epoch 11709) v4 -- ?+0 0x22edb800 con 0xcbecec0 -5263> 2014-09-18 18:08:30.683980 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11709 n=519 ec=188 les/c 11485/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -5262> 2014-09-18 18:08:30.684090 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3294 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x12bf0900 con 0x4900580 -5261> 2014-09-18 18:08:30.684106 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5260> 2014-09-18 18:08:30.684154 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.928269 3 0.000254 -5259> 2014-09-18 18:08:30.684281 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.966491 0 0.000000 -5258> 2014-09-18 18:08:30.684299 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11485 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -5257> 2014-09-18 18:08:30.684348 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.756(27) epoch 11709) v5 -- ?+0 0xf7a7e00 con 0x140f95a0 -5256> 2014-09-18 18:08:30.684371 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x1533bb00 con 0x1395ef20 -5255> 2014-09-18 18:08:30.684382 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(2.352(11) epoch 11709) v5 -- ?+0 0xf7a8760 con 0x1395ef20 -5254> 2014-09-18 18:08:30.684394 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_info(1 pgs e11709:2.6f8(10)) v4 -- ?+0 0xf91d840 con 0x1395edc0 -5253> 2014-09-18 18:08:30.684715 7f4c97241700 0 log_channel(default) log [INF] : 2.179 restarting backfill on osd.35 from (0'0,0'0] MAX to 6752'29654 -5252> 2014-09-18 18:08:30.684736 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 8 ==== pg_notify(2.1a2(20) epoch 11709) v5 ==== 1798+0+0 (891929268 0 0) 0x2b93af00 con 0x12b3aec0 -5251> 2014-09-18 18:08:30.684752 7f4c9f251700 5 -- op tracker -- , seq: 85949, time: 2014-09-18 18:08:30.684681, event: header_read, op: pg_notify(2.1a2(20) epoch 11709) -5250> 2014-09-18 18:08:30.684760 7f4c9f251700 5 -- op tracker -- , seq: 85949, time: 2014-09-18 18:08:30.684682, event: throttled, op: pg_notify(2.1a2(20) epoch 11709) -5249> 2014-09-18 18:08:30.684766 7f4c9f251700 5 -- op tracker -- , seq: 85949, time: 2014-09-18 18:08:30.684720, event: all_read, op: pg_notify(2.1a2(20) epoch 11709) -5248> 2014-09-18 18:08:30.684774 7f4c9f251700 5 -- op tracker -- , seq: 85949, time: 2014-09-18 18:08:30.684748, event: dispatched, op: pg_notify(2.1a2(20) epoch 11709) -5247> 2014-09-18 18:08:30.684781 7f4c9f251700 5 -- op tracker -- , seq: 85949, time: 2014-09-18 18:08:30.684780, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11709) -5246> 2014-09-18 18:08:30.684791 7f4c9f251700 5 -- op tracker -- , seq: 85949, time: 2014-09-18 18:08:30.684787, event: started, op: pg_notify(2.1a2(20) epoch 11709) -5245> 2014-09-18 18:08:30.684808 7f4c9f251700 5 -- op tracker -- , seq: 85949, time: 2014-09-18 18:08:30.684805, event: done, op: pg_notify(2.1a2(20) epoch 11709) -5244> 2014-09-18 18:08:30.685185 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1280 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x1e285180 con 0x1f102840 -5243> 2014-09-18 18:08:30.685205 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5242> 2014-09-18 18:08:30.685215 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3295 ==== pg_notify(2.4f6(184) epoch 11709) v5 ==== 11296+0+0 (3975906946 0 0) 0xf1e65a0 con 0x4900580 -5241> 2014-09-18 18:08:30.685233 7f4c9f251700 5 -- op tracker -- , seq: 85950, time: 2014-09-18 18:08:30.684984, event: header_read, op: pg_notify(2.4f6(184) epoch 11709) -5240> 2014-09-18 18:08:30.685243 7f4c9f251700 5 -- op tracker -- , seq: 85950, time: 2014-09-18 18:08:30.684985, event: throttled, op: pg_notify(2.4f6(184) epoch 11709) -5239> 2014-09-18 18:08:30.685254 7f4c9f251700 5 -- op tracker -- , seq: 85950, time: 2014-09-18 18:08:30.685185, event: all_read, op: pg_notify(2.4f6(184) epoch 11709) -5238> 2014-09-18 18:08:30.685266 7f4c9f251700 5 -- op tracker -- , seq: 85950, time: 2014-09-18 18:08:30.685229, event: dispatched, op: pg_notify(2.4f6(184) epoch 11709) -5237> 2014-09-18 18:08:30.685274 7f4c9f251700 5 -- op tracker -- , seq: 85950, time: 2014-09-18 18:08:30.685272, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11709) -5236> 2014-09-18 18:08:30.685282 7f4c9f251700 5 -- op tracker -- , seq: 85950, time: 2014-09-18 18:08:30.685281, event: started, op: pg_notify(2.4f6(184) epoch 11709) -5235> 2014-09-18 18:08:30.685300 7f4c9f251700 5 -- op tracker -- , seq: 85950, time: 2014-09-18 18:08:30.685298, event: done, op: pg_notify(2.4f6(184) epoch 11709) -5234> 2014-09-18 18:08:30.685442 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1281 ==== pg_notify(2.4f6(15) epoch 11709) v5 ==== 1543+0+0 (1291506926 0 0) 0xf4e72c0 con 0x1f102840 -5233> 2014-09-18 18:08:30.685457 7f4c9f251700 5 -- op tracker -- , seq: 85951, time: 2014-09-18 18:08:30.685159, event: header_read, op: pg_notify(2.4f6(15) epoch 11709) -5232> 2014-09-18 18:08:30.685464 7f4c9f251700 5 -- op tracker -- , seq: 85951, time: 2014-09-18 18:08:30.685159, event: throttled, op: pg_notify(2.4f6(15) epoch 11709) -5231> 2014-09-18 18:08:30.685468 7f4c9f251700 5 -- op tracker -- , seq: 85951, time: 2014-09-18 18:08:30.685205, event: all_read, op: pg_notify(2.4f6(15) epoch 11709) -5230> 2014-09-18 18:08:30.685475 7f4c9f251700 5 -- op tracker -- , seq: 85951, time: 2014-09-18 18:08:30.685454, event: dispatched, op: pg_notify(2.4f6(15) epoch 11709) -5229> 2014-09-18 18:08:30.685484 7f4c9f251700 5 -- op tracker -- , seq: 85951, time: 2014-09-18 18:08:30.685482, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11709) -5228> 2014-09-18 18:08:30.685495 7f4c9f251700 5 -- op tracker -- , seq: 85951, time: 2014-09-18 18:08:30.685492, event: started, op: pg_notify(2.4f6(15) epoch 11709) -5227> 2014-09-18 18:08:30.685523 7f4c9f251700 5 -- op tracker -- , seq: 85951, time: 2014-09-18 18:08:30.685520, event: done, op: pg_notify(2.4f6(15) epoch 11709) -5226> 2014-09-18 18:08:30.685549 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3296 ==== pg_notify(2.17e(17) epoch 11709) v5 ==== 1593+0+0 (1171030039 0 0) 0xf1e7e00 con 0x4900580 -5225> 2014-09-18 18:08:30.685563 7f4c9f251700 5 -- op tracker -- , seq: 85952, time: 2014-09-18 18:08:30.685206, event: header_read, op: pg_notify(2.17e(17) epoch 11709) -5224> 2014-09-18 18:08:30.685601 7f4c9f251700 5 -- op tracker -- , seq: 85952, time: 2014-09-18 18:08:30.685207, event: throttled, op: pg_notify(2.17e(17) epoch 11709) -5223> 2014-09-18 18:08:30.685610 7f4c9f251700 5 -- op tracker -- , seq: 85952, time: 2014-09-18 18:08:30.685251, event: all_read, op: pg_notify(2.17e(17) epoch 11709) -5222> 2014-09-18 18:08:30.685617 7f4c9f251700 5 -- op tracker -- , seq: 85952, time: 2014-09-18 18:08:30.685559, event: dispatched, op: pg_notify(2.17e(17) epoch 11709) -5221> 2014-09-18 18:08:30.685625 7f4c9f251700 5 -- op tracker -- , seq: 85952, time: 2014-09-18 18:08:30.685623, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11709) -5220> 2014-09-18 18:08:30.685634 7f4c9f251700 5 -- op tracker -- , seq: 85952, time: 2014-09-18 18:08:30.685631, event: started, op: pg_notify(2.17e(17) epoch 11709) -5219> 2014-09-18 18:08:30.685655 7f4c9f251700 5 -- op tracker -- , seq: 85952, time: 2014-09-18 18:08:30.685653, event: done, op: pg_notify(2.17e(17) epoch 11709) -5218> 2014-09-18 18:08:30.685668 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2011 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0xef56d80 con 0xe2d9a20 -5217> 2014-09-18 18:08:30.685678 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5216> 2014-09-18 18:08:30.685685 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 534 ==== pg_notify(2.178(3) epoch 11709) v5 ==== 907+0+0 (1648789011 0 0) 0xf7a8760 con 0x1395ef20 -5215> 2014-09-18 18:08:30.685692 7f4c9f251700 5 -- op tracker -- , seq: 85953, time: 2014-09-18 18:08:30.685488, event: header_read, op: pg_notify(2.178(3) epoch 11709) -5214> 2014-09-18 18:08:30.685697 7f4c9f251700 5 -- op tracker -- , seq: 85953, time: 2014-09-18 18:08:30.685490, event: throttled, op: pg_notify(2.178(3) epoch 11709) -5213> 2014-09-18 18:08:30.685701 7f4c9f251700 5 -- op tracker -- , seq: 85953, time: 2014-09-18 18:08:30.685525, event: all_read, op: pg_notify(2.178(3) epoch 11709) -5212> 2014-09-18 18:08:30.685705 7f4c9f251700 5 -- op tracker -- , seq: 85953, time: 2014-09-18 18:08:30.685689, event: dispatched, op: pg_notify(2.178(3) epoch 11709) -5211> 2014-09-18 18:08:30.685723 7f4c9f251700 5 -- op tracker -- , seq: 85953, time: 2014-09-18 18:08:30.685721, event: waiting_for_osdmap, op: pg_notify(2.178(3) epoch 11709) -5210> 2014-09-18 18:08:30.685734 7f4c9f251700 5 -- op tracker -- , seq: 85953, time: 2014-09-18 18:08:30.685733, event: started, op: pg_notify(2.178(3) epoch 11709) -5209> 2014-09-18 18:08:30.685757 7f4c9f251700 5 -- op tracker -- , seq: 85953, time: 2014-09-18 18:08:30.685756, event: done, op: pg_notify(2.178(3) epoch 11709) -5208> 2014-09-18 18:08:30.685768 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 109 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0xe339180 con 0x1395edc0 -5207> 2014-09-18 18:08:30.685780 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5206> 2014-09-18 18:08:30.685786 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2356 ==== pg_notify(2.457(21) epoch 11709) v5 ==== 1875+0+0 (4050102715 0 0) 0xf2b8940 con 0xdd8c940 -5205> 2014-09-18 18:08:30.685795 7f4c9f251700 5 -- op tracker -- , seq: 85954, time: 2014-09-18 18:08:30.685409, event: header_read, op: pg_notify(2.457(21) epoch 11709) -5204> 2014-09-18 18:08:30.685802 7f4c9f251700 5 -- op tracker -- , seq: 85954, time: 2014-09-18 18:08:30.685410, event: throttled, op: pg_notify(2.457(21) epoch 11709) -5203> 2014-09-18 18:08:30.685808 7f4c9f251700 5 -- op tracker -- , seq: 85954, time: 2014-09-18 18:08:30.685457, event: all_read, op: pg_notify(2.457(21) epoch 11709) -5202> 2014-09-18 18:08:30.685814 7f4c9f251700 5 -- op tracker -- , seq: 85954, time: 2014-09-18 18:08:30.685792, event: dispatched, op: pg_notify(2.457(21) epoch 11709) -5201> 2014-09-18 18:08:30.685821 7f4c9f251700 5 -- op tracker -- , seq: 85954, time: 2014-09-18 18:08:30.685819, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11709) -5200> 2014-09-18 18:08:30.685828 7f4c9f251700 5 -- op tracker -- , seq: 85954, time: 2014-09-18 18:08:30.685827, event: started, op: pg_notify(2.457(21) epoch 11709) -5199> 2014-09-18 18:08:30.685849 7f4c9f251700 5 -- op tracker -- , seq: 85954, time: 2014-09-18 18:08:30.685846, event: done, op: pg_notify(2.457(21) epoch 11709) -5198> 2014-09-18 18:08:30.685864 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2596 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x14408240 con 0x4900b00 -5197> 2014-09-18 18:08:30.685932 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5196> 2014-09-18 18:08:30.685938 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2012 ==== pg_notify(2.26d(9) epoch 11709) v5 ==== 1257+0+0 (3207849697 0 0) 0xec692c0 con 0xe2d9a20 -5195> 2014-09-18 18:08:30.685947 7f4c9f251700 5 -- op tracker -- , seq: 85955, time: 2014-09-18 18:08:30.685466, event: header_read, op: pg_notify(2.26d(9) epoch 11709) -5194> 2014-09-18 18:08:30.685954 7f4c9f251700 5 -- op tracker -- , seq: 85955, time: 2014-09-18 18:08:30.685467, event: throttled, op: pg_notify(2.26d(9) epoch 11709) -5193> 2014-09-18 18:08:30.685975 7f4c9f251700 5 -- op tracker -- , seq: 85955, time: 2014-09-18 18:08:30.685505, event: all_read, op: pg_notify(2.26d(9) epoch 11709) -5192> 2014-09-18 18:08:30.686000 7f4c9f251700 5 -- op tracker -- , seq: 85955, time: 2014-09-18 18:08:30.685945, event: dispatched, op: pg_notify(2.26d(9) epoch 11709) -5191> 2014-09-18 18:08:30.686013 7f4c9f251700 5 -- op tracker -- , seq: 85955, time: 2014-09-18 18:08:30.686010, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11709) -5190> 2014-09-18 18:08:30.686037 7f4c9f251700 5 -- op tracker -- , seq: 85955, time: 2014-09-18 18:08:30.686034, event: started, op: pg_notify(2.26d(9) epoch 11709) -5189> 2014-09-18 18:08:30.686076 7f4c9f251700 5 -- op tracker -- , seq: 85955, time: 2014-09-18 18:08:30.686073, event: done, op: pg_notify(2.26d(9) epoch 11709) -5188> 2014-09-18 18:08:30.686240 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 110 ==== pg_notify(2.6f8(141) epoch 11709) v5 ==== 8801+0+0 (1655622580 0 0) 0xf91d840 con 0x1395edc0 -5187> 2014-09-18 18:08:30.686263 7f4c9f251700 5 -- op tracker -- , seq: 85956, time: 2014-09-18 18:08:30.685562, event: header_read, op: pg_notify(2.6f8(141) epoch 11709) -5186> 2014-09-18 18:08:30.686274 7f4c9f251700 5 -- op tracker -- , seq: 85956, time: 2014-09-18 18:08:30.685562, event: throttled, op: pg_notify(2.6f8(141) epoch 11709) -5185> 2014-09-18 18:08:30.686282 7f4c9f251700 5 -- op tracker -- , seq: 85956, time: 2014-09-18 18:08:30.685715, event: all_read, op: pg_notify(2.6f8(141) epoch 11709) -5184> 2014-09-18 18:08:30.686318 7f4c9f251700 5 -- op tracker -- , seq: 85956, time: 2014-09-18 18:08:30.686254, event: dispatched, op: pg_notify(2.6f8(141) epoch 11709) -5183> 2014-09-18 18:08:30.686330 7f4c9f251700 5 -- op tracker -- , seq: 85956, time: 2014-09-18 18:08:30.686328, event: waiting_for_osdmap, op: pg_notify(2.6f8(141) epoch 11709) -5182> 2014-09-18 18:08:30.686394 7f4c9f251700 5 -- op tracker -- , seq: 85956, time: 2014-09-18 18:08:30.686363, event: started, op: pg_notify(2.6f8(141) epoch 11709) -5181> 2014-09-18 18:08:30.686417 7f4c9f251700 5 -- op tracker -- , seq: 85956, time: 2014-09-18 18:08:30.686416, event: done, op: pg_notify(2.6f8(141) epoch 11709) -5180> 2014-09-18 18:08:30.686512 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3297 ==== pg_notify(2.1bb(9) epoch 11709) v5 ==== 1189+0+0 (4015314158 0 0) 0x2b93b2c0 con 0x4900580 -5179> 2014-09-18 18:08:30.686536 7f4c9f251700 5 -- op tracker -- , seq: 85957, time: 2014-09-18 18:08:30.686005, event: header_read, op: pg_notify(2.1bb(9) epoch 11709) -5178> 2014-09-18 18:08:30.686580 7f4c9f251700 5 -- op tracker -- , seq: 85957, time: 2014-09-18 18:08:30.686006, event: throttled, op: pg_notify(2.1bb(9) epoch 11709) -5177> 2014-09-18 18:08:30.686608 7f4c9f251700 5 -- op tracker -- , seq: 85957, time: 2014-09-18 18:08:30.686044, event: all_read, op: pg_notify(2.1bb(9) epoch 11709) -5176> 2014-09-18 18:08:30.686619 7f4c9f251700 5 -- op tracker -- , seq: 85957, time: 2014-09-18 18:08:30.686530, event: dispatched, op: pg_notify(2.1bb(9) epoch 11709) -5175> 2014-09-18 18:08:30.686630 7f4c9f251700 5 -- op tracker -- , seq: 85957, time: 2014-09-18 18:08:30.686627, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11709) -5174> 2014-09-18 18:08:30.686642 7f4c9f251700 5 -- op tracker -- , seq: 85957, time: 2014-09-18 18:08:30.686639, event: started, op: pg_notify(2.1bb(9) epoch 11709) -5173> 2014-09-18 18:08:30.686663 7f4c9f251700 5 -- op tracker -- , seq: 85957, time: 2014-09-18 18:08:30.686660, event: done, op: pg_notify(2.1bb(9) epoch 11709) -5172> 2014-09-18 18:08:30.686603 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_log(2.179 epoch 11709 log log((4088'25910,6752'29654], crt=6752'29654) query_epoch 11709) v4 -- ?+0 0x22fec300 con 0xcbecec0 -5171> 2014-09-18 18:08:30.686680 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2597 ==== pg_notify(2.71c(12) epoch 11709) v5 ==== 1424+0+0 (3375957120 0 0) 0xf7a7680 con 0x4900b00 -5170> 2014-09-18 18:08:30.686700 7f4c9f251700 5 -- op tracker -- , seq: 85958, time: 2014-09-18 18:08:30.685796, event: header_read, op: pg_notify(2.71c(12) epoch 11709) -5169> 2014-09-18 18:08:30.686709 7f4c9f251700 5 -- op tracker -- , seq: 85958, time: 2014-09-18 18:08:30.685797, event: throttled, op: pg_notify(2.71c(12) epoch 11709) -5168> 2014-09-18 18:08:30.686715 7f4c9f251700 5 -- op tracker -- , seq: 85958, time: 2014-09-18 18:08:30.685837, event: all_read, op: pg_notify(2.71c(12) epoch 11709) -5167> 2014-09-18 18:08:30.686721 7f4c9f251700 5 -- op tracker -- , seq: 85958, time: 2014-09-18 18:08:30.686697, event: dispatched, op: pg_notify(2.71c(12) epoch 11709) -5166> 2014-09-18 18:08:30.686727 7f4c9f251700 5 -- op tracker -- , seq: 85958, time: 2014-09-18 18:08:30.686725, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11709) -5165> 2014-09-18 18:08:30.686734 7f4c9f251700 5 -- op tracker -- , seq: 85958, time: 2014-09-18 18:08:30.686733, event: started, op: pg_notify(2.71c(12) epoch 11709) -5164> 2014-09-18 18:08:30.686766 7f4c9f251700 5 -- op tracker -- , seq: 85958, time: 2014-09-18 18:08:30.686764, event: done, op: pg_notify(2.71c(12) epoch 11709) -5163> 2014-09-18 18:08:30.686821 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11709 n=519 ec=188 les/c 11485/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -5162> 2014-09-18 18:08:30.687152 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(11) epoch 11709) v5 -- ?+0 0x19f25660 con 0xcbecec0 -5161> 2014-09-18 18:08:30.687303 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11709) v5 -- ?+0 0xe1aa5a0 con 0x140f95a0 -5160> 2014-09-18 18:08:30.687348 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6854/39317 9 ==== pg_notify(2.78f(155),2.4c6(15) epoch 11709) v5 ==== 10834+0+0 (2480085222 0 0) 0x2b93c580 con 0x12b3aec0 -5159> 2014-09-18 18:08:30.687379 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11709:2.178(3)) v4 -- ?+0 0xe1adc00 con 0x1395ec60 -5158> 2014-09-18 18:08:30.687398 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_info(1 pgs e11709:2.178(3)) v4 -- ?+0 0xe1ac3a0 con 0x1395ef20 -5157> 2014-09-18 18:08:30.687365 7f4c9f251700 5 -- op tracker -- , seq: 85959, time: 2014-09-18 18:08:30.687156, event: header_read, op: pg_notify(2.78f(155),2.4c6(15) epoch 11709) -5156> 2014-09-18 18:08:30.687502 7f4c9f251700 5 -- op tracker -- , seq: 85959, time: 2014-09-18 18:08:30.687157, event: throttled, op: pg_notify(2.78f(155),2.4c6(15) epoch 11709) -5155> 2014-09-18 18:08:30.687509 7f4c9f251700 5 -- op tracker -- , seq: 85959, time: 2014-09-18 18:08:30.687330, event: all_read, op: pg_notify(2.78f(155),2.4c6(15) epoch 11709) -5154> 2014-09-18 18:08:30.687515 7f4c9f251700 5 -- op tracker -- , seq: 85959, time: 2014-09-18 18:08:30.687360, event: dispatched, op: pg_notify(2.78f(155),2.4c6(15) epoch 11709) -5153> 2014-09-18 18:08:30.687524 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x11174ac0 con 0x1395f080 -5152> 2014-09-18 18:08:30.687536 7f4c9f251700 5 -- op tracker -- , seq: 85959, time: 2014-09-18 18:08:30.687534, event: waiting_for_osdmap, op: pg_notify(2.78f(155),2.4c6(15) epoch 11709) -5151> 2014-09-18 18:08:30.687544 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- pg_info(1 pgs e11709:2.179(10)) v4 -- ?+0 0xe1abe00 con 0x1395f080 -5150> 2014-09-18 18:08:30.687548 7f4c9f251700 5 -- op tracker -- , seq: 85959, time: 2014-09-18 18:08:30.687546, event: started, op: pg_notify(2.78f(155),2.4c6(15) epoch 11709) -5149> 2014-09-18 18:08:30.687597 7f4c9f251700 5 -- op tracker -- , seq: 85959, time: 2014-09-18 18:08:30.687593, event: done, op: pg_notify(2.78f(155),2.4c6(15) epoch 11709) -5148> 2014-09-18 18:08:30.687835 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.958436 5 0.000256 -5147> 2014-09-18 18:08:30.687911 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.964393 0 0.000000 -5146> 2014-09-18 18:08:30.687934 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11706 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -5145> 2014-09-18 18:08:30.688104 7f4c97241700 0 log_channel(default) log [INF] : 2.410 restarting backfill on osd.56 from (0'0,0'0] MAX to 6840'31228 -5144> 2014-09-18 18:08:30.688171 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.099490 3 0.000933 -5143> 2014-09-18 18:08:30.688197 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.966540 0 0.000000 -5142> 2014-09-18 18:08:30.688216 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11507 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -5141> 2014-09-18 18:08:30.688533 7f4c96a40700 0 log_channel(default) log [INF] : 2.11b restarting backfill on osd.9 from (0'0,0'0] MAX to 6775'28078 -5140> 2014-09-18 18:08:30.688547 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0xd7c4d00 con 0x14358260 -5139> 2014-09-18 18:08:30.688561 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- pg_log(2.410 epoch 11709 log log((4052'27523,6840'31228], crt=6840'31228) query_epoch 11709) v4 -- ?+0 0x15e8d500 con 0x14358260 -5138> 2014-09-18 18:08:30.688592 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11709 n=527 ec=188 les/c 11706/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -5137> 2014-09-18 18:08:30.689711 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 934 ==== pg_notify(2.11b(27) epoch 11709) v5 ==== 2247+0+0 (123940443 0 0) 0xf4e9660 con 0x1395f8c0 -5136> 2014-09-18 18:08:30.689734 7f4c9f251700 5 -- op tracker -- , seq: 85960, time: 2014-09-18 18:08:30.689629, event: header_read, op: pg_notify(2.11b(27) epoch 11709) -5135> 2014-09-18 18:08:30.689748 7f4c9f251700 5 -- op tracker -- , seq: 85960, time: 2014-09-18 18:08:30.689630, event: throttled, op: pg_notify(2.11b(27) epoch 11709) -5134> 2014-09-18 18:08:30.689757 7f4c9f251700 5 -- op tracker -- , seq: 85960, time: 2014-09-18 18:08:30.689695, event: all_read, op: pg_notify(2.11b(27) epoch 11709) -5133> 2014-09-18 18:08:30.689763 7f4c9f251700 5 -- op tracker -- , seq: 85960, time: 2014-09-18 18:08:30.689729, event: dispatched, op: pg_notify(2.11b(27) epoch 11709) -5132> 2014-09-18 18:08:30.689770 7f4c9f251700 5 -- op tracker -- , seq: 85960, time: 2014-09-18 18:08:30.689768, event: waiting_for_osdmap, op: pg_notify(2.11b(27) epoch 11709) -5131> 2014-09-18 18:08:30.689778 7f4c9f251700 5 -- op tracker -- , seq: 85960, time: 2014-09-18 18:08:30.689776, event: started, op: pg_notify(2.11b(27) epoch 11709) -5130> 2014-09-18 18:08:30.691103 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_log(2.11b epoch 11709 log log((4153'25079,6775'28078], crt=6775'28075) query_epoch 11709) v4 -- ?+0 0x21fad800 con 0x1395ec60 -5129> 2014-09-18 18:08:30.691327 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11709 n=460 ec=188 les/c 11507/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -5128> 2014-09-18 18:08:30.693511 7f4c9f251700 5 -- op tracker -- , seq: 85960, time: 2014-09-18 18:08:30.693507, event: done, op: pg_notify(2.11b(27) epoch 11709) -5127> 2014-09-18 18:08:30.693531 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.30 10.242.42.176:6829/1204 111 ==== pg_info(1 pgs e11709:2.6f8) v4 ==== 738+0+0 (2855313152 0 0) 0x121150c0 con 0x1395edc0 -5126> 2014-09-18 18:08:30.693543 7f4c9f251700 5 -- op tracker -- , seq: 85961, time: 2014-09-18 18:08:30.690441, event: header_read, op: pg_info(1 pgs e11709:2.6f8) -5125> 2014-09-18 18:08:30.693550 7f4c9f251700 5 -- op tracker -- , seq: 85961, time: 2014-09-18 18:08:30.690442, event: throttled, op: pg_info(1 pgs e11709:2.6f8) -5124> 2014-09-18 18:08:30.693556 7f4c9f251700 5 -- op tracker -- , seq: 85961, time: 2014-09-18 18:08:30.690466, event: all_read, op: pg_info(1 pgs e11709:2.6f8) -5123> 2014-09-18 18:08:30.693562 7f4c9f251700 5 -- op tracker -- , seq: 85961, time: 2014-09-18 18:08:30.693540, event: dispatched, op: pg_info(1 pgs e11709:2.6f8) -5122> 2014-09-18 18:08:30.693568 7f4c9f251700 5 -- op tracker -- , seq: 85961, time: 2014-09-18 18:08:30.693567, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.6f8) -5121> 2014-09-18 18:08:30.693576 7f4c9f251700 5 -- op tracker -- , seq: 85961, time: 2014-09-18 18:08:30.693575, event: started, op: pg_info(1 pgs e11709:2.6f8) -5120> 2014-09-18 18:08:30.693599 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.67b(4) epoch 11709) v5 -- ?+0 0x24fd3660 con 0x1395ec60 -5119> 2014-09-18 18:08:30.693619 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x1e284880 con 0x23a74f20 -5118> 2014-09-18 18:08:30.693629 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11709) v5 -- ?+0 0x24fd21c0 con 0x23a74f20 -5117> 2014-09-18 18:08:30.693670 7f4c9f251700 5 -- op tracker -- , seq: 85961, time: 2014-09-18 18:08:30.693668, event: done, op: pg_info(1 pgs e11709:2.6f8) -5116> 2014-09-18 18:08:30.693726 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x1e282b40 con 0x1395f8c0 -5115> 2014-09-18 18:08:30.693744 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11709:2.410(12)) v4 -- ?+0 0x24fd1c20 con 0x1395f8c0 -5114> 2014-09-18 18:08:30.693800 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] exit Started/Stray 0.974620 7 0.000367 -5113> 2014-09-18 18:08:30.693822 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] enter Started/ReplicaActive -5112> 2014-09-18 18:08:30.693835 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11702 n=512 ec=188 les/c 11702/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 crt=6789'26114 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -5111> 2014-09-18 18:08:30.693848 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x22e16900 con 0x1395f8c0 -5110> 2014-09-18 18:08:30.693900 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11709:2.11b(27)) v4 -- ?+0 0xf195a20 con 0x1395f8c0 -5109> 2014-09-18 18:08:30.694140 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11485/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started/Stray 0.973611 7 0.000372 -5108> 2014-09-18 18:08:30.694165 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11485/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -5107> 2014-09-18 18:08:30.694175 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.352( v 6785'33670 (4157'30669,6785'33670] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11485/10662 11708/11708/11483) [32,42,47]/[32,46] r=-1 lpr=11708 pi=10661-11707/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -5106> 2014-09-18 18:08:30.694614 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_info(1 pgs e11709:2.756) v4 -- ?+0 0xec6b480 con 0x140f95a0 -5105> 2014-09-18 18:08:30.694853 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_info(1 pgs e11709:2.352) v4 -- ?+0 0x2b941480 con 0x1395ef20 -5104> 2014-09-18 18:08:30.694975 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 535 ==== pg_notify(2.4d0(10) epoch 11709) v5 ==== 1292+0+0 (1674185328 0 0) 0xe1ac3a0 con 0x1395ef20 -5103> 2014-09-18 18:08:30.694991 7f4c9f251700 5 -- op tracker -- , seq: 85962, time: 2014-09-18 18:08:30.694927, event: header_read, op: pg_notify(2.4d0(10) epoch 11709) -5102> 2014-09-18 18:08:30.695000 7f4c9f251700 5 -- op tracker -- , seq: 85962, time: 2014-09-18 18:08:30.694927, event: throttled, op: pg_notify(2.4d0(10) epoch 11709) -5101> 2014-09-18 18:08:30.695006 7f4c9f251700 5 -- op tracker -- , seq: 85962, time: 2014-09-18 18:08:30.694964, event: all_read, op: pg_notify(2.4d0(10) epoch 11709) -5100> 2014-09-18 18:08:30.695012 7f4c9f251700 5 -- op tracker -- , seq: 85962, time: 2014-09-18 18:08:30.694988, event: dispatched, op: pg_notify(2.4d0(10) epoch 11709) -5099> 2014-09-18 18:08:30.695019 7f4c9f251700 5 -- op tracker -- , seq: 85962, time: 2014-09-18 18:08:30.695017, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11709) -5098> 2014-09-18 18:08:30.695026 7f4c9f251700 5 -- op tracker -- , seq: 85962, time: 2014-09-18 18:08:30.695025, event: started, op: pg_notify(2.4d0(10) epoch 11709) -5097> 2014-09-18 18:08:30.695043 7f4c9f251700 5 -- op tracker -- , seq: 85962, time: 2014-09-18 18:08:30.695042, event: done, op: pg_notify(2.4d0(10) epoch 11709) -5096> 2014-09-18 18:08:30.698491 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 536 ==== pg_info(1 pgs e11709:2.178) v4 ==== 738+0+0 (3157995032 0 0) 0x2b941480 con 0x1395ef20 -5095> 2014-09-18 18:08:30.698506 7f4c9f251700 5 -- op tracker -- , seq: 85963, time: 2014-09-18 18:08:30.698455, event: header_read, op: pg_info(1 pgs e11709:2.178) -5094> 2014-09-18 18:08:30.698515 7f4c9f251700 5 -- op tracker -- , seq: 85963, time: 2014-09-18 18:08:30.698456, event: throttled, op: pg_info(1 pgs e11709:2.178) -5093> 2014-09-18 18:08:30.698521 7f4c9f251700 5 -- op tracker -- , seq: 85963, time: 2014-09-18 18:08:30.698477, event: all_read, op: pg_info(1 pgs e11709:2.178) -5092> 2014-09-18 18:08:30.698527 7f4c9f251700 5 -- op tracker -- , seq: 85963, time: 2014-09-18 18:08:30.698502, event: dispatched, op: pg_info(1 pgs e11709:2.178) -5091> 2014-09-18 18:08:30.698534 7f4c9f251700 5 -- op tracker -- , seq: 85963, time: 2014-09-18 18:08:30.698532, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.178) -5090> 2014-09-18 18:08:30.698550 7f4c9f251700 5 -- op tracker -- , seq: 85963, time: 2014-09-18 18:08:30.698548, event: started, op: pg_info(1 pgs e11709:2.178) -5089> 2014-09-18 18:08:30.698568 7f4c9f251700 5 -- op tracker -- , seq: 85963, time: 2014-09-18 18:08:30.698566, event: done, op: pg_info(1 pgs e11709:2.178) -5088> 2014-09-18 18:08:30.705835 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 123 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x11174ac0 con 0x1395f080 -5087> 2014-09-18 18:08:30.705849 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5086> 2014-09-18 18:08:30.705915 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 124 ==== pg_notify(2.179(81) epoch 11709) v5 ==== 5289+0+0 (3998831481 0 0) 0xe1abe00 con 0x1395f080 -5085> 2014-09-18 18:08:30.705926 7f4c9f251700 5 -- op tracker -- , seq: 85964, time: 2014-09-18 18:08:30.705834, event: header_read, op: pg_notify(2.179(81) epoch 11709) -5084> 2014-09-18 18:08:30.705945 7f4c9f251700 5 -- op tracker -- , seq: 85964, time: 2014-09-18 18:08:30.705834, event: throttled, op: pg_notify(2.179(81) epoch 11709) -5083> 2014-09-18 18:08:30.705950 7f4c9f251700 5 -- op tracker -- , seq: 85964, time: 2014-09-18 18:08:30.705904, event: all_read, op: pg_notify(2.179(81) epoch 11709) -5082> 2014-09-18 18:08:30.705956 7f4c9f251700 5 -- op tracker -- , seq: 85964, time: 2014-09-18 18:08:30.705923, event: dispatched, op: pg_notify(2.179(81) epoch 11709) -5081> 2014-09-18 18:08:30.705963 7f4c9f251700 5 -- op tracker -- , seq: 85964, time: 2014-09-18 18:08:30.705961, event: waiting_for_osdmap, op: pg_notify(2.179(81) epoch 11709) -5080> 2014-09-18 18:08:30.705970 7f4c9f251700 5 -- op tracker -- , seq: 85964, time: 2014-09-18 18:08:30.705969, event: started, op: pg_notify(2.179(81) epoch 11709) -5079> 2014-09-18 18:08:30.705986 7f4c9f251700 5 -- op tracker -- , seq: 85964, time: 2014-09-18 18:08:30.705985, event: done, op: pg_notify(2.179(81) epoch 11709) -5078> 2014-09-18 18:08:30.707141 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.27 10.242.42.174:0/21548 149 ==== osd_ping(ping e11709 stamp 2014-09-18 18:06:36.538063) v2 ==== 47+0+0 (3957672296 0 0) 0x13041e00 con 0x107358c0 -5077> 2014-09-18 18:08:30.707168 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.174:0/21548 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:06:36.538063) v2 -- ?+0 0xf7a6960 con 0x107358c0 -5076> 2014-09-18 18:08:30.707206 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.27 10.242.42.174:0/21548 149 ==== osd_ping(ping e11709 stamp 2014-09-18 18:06:36.538063) v2 ==== 47+0+0 (3957672296 0 0) 0xf273860 con 0x101cb4a0 -5075> 2014-09-18 18:08:30.707231 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.174:0/21548 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:06:36.538063) v2 -- ?+0 0xe0bba40 con 0x101cb4a0 -5074> 2014-09-18 18:08:30.707288 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 125 ==== pg_notify(2.519(4) epoch 11709) v5 ==== 962+0+0 (676894838 0 0) 0x2b926f00 con 0x1395f080 -5073> 2014-09-18 18:08:30.707310 7f4c9f251700 5 -- op tracker -- , seq: 85965, time: 2014-09-18 18:08:30.707224, event: header_read, op: pg_notify(2.519(4) epoch 11709) -5072> 2014-09-18 18:08:30.707320 7f4c9f251700 5 -- op tracker -- , seq: 85965, time: 2014-09-18 18:08:30.707225, event: throttled, op: pg_notify(2.519(4) epoch 11709) -5071> 2014-09-18 18:08:30.707327 7f4c9f251700 5 -- op tracker -- , seq: 85965, time: 2014-09-18 18:08:30.707276, event: all_read, op: pg_notify(2.519(4) epoch 11709) -5070> 2014-09-18 18:08:30.707332 7f4c9f251700 5 -- op tracker -- , seq: 85965, time: 2014-09-18 18:08:30.707305, event: dispatched, op: pg_notify(2.519(4) epoch 11709) -5069> 2014-09-18 18:08:30.707339 7f4c9f251700 5 -- op tracker -- , seq: 85965, time: 2014-09-18 18:08:30.707338, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11709) -5068> 2014-09-18 18:08:30.707347 7f4c9f251700 5 -- op tracker -- , seq: 85965, time: 2014-09-18 18:08:30.707346, event: started, op: pg_notify(2.519(4) epoch 11709) -5067> 2014-09-18 18:08:30.707366 7f4c9f251700 5 -- op tracker -- , seq: 85965, time: 2014-09-18 18:08:30.707365, event: done, op: pg_notify(2.519(4) epoch 11709) -5066> 2014-09-18 18:08:30.708072 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 126 ==== pg_notify(2.4e8(37) epoch 11709) v5 ==== 2849+0+0 (3996605728 0 0) 0x21e11860 con 0x1395f080 -5065> 2014-09-18 18:08:30.708090 7f4c9f251700 5 -- op tracker -- , seq: 85966, time: 2014-09-18 18:08:30.708011, event: header_read, op: pg_notify(2.4e8(37) epoch 11709) -5064> 2014-09-18 18:08:30.708101 7f4c9f251700 5 -- op tracker -- , seq: 85966, time: 2014-09-18 18:08:30.708012, event: throttled, op: pg_notify(2.4e8(37) epoch 11709) -5063> 2014-09-18 18:08:30.708107 7f4c9f251700 5 -- op tracker -- , seq: 85966, time: 2014-09-18 18:08:30.708058, event: all_read, op: pg_notify(2.4e8(37) epoch 11709) -5062> 2014-09-18 18:08:30.708113 7f4c9f251700 5 -- op tracker -- , seq: 85966, time: 2014-09-18 18:08:30.708086, event: dispatched, op: pg_notify(2.4e8(37) epoch 11709) -5061> 2014-09-18 18:08:30.708120 7f4c9f251700 5 -- op tracker -- , seq: 85966, time: 2014-09-18 18:08:30.708118, event: waiting_for_osdmap, op: pg_notify(2.4e8(37) epoch 11709) -5060> 2014-09-18 18:08:30.708128 7f4c9f251700 5 -- op tracker -- , seq: 85966, time: 2014-09-18 18:08:30.708126, event: started, op: pg_notify(2.4e8(37) epoch 11709) -5059> 2014-09-18 18:08:30.708146 7f4c9f251700 5 -- op tracker -- , seq: 85966, time: 2014-09-18 18:08:30.708144, event: done, op: pg_notify(2.4e8(37) epoch 11709) -5058> 2014-09-18 18:08:30.712508 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 127 ==== pg_info(1 pgs e11709:2.179) v4 ==== 738+0+0 (880307379 0 0) 0x21e103c0 con 0x1395f080 -5057> 2014-09-18 18:08:30.712527 7f4c9f251700 5 -- op tracker -- , seq: 85967, time: 2014-09-18 18:08:30.712444, event: header_read, op: pg_info(1 pgs e11709:2.179) -5056> 2014-09-18 18:08:30.712542 7f4c9f251700 5 -- op tracker -- , seq: 85967, time: 2014-09-18 18:08:30.712446, event: throttled, op: pg_info(1 pgs e11709:2.179) -5055> 2014-09-18 18:08:30.712548 7f4c9f251700 5 -- op tracker -- , seq: 85967, time: 2014-09-18 18:08:30.712473, event: all_read, op: pg_info(1 pgs e11709:2.179) -5054> 2014-09-18 18:08:30.712554 7f4c9f251700 5 -- op tracker -- , seq: 85967, time: 2014-09-18 18:08:30.712522, event: dispatched, op: pg_info(1 pgs e11709:2.179) -5053> 2014-09-18 18:08:30.712560 7f4c9f251700 5 -- op tracker -- , seq: 85967, time: 2014-09-18 18:08:30.712558, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.179) -5052> 2014-09-18 18:08:30.712568 7f4c9f251700 5 -- op tracker -- , seq: 85967, time: 2014-09-18 18:08:30.712567, event: started, op: pg_info(1 pgs e11709:2.179) -5051> 2014-09-18 18:08:30.712585 7f4c9f251700 5 -- op tracker -- , seq: 85967, time: 2014-09-18 18:08:30.712584, event: done, op: pg_info(1 pgs e11709:2.179) -5050> 2014-09-18 18:08:30.716828 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 935 ==== pg_info(1 pgs e11709:2.410) v4 ==== 734+0+0 (2510772291 0 0) 0xf195a20 con 0x1395f8c0 -5049> 2014-09-18 18:08:30.716846 7f4c9f251700 5 -- op tracker -- , seq: 85968, time: 2014-09-18 18:08:30.716767, event: header_read, op: pg_info(1 pgs e11709:2.410) -5048> 2014-09-18 18:08:30.716860 7f4c9f251700 5 -- op tracker -- , seq: 85968, time: 2014-09-18 18:08:30.716768, event: throttled, op: pg_info(1 pgs e11709:2.410) -5047> 2014-09-18 18:08:30.716867 7f4c9f251700 5 -- op tracker -- , seq: 85968, time: 2014-09-18 18:08:30.716814, event: all_read, op: pg_info(1 pgs e11709:2.410) -5046> 2014-09-18 18:08:30.716873 7f4c9f251700 5 -- op tracker -- , seq: 85968, time: 2014-09-18 18:08:30.716842, event: dispatched, op: pg_info(1 pgs e11709:2.410) -5045> 2014-09-18 18:08:30.716879 7f4c9f251700 5 -- op tracker -- , seq: 85968, time: 2014-09-18 18:08:30.716878, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.410) -5044> 2014-09-18 18:08:30.716887 7f4c9f251700 5 -- op tracker -- , seq: 85968, time: 2014-09-18 18:08:30.716886, event: started, op: pg_info(1 pgs e11709:2.410) -5043> 2014-09-18 18:08:30.716913 7f4c9f251700 5 -- op tracker -- , seq: 85968, time: 2014-09-18 18:08:30.716911, event: done, op: pg_info(1 pgs e11709:2.410) -5042> 2014-09-18 18:08:30.716925 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 936 ==== pg_info(1 pgs e11709:2.11b) v4 ==== 734+0+0 (2451371661 0 0) 0x24fd1c20 con 0x1395f8c0 -5041> 2014-09-18 18:08:30.716934 7f4c9f251700 5 -- op tracker -- , seq: 85969, time: 2014-09-18 18:08:30.716827, event: header_read, op: pg_info(1 pgs e11709:2.11b) -5040> 2014-09-18 18:08:30.716941 7f4c9f251700 5 -- op tracker -- , seq: 85969, time: 2014-09-18 18:08:30.716828, event: throttled, op: pg_info(1 pgs e11709:2.11b) -5039> 2014-09-18 18:08:30.716947 7f4c9f251700 5 -- op tracker -- , seq: 85969, time: 2014-09-18 18:08:30.716846, event: all_read, op: pg_info(1 pgs e11709:2.11b) -5038> 2014-09-18 18:08:30.716953 7f4c9f251700 5 -- op tracker -- , seq: 85969, time: 2014-09-18 18:08:30.716932, event: dispatched, op: pg_info(1 pgs e11709:2.11b) -5037> 2014-09-18 18:08:30.716959 7f4c9f251700 5 -- op tracker -- , seq: 85969, time: 2014-09-18 18:08:30.716958, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.11b) -5036> 2014-09-18 18:08:30.716993 7f4c9f251700 5 -- op tracker -- , seq: 85969, time: 2014-09-18 18:08:30.716992, event: started, op: pg_info(1 pgs e11709:2.11b) -5035> 2014-09-18 18:08:30.717013 7f4c9f251700 5 -- op tracker -- , seq: 85969, time: 2014-09-18 18:08:30.717011, event: done, op: pg_info(1 pgs e11709:2.11b) -5034> 2014-09-18 18:08:30.718878 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1724 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x1533c640 con 0x4900000 -5033> 2014-09-18 18:08:30.718896 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -5032> 2014-09-18 18:08:30.719611 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1725 ==== pg_notify(2.e0(79) epoch 11709) v5 ==== 5119+0+0 (4262343717 0 0) 0xf1eb680 con 0x4900000 -5031> 2014-09-18 18:08:30.719628 7f4c9f251700 5 -- op tracker -- , seq: 85970, time: 2014-09-18 18:08:30.719515, event: header_read, op: pg_notify(2.e0(79) epoch 11709) -5030> 2014-09-18 18:08:30.719638 7f4c9f251700 5 -- op tracker -- , seq: 85970, time: 2014-09-18 18:08:30.719516, event: throttled, op: pg_notify(2.e0(79) epoch 11709) -5029> 2014-09-18 18:08:30.719644 7f4c9f251700 5 -- op tracker -- , seq: 85970, time: 2014-09-18 18:08:30.719597, event: all_read, op: pg_notify(2.e0(79) epoch 11709) -5028> 2014-09-18 18:08:30.719650 7f4c9f251700 5 -- op tracker -- , seq: 85970, time: 2014-09-18 18:08:30.719624, event: dispatched, op: pg_notify(2.e0(79) epoch 11709) -5027> 2014-09-18 18:08:30.719657 7f4c9f251700 5 -- op tracker -- , seq: 85970, time: 2014-09-18 18:08:30.719655, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11709) -5026> 2014-09-18 18:08:30.719664 7f4c9f251700 5 -- op tracker -- , seq: 85970, time: 2014-09-18 18:08:30.719663, event: started, op: pg_notify(2.e0(79) epoch 11709) -5025> 2014-09-18 18:08:30.719847 7f4c9f251700 5 -- op tracker -- , seq: 85970, time: 2014-09-18 18:08:30.719845, event: done, op: pg_notify(2.e0(79) epoch 11709) -5024> 2014-09-18 18:08:30.732611 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 962 ==== pg_notify(2.517(11) epoch 11708) v5 ==== 1335+0+0 (2124541876 0 0) 0x162510c0 con 0x10a878c0 -5023> 2014-09-18 18:08:30.732651 7f4c9f251700 5 -- op tracker -- , seq: 85971, time: 2014-09-18 18:08:30.732527, event: header_read, op: pg_notify(2.517(11) epoch 11708) -5022> 2014-09-18 18:08:30.732666 7f4c9f251700 5 -- op tracker -- , seq: 85971, time: 2014-09-18 18:08:30.732529, event: throttled, op: pg_notify(2.517(11) epoch 11708) -5021> 2014-09-18 18:08:30.732675 7f4c9f251700 5 -- op tracker -- , seq: 85971, time: 2014-09-18 18:08:30.732587, event: all_read, op: pg_notify(2.517(11) epoch 11708) -5020> 2014-09-18 18:08:30.732692 7f4c9f251700 5 -- op tracker -- , seq: 85971, time: 2014-09-18 18:08:30.732644, event: dispatched, op: pg_notify(2.517(11) epoch 11708) -5019> 2014-09-18 18:08:30.732702 7f4c9f251700 5 -- op tracker -- , seq: 85971, time: 2014-09-18 18:08:30.732700, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11708) -5018> 2014-09-18 18:08:30.732715 7f4c9f251700 5 -- op tracker -- , seq: 85971, time: 2014-09-18 18:08:30.732713, event: started, op: pg_notify(2.517(11) epoch 11708) -5017> 2014-09-18 18:08:30.732743 7f4c9f251700 5 -- op tracker -- , seq: 85971, time: 2014-09-18 18:08:30.732741, event: done, op: pg_notify(2.517(11) epoch 11708) -5016> 2014-09-18 18:08:30.734689 7f4caae41700 5 osd.42 11709 tick -5015> 2014-09-18 18:08:30.737916 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1041 ==== pg_info(1 pgs e11709:2.410) v4 ==== 718+0+0 (2122670793 0 0) 0xf72ed20 con 0x14358260 -5014> 2014-09-18 18:08:30.737939 7f4c9f251700 5 -- op tracker -- , seq: 85972, time: 2014-09-18 18:08:30.737832, event: header_read, op: pg_info(1 pgs e11709:2.410) -5013> 2014-09-18 18:08:30.737948 7f4c9f251700 5 -- op tracker -- , seq: 85972, time: 2014-09-18 18:08:30.737833, event: throttled, op: pg_info(1 pgs e11709:2.410) -5012> 2014-09-18 18:08:30.737952 7f4c9f251700 5 -- op tracker -- , seq: 85972, time: 2014-09-18 18:08:30.737867, event: all_read, op: pg_info(1 pgs e11709:2.410) -5011> 2014-09-18 18:08:30.737956 7f4c9f251700 5 -- op tracker -- , seq: 85972, time: 2014-09-18 18:08:30.737933, event: dispatched, op: pg_info(1 pgs e11709:2.410) -5010> 2014-09-18 18:08:30.737960 7f4c9f251700 5 -- op tracker -- , seq: 85972, time: 2014-09-18 18:08:30.737959, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.410) -5009> 2014-09-18 18:08:30.737976 7f4c9f251700 5 -- op tracker -- , seq: 85972, time: 2014-09-18 18:08:30.737975, event: started, op: pg_info(1 pgs e11709:2.410) -5008> 2014-09-18 18:08:30.737997 7f4c9f251700 5 -- op tracker -- , seq: 85972, time: 2014-09-18 18:08:30.737996, event: done, op: pg_info(1 pgs e11709:2.410) -5007> 2014-09-18 18:08:30.738033 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11709:2.410) v4 -- ?+0 0xddf2f00 con 0x1395f8c0 -5006> 2014-09-18 18:08:30.738166 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- pg_info(1 pgs e11709:2.410) v4 -- ?+0 0xf815660 con 0x14358260 -5005> 2014-09-18 18:08:30.738340 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11709 n=527 ec=188 les/c 11709/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=27,56 from 0//0//-1 -5004> 2014-09-18 18:08:30.738372 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11709 n=527 ec=188 les/c 11709/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 27 from 0//0//-1 -5003> 2014-09-18 18:08:30.738445 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11709 n=527 ec=188 les/c 11709/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 56 from 0//0//-1 -5002> 2014-09-18 18:08:30.738465 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11709 n=527 ec=188 les/c 11709/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 0.049873 7 0.001213 -5001> 2014-09-18 18:08:30.738480 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.410( v 6840'31228 (4052'27523,6840'31228] local-les=11709 n=527 ec=188 les/c 11709/9946 11708/11708/9945) [42,56,27]/[42] r=0 lpr=11708 pi=9945-11707/12 bft=27,56 crt=6840'31228 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -5000> 2014-09-18 18:08:30.741060 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1842 ==== pg_notify(2.4f6(15) epoch 11707) v5 ==== 1581+0+0 (199295532 0 0) 0x21d15660 con 0x10f60520 -4999> 2014-09-18 18:08:30.741091 7f4c9f251700 5 -- op tracker -- , seq: 85973, time: 2014-09-18 18:08:30.740955, event: header_read, op: pg_notify(2.4f6(15) epoch 11707) -4998> 2014-09-18 18:08:30.741104 7f4c9f251700 5 -- op tracker -- , seq: 85973, time: 2014-09-18 18:08:30.740957, event: throttled, op: pg_notify(2.4f6(15) epoch 11707) -4997> 2014-09-18 18:08:30.741114 7f4c9f251700 5 -- op tracker -- , seq: 85973, time: 2014-09-18 18:08:30.741019, event: all_read, op: pg_notify(2.4f6(15) epoch 11707) -4996> 2014-09-18 18:08:30.741134 7f4c9f251700 5 -- op tracker -- , seq: 85973, time: 2014-09-18 18:08:30.741084, event: dispatched, op: pg_notify(2.4f6(15) epoch 11707) -4995> 2014-09-18 18:08:30.741146 7f4c9f251700 5 -- op tracker -- , seq: 85973, time: 2014-09-18 18:08:30.741143, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11707) -4994> 2014-09-18 18:08:30.741159 7f4c9f251700 5 -- op tracker -- , seq: 85973, time: 2014-09-18 18:08:30.741157, event: started, op: pg_notify(2.4f6(15) epoch 11707) -4993> 2014-09-18 18:08:30.741185 7f4c9f251700 5 -- op tracker -- , seq: 85973, time: 2014-09-18 18:08:30.741183, event: done, op: pg_notify(2.4f6(15) epoch 11707) -4992> 2014-09-18 18:08:30.743210 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1843 ==== pg_notify(2.517(11) epoch 11707) v5 ==== 1335+0+0 (866778327 0 0) 0x21d14760 con 0x10f60520 -4991> 2014-09-18 18:08:30.743231 7f4c9f251700 5 -- op tracker -- , seq: 85974, time: 2014-09-18 18:08:30.743146, event: header_read, op: pg_notify(2.517(11) epoch 11707) -4990> 2014-09-18 18:08:30.743240 7f4c9f251700 5 -- op tracker -- , seq: 85974, time: 2014-09-18 18:08:30.743147, event: throttled, op: pg_notify(2.517(11) epoch 11707) -4989> 2014-09-18 18:08:30.743246 7f4c9f251700 5 -- op tracker -- , seq: 85974, time: 2014-09-18 18:08:30.743197, event: all_read, op: pg_notify(2.517(11) epoch 11707) -4988> 2014-09-18 18:08:30.743253 7f4c9f251700 5 -- op tracker -- , seq: 85974, time: 2014-09-18 18:08:30.743228, event: dispatched, op: pg_notify(2.517(11) epoch 11707) -4987> 2014-09-18 18:08:30.743260 7f4c9f251700 5 -- op tracker -- , seq: 85974, time: 2014-09-18 18:08:30.743258, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11707) -4986> 2014-09-18 18:08:30.743267 7f4c9f251700 5 -- op tracker -- , seq: 85974, time: 2014-09-18 18:08:30.743266, event: started, op: pg_notify(2.517(11) epoch 11707) -4985> 2014-09-18 18:08:30.743294 7f4c9f251700 5 -- op tracker -- , seq: 85974, time: 2014-09-18 18:08:30.743293, event: done, op: pg_notify(2.517(11) epoch 11707) -4984> 2014-09-18 18:08:30.744154 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1844 ==== pg_notify(2.1d8(9) epoch 11707) v5 ==== 1261+0+0 (1785722038 0 0) 0xf12a1e0 con 0x10f60520 -4983> 2014-09-18 18:08:30.744169 7f4c9f251700 5 -- op tracker -- , seq: 85975, time: 2014-09-18 18:08:30.744091, event: header_read, op: pg_notify(2.1d8(9) epoch 11707) -4982> 2014-09-18 18:08:30.744177 7f4c9f251700 5 -- op tracker -- , seq: 85975, time: 2014-09-18 18:08:30.744092, event: throttled, op: pg_notify(2.1d8(9) epoch 11707) -4981> 2014-09-18 18:08:30.744183 7f4c9f251700 5 -- op tracker -- , seq: 85975, time: 2014-09-18 18:08:30.744142, event: all_read, op: pg_notify(2.1d8(9) epoch 11707) -4980> 2014-09-18 18:08:30.744189 7f4c9f251700 5 -- op tracker -- , seq: 85975, time: 2014-09-18 18:08:30.744165, event: dispatched, op: pg_notify(2.1d8(9) epoch 11707) -4979> 2014-09-18 18:08:30.744195 7f4c9f251700 5 -- op tracker -- , seq: 85975, time: 2014-09-18 18:08:30.744193, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11707) -4978> 2014-09-18 18:08:30.744202 7f4c9f251700 5 -- op tracker -- , seq: 85975, time: 2014-09-18 18:08:30.744201, event: started, op: pg_notify(2.1d8(9) epoch 11707) -4977> 2014-09-18 18:08:30.744229 7f4c9f251700 5 -- op tracker -- , seq: 85975, time: 2014-09-18 18:08:30.744227, event: done, op: pg_notify(2.1d8(9) epoch 11707) -4976> 2014-09-18 18:08:30.745694 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1845 ==== osd_map(11708..11708 src has 1..11708) v3 ==== 4044+0+0 (1164821003 0 0) 0xe2efa80 con 0x10f60520 -4975> 2014-09-18 18:08:30.745714 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11708,11708], i have 11709, src has [1,11708] -4974> 2014-09-18 18:08:30.745736 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1846 ==== pg_notify(2.4f6(15) epoch 11708) v5 ==== 1581+0+0 (727873556 0 0) 0xf12dde0 con 0x10f60520 -4973> 2014-09-18 18:08:30.745750 7f4c9f251700 5 -- op tracker -- , seq: 85976, time: 2014-09-18 18:08:30.745697, event: header_read, op: pg_notify(2.4f6(15) epoch 11708) -4972> 2014-09-18 18:08:30.745759 7f4c9f251700 5 -- op tracker -- , seq: 85976, time: 2014-09-18 18:08:30.745698, event: throttled, op: pg_notify(2.4f6(15) epoch 11708) -4971> 2014-09-18 18:08:30.745767 7f4c9f251700 5 -- op tracker -- , seq: 85976, time: 2014-09-18 18:08:30.745724, event: all_read, op: pg_notify(2.4f6(15) epoch 11708) -4970> 2014-09-18 18:08:30.745775 7f4c9f251700 5 -- op tracker -- , seq: 85976, time: 2014-09-18 18:08:30.745747, event: dispatched, op: pg_notify(2.4f6(15) epoch 11708) -4969> 2014-09-18 18:08:30.745785 7f4c9f251700 5 -- op tracker -- , seq: 85976, time: 2014-09-18 18:08:30.745783, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11708) -4968> 2014-09-18 18:08:30.745799 7f4c9f251700 5 -- op tracker -- , seq: 85976, time: 2014-09-18 18:08:30.745796, event: started, op: pg_notify(2.4f6(15) epoch 11708) -4967> 2014-09-18 18:08:30.745819 7f4c9f251700 5 -- op tracker -- , seq: 85976, time: 2014-09-18 18:08:30.745818, event: done, op: pg_notify(2.4f6(15) epoch 11708) -4966> 2014-09-18 18:08:30.748031 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1847 ==== pg_notify(2.517(11) epoch 11708) v5 ==== 1335+0+0 (887477911 0 0) 0xf12c1c0 con 0x10f60520 -4965> 2014-09-18 18:08:30.748051 7f4c9f251700 5 -- op tracker -- , seq: 85977, time: 2014-09-18 18:08:30.747952, event: header_read, op: pg_notify(2.517(11) epoch 11708) -4964> 2014-09-18 18:08:30.748066 7f4c9f251700 5 -- op tracker -- , seq: 85977, time: 2014-09-18 18:08:30.747953, event: throttled, op: pg_notify(2.517(11) epoch 11708) -4963> 2014-09-18 18:08:30.748072 7f4c9f251700 5 -- op tracker -- , seq: 85977, time: 2014-09-18 18:08:30.747990, event: all_read, op: pg_notify(2.517(11) epoch 11708) -4962> 2014-09-18 18:08:30.748079 7f4c9f251700 5 -- op tracker -- , seq: 85977, time: 2014-09-18 18:08:30.748046, event: dispatched, op: pg_notify(2.517(11) epoch 11708) -4961> 2014-09-18 18:08:30.748086 7f4c9f251700 5 -- op tracker -- , seq: 85977, time: 2014-09-18 18:08:30.748084, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11708) -4960> 2014-09-18 18:08:30.748094 7f4c9f251700 5 -- op tracker -- , seq: 85977, time: 2014-09-18 18:08:30.748092, event: started, op: pg_notify(2.517(11) epoch 11708) -4959> 2014-09-18 18:08:30.748112 7f4c9f251700 5 -- op tracker -- , seq: 85977, time: 2014-09-18 18:08:30.748111, event: done, op: pg_notify(2.517(11) epoch 11708) -4958> 2014-09-18 18:08:30.748900 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1848 ==== pg_notify(2.1d8(9) epoch 11708) v5 ==== 1261+0+0 (2077326071 0 0) 0xf12ba40 con 0x10f60520 -4957> 2014-09-18 18:08:30.748919 7f4c9f251700 5 -- op tracker -- , seq: 85978, time: 2014-09-18 18:08:30.748849, event: header_read, op: pg_notify(2.1d8(9) epoch 11708) -4956> 2014-09-18 18:08:30.748930 7f4c9f251700 5 -- op tracker -- , seq: 85978, time: 2014-09-18 18:08:30.748850, event: throttled, op: pg_notify(2.1d8(9) epoch 11708) -4955> 2014-09-18 18:08:30.748936 7f4c9f251700 5 -- op tracker -- , seq: 85978, time: 2014-09-18 18:08:30.748886, event: all_read, op: pg_notify(2.1d8(9) epoch 11708) -4954> 2014-09-18 18:08:30.748942 7f4c9f251700 5 -- op tracker -- , seq: 85978, time: 2014-09-18 18:08:30.748915, event: dispatched, op: pg_notify(2.1d8(9) epoch 11708) -4953> 2014-09-18 18:08:30.748949 7f4c9f251700 5 -- op tracker -- , seq: 85978, time: 2014-09-18 18:08:30.748947, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11708) -4952> 2014-09-18 18:08:30.748957 7f4c9f251700 5 -- op tracker -- , seq: 85978, time: 2014-09-18 18:08:30.748955, event: started, op: pg_notify(2.1d8(9) epoch 11708) -4951> 2014-09-18 18:08:30.748976 7f4c9f251700 5 -- op tracker -- , seq: 85978, time: 2014-09-18 18:08:30.748975, event: done, op: pg_notify(2.1d8(9) epoch 11708) -4950> 2014-09-18 18:08:30.750757 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1849 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x1182d180 con 0x10f60520 -4949> 2014-09-18 18:08:30.750782 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -4948> 2014-09-18 18:08:30.750795 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1850 ==== pg_notify(2.4f6(15) epoch 11709) v5 ==== 1581+0+0 (4269548682 0 0) 0xf12ad20 con 0x10f60520 -4947> 2014-09-18 18:08:30.750808 7f4c9f251700 5 -- op tracker -- , seq: 85979, time: 2014-09-18 18:08:30.750726, event: header_read, op: pg_notify(2.4f6(15) epoch 11709) -4946> 2014-09-18 18:08:30.750819 7f4c9f251700 5 -- op tracker -- , seq: 85979, time: 2014-09-18 18:08:30.750727, event: throttled, op: pg_notify(2.4f6(15) epoch 11709) -4945> 2014-09-18 18:08:30.750829 7f4c9f251700 5 -- op tracker -- , seq: 85979, time: 2014-09-18 18:08:30.750767, event: all_read, op: pg_notify(2.4f6(15) epoch 11709) -4944> 2014-09-18 18:08:30.750835 7f4c9f251700 5 -- op tracker -- , seq: 85979, time: 2014-09-18 18:08:30.750804, event: dispatched, op: pg_notify(2.4f6(15) epoch 11709) -4943> 2014-09-18 18:08:30.750842 7f4c9f251700 5 -- op tracker -- , seq: 85979, time: 2014-09-18 18:08:30.750840, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11709) -4942> 2014-09-18 18:08:30.750850 7f4c9f251700 5 -- op tracker -- , seq: 85979, time: 2014-09-18 18:08:30.750848, event: started, op: pg_notify(2.4f6(15) epoch 11709) -4941> 2014-09-18 18:08:30.750869 7f4c9f251700 5 -- op tracker -- , seq: 85979, time: 2014-09-18 18:08:30.750867, event: done, op: pg_notify(2.4f6(15) epoch 11709) -4940> 2014-09-18 18:08:30.752832 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1851 ==== pg_notify(2.517(11) epoch 11709) v5 ==== 1335+0+0 (905735767 0 0) 0xf12d840 con 0x10f60520 -4939> 2014-09-18 18:08:30.752851 7f4c9f251700 5 -- op tracker -- , seq: 85980, time: 2014-09-18 18:08:30.752773, event: header_read, op: pg_notify(2.517(11) epoch 11709) -4938> 2014-09-18 18:08:30.752859 7f4c9f251700 5 -- op tracker -- , seq: 85980, time: 2014-09-18 18:08:30.752774, event: throttled, op: pg_notify(2.517(11) epoch 11709) -4937> 2014-09-18 18:08:30.752864 7f4c9f251700 5 -- op tracker -- , seq: 85980, time: 2014-09-18 18:08:30.752814, event: all_read, op: pg_notify(2.517(11) epoch 11709) -4936> 2014-09-18 18:08:30.752868 7f4c9f251700 5 -- op tracker -- , seq: 85980, time: 2014-09-18 18:08:30.752847, event: dispatched, op: pg_notify(2.517(11) epoch 11709) -4935> 2014-09-18 18:08:30.752872 7f4c9f251700 5 -- op tracker -- , seq: 85980, time: 2014-09-18 18:08:30.752871, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11709) -4934> 2014-09-18 18:08:30.752878 7f4c9f251700 5 -- op tracker -- , seq: 85980, time: 2014-09-18 18:08:30.752877, event: started, op: pg_notify(2.517(11) epoch 11709) -4933> 2014-09-18 18:08:30.752895 7f4c9f251700 5 -- op tracker -- , seq: 85980, time: 2014-09-18 18:08:30.752893, event: done, op: pg_notify(2.517(11) epoch 11709) -4932> 2014-09-18 18:08:30.753474 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1852 ==== pg_notify(2.1d8(9) epoch 11709) v5 ==== 1261+0+0 (2781272698 0 0) 0xf12a000 con 0x10f60520 -4931> 2014-09-18 18:08:30.753498 7f4c9f251700 5 -- op tracker -- , seq: 85981, time: 2014-09-18 18:08:30.753421, event: header_read, op: pg_notify(2.1d8(9) epoch 11709) -4930> 2014-09-18 18:08:30.753509 7f4c9f251700 5 -- op tracker -- , seq: 85981, time: 2014-09-18 18:08:30.753421, event: throttled, op: pg_notify(2.1d8(9) epoch 11709) -4929> 2014-09-18 18:08:30.753515 7f4c9f251700 5 -- op tracker -- , seq: 85981, time: 2014-09-18 18:08:30.753456, event: all_read, op: pg_notify(2.1d8(9) epoch 11709) -4928> 2014-09-18 18:08:30.753521 7f4c9f251700 5 -- op tracker -- , seq: 85981, time: 2014-09-18 18:08:30.753494, event: dispatched, op: pg_notify(2.1d8(9) epoch 11709) -4927> 2014-09-18 18:08:30.753527 7f4c9f251700 5 -- op tracker -- , seq: 85981, time: 2014-09-18 18:08:30.753526, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11709) -4926> 2014-09-18 18:08:30.753535 7f4c9f251700 5 -- op tracker -- , seq: 85981, time: 2014-09-18 18:08:30.753534, event: started, op: pg_notify(2.1d8(9) epoch 11709) -4925> 2014-09-18 18:08:30.753554 7f4c9f251700 5 -- op tracker -- , seq: 85981, time: 2014-09-18 18:08:30.753552, event: done, op: pg_notify(2.1d8(9) epoch 11709) -4924> 2014-09-18 18:08:30.775685 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.59 10.242.42.178:6840/4508 3 ==== pg_notify(2.4e8(7) epoch 11709) v5 ==== 1139+0+0 (3332230783 0 0) 0xf730ee0 con 0x23af51e0 -4923> 2014-09-18 18:08:30.775712 7f4c9f251700 5 -- op tracker -- , seq: 85982, time: 2014-09-18 18:08:30.775594, event: header_read, op: pg_notify(2.4e8(7) epoch 11709) -4922> 2014-09-18 18:08:30.775724 7f4c9f251700 5 -- op tracker -- , seq: 85982, time: 2014-09-18 18:08:30.775596, event: throttled, op: pg_notify(2.4e8(7) epoch 11709) -4921> 2014-09-18 18:08:30.775730 7f4c9f251700 5 -- op tracker -- , seq: 85982, time: 2014-09-18 18:08:30.775667, event: all_read, op: pg_notify(2.4e8(7) epoch 11709) -4920> 2014-09-18 18:08:30.775737 7f4c9f251700 5 -- op tracker -- , seq: 85982, time: 2014-09-18 18:08:30.775705, event: dispatched, op: pg_notify(2.4e8(7) epoch 11709) -4919> 2014-09-18 18:08:30.775744 7f4c9f251700 5 -- op tracker -- , seq: 85982, time: 2014-09-18 18:08:30.775742, event: waiting_for_osdmap, op: pg_notify(2.4e8(7) epoch 11709) -4918> 2014-09-18 18:08:30.775753 7f4c9f251700 5 -- op tracker -- , seq: 85982, time: 2014-09-18 18:08:30.775752, event: started, op: pg_notify(2.4e8(7) epoch 11709) -4917> 2014-09-18 18:08:30.775775 7f4c9f251700 5 -- op tracker -- , seq: 85982, time: 2014-09-18 18:08:30.775774, event: done, op: pg_notify(2.4e8(7) epoch 11709) -4916> 2014-09-18 18:08:30.824054 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3192 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x1440b840 con 0x1395ec60 -4915> 2014-09-18 18:08:30.824084 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -4914> 2014-09-18 18:08:30.824107 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3193 ==== pg_info(1 pgs e11709:2.13(3)) v4 ==== 903+0+0 (2090268922 0 0) 0x24fd3660 con 0x1395ec60 -4913> 2014-09-18 18:08:30.824120 7f4c9f251700 5 -- op tracker -- , seq: 85983, time: 2014-09-18 18:08:30.824053, event: header_read, op: pg_info(1 pgs e11709:2.13(3)) -4912> 2014-09-18 18:08:30.824132 7f4c9f251700 5 -- op tracker -- , seq: 85983, time: 2014-09-18 18:08:30.824053, event: throttled, op: pg_info(1 pgs e11709:2.13(3)) -4911> 2014-09-18 18:08:30.824140 7f4c9f251700 5 -- op tracker -- , seq: 85983, time: 2014-09-18 18:08:30.824088, event: all_read, op: pg_info(1 pgs e11709:2.13(3)) -4910> 2014-09-18 18:08:30.824147 7f4c9f251700 5 -- op tracker -- , seq: 85983, time: 2014-09-18 18:08:30.824115, event: dispatched, op: pg_info(1 pgs e11709:2.13(3)) -4909> 2014-09-18 18:08:30.824154 7f4c9f251700 5 -- op tracker -- , seq: 85983, time: 2014-09-18 18:08:30.824152, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.13(3)) -4908> 2014-09-18 18:08:30.824163 7f4c9f251700 5 -- op tracker -- , seq: 85983, time: 2014-09-18 18:08:30.824161, event: started, op: pg_info(1 pgs e11709:2.13(3)) -4907> 2014-09-18 18:08:30.824192 7f4c9f251700 5 -- op tracker -- , seq: 85983, time: 2014-09-18 18:08:30.824190, event: done, op: pg_info(1 pgs e11709:2.13(3)) -4906> 2014-09-18 18:08:30.824231 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] exit Started/Stray 1.107229 7 0.000922 -4905> 2014-09-18 18:08:30.824255 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -4904> 2014-09-18 18:08:30.824419 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.13( v 6725'30144 (4155'27145,6725'30144] local-les=11506 n=484 ec=188 les/c 11506/11491 11708/11708/11490) [20,42,27]/[9,42] r=1 lpr=11708 pi=11258-11707/6 crt=6705'30135 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -4903> 2014-09-18 18:08:30.824804 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3194 ==== pg_notify(2.71(19) epoch 11709) v5 ==== 1795+0+0 (559125272 0 0) 0xe1adc00 con 0x1395ec60 -4902> 2014-09-18 18:08:30.824816 7f4c9f251700 5 -- op tracker -- , seq: 85984, time: 2014-09-18 18:08:30.824742, event: header_read, op: pg_notify(2.71(19) epoch 11709) -4901> 2014-09-18 18:08:30.824826 7f4c9f251700 5 -- op tracker -- , seq: 85984, time: 2014-09-18 18:08:30.824744, event: throttled, op: pg_notify(2.71(19) epoch 11709) -4900> 2014-09-18 18:08:30.824833 7f4c9f251700 5 -- op tracker -- , seq: 85984, time: 2014-09-18 18:08:30.824791, event: all_read, op: pg_notify(2.71(19) epoch 11709) -4899> 2014-09-18 18:08:30.824840 7f4c9f251700 5 -- op tracker -- , seq: 85984, time: 2014-09-18 18:08:30.824813, event: dispatched, op: pg_notify(2.71(19) epoch 11709) -4898> 2014-09-18 18:08:30.824848 7f4c9f251700 5 -- op tracker -- , seq: 85984, time: 2014-09-18 18:08:30.824845, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11709) -4897> 2014-09-18 18:08:30.824858 7f4c9f251700 5 -- op tracker -- , seq: 85984, time: 2014-09-18 18:08:30.824855, event: started, op: pg_notify(2.71(19) epoch 11709) -4896> 2014-09-18 18:08:30.824880 7f4c9f251700 5 -- op tracker -- , seq: 85984, time: 2014-09-18 18:08:30.824878, event: done, op: pg_notify(2.71(19) epoch 11709) -4895> 2014-09-18 18:08:30.825002 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11709:2.13) v4 -- ?+0 0xf974b20 con 0x1395ec60 -4894> 2014-09-18 18:08:30.900970 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3195 ==== pg_notify(2.519(4) epoch 11709) v5 ==== 962+0+0 (676894838 0 0) 0xf974b20 con 0x1395ec60 -4893> 2014-09-18 18:08:30.900991 7f4c9f251700 5 -- op tracker -- , seq: 85985, time: 2014-09-18 18:08:30.900891, event: header_read, op: pg_notify(2.519(4) epoch 11709) -4892> 2014-09-18 18:08:30.901003 7f4c9f251700 5 -- op tracker -- , seq: 85985, time: 2014-09-18 18:08:30.900892, event: throttled, op: pg_notify(2.519(4) epoch 11709) -4891> 2014-09-18 18:08:30.901011 7f4c9f251700 5 -- op tracker -- , seq: 85985, time: 2014-09-18 18:08:30.900953, event: all_read, op: pg_notify(2.519(4) epoch 11709) -4890> 2014-09-18 18:08:30.901017 7f4c9f251700 5 -- op tracker -- , seq: 85985, time: 2014-09-18 18:08:30.900986, event: dispatched, op: pg_notify(2.519(4) epoch 11709) -4889> 2014-09-18 18:08:30.901023 7f4c9f251700 5 -- op tracker -- , seq: 85985, time: 2014-09-18 18:08:30.901022, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11709) -4888> 2014-09-18 18:08:30.901032 7f4c9f251700 5 -- op tracker -- , seq: 85985, time: 2014-09-18 18:08:30.901030, event: started, op: pg_notify(2.519(4) epoch 11709) -4887> 2014-09-18 18:08:30.901052 7f4c9f251700 5 -- op tracker -- , seq: 85985, time: 2014-09-18 18:08:30.901051, event: done, op: pg_notify(2.519(4) epoch 11709) -4886> 2014-09-18 18:08:30.903938 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3196 ==== pg_notify(2.178(24) epoch 11709) v5 ==== 1986+0+0 (3386332080 0 0) 0xf91d480 con 0x1395ec60 -4885> 2014-09-18 18:08:30.903959 7f4c9f251700 5 -- op tracker -- , seq: 85986, time: 2014-09-18 18:08:30.903872, event: header_read, op: pg_notify(2.178(24) epoch 11709) -4884> 2014-09-18 18:08:30.903970 7f4c9f251700 5 -- op tracker -- , seq: 85986, time: 2014-09-18 18:08:30.903873, event: throttled, op: pg_notify(2.178(24) epoch 11709) -4883> 2014-09-18 18:08:30.903976 7f4c9f251700 5 -- op tracker -- , seq: 85986, time: 2014-09-18 18:08:30.903923, event: all_read, op: pg_notify(2.178(24) epoch 11709) -4882> 2014-09-18 18:08:30.903982 7f4c9f251700 5 -- op tracker -- , seq: 85986, time: 2014-09-18 18:08:30.903954, event: dispatched, op: pg_notify(2.178(24) epoch 11709) -4881> 2014-09-18 18:08:30.903989 7f4c9f251700 5 -- op tracker -- , seq: 85986, time: 2014-09-18 18:08:30.903987, event: waiting_for_osdmap, op: pg_notify(2.178(24) epoch 11709) -4880> 2014-09-18 18:08:30.903997 7f4c9f251700 5 -- op tracker -- , seq: 85986, time: 2014-09-18 18:08:30.903995, event: started, op: pg_notify(2.178(24) epoch 11709) -4879> 2014-09-18 18:08:30.904017 7f4c9f251700 5 -- op tracker -- , seq: 85986, time: 2014-09-18 18:08:30.904015, event: done, op: pg_notify(2.178(24) epoch 11709) -4878> 2014-09-18 18:08:30.904333 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3197 ==== pg_notify(2.410(12) epoch 11709) v5 ==== 1376+0+0 (2169784382 0 0) 0x24f7a580 con 0x1395ec60 -4877> 2014-09-18 18:08:30.904344 7f4c9f251700 5 -- op tracker -- , seq: 85987, time: 2014-09-18 18:08:30.904291, event: header_read, op: pg_notify(2.410(12) epoch 11709) -4876> 2014-09-18 18:08:30.904351 7f4c9f251700 5 -- op tracker -- , seq: 85987, time: 2014-09-18 18:08:30.904292, event: throttled, op: pg_notify(2.410(12) epoch 11709) -4875> 2014-09-18 18:08:30.904357 7f4c9f251700 5 -- op tracker -- , seq: 85987, time: 2014-09-18 18:08:30.904322, event: all_read, op: pg_notify(2.410(12) epoch 11709) -4874> 2014-09-18 18:08:30.904363 7f4c9f251700 5 -- op tracker -- , seq: 85987, time: 2014-09-18 18:08:30.904341, event: dispatched, op: pg_notify(2.410(12) epoch 11709) -4873> 2014-09-18 18:08:30.904370 7f4c9f251700 5 -- op tracker -- , seq: 85987, time: 2014-09-18 18:08:30.904368, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11709) -4872> 2014-09-18 18:08:30.904377 7f4c9f251700 5 -- op tracker -- , seq: 85987, time: 2014-09-18 18:08:30.904375, event: started, op: pg_notify(2.410(12) epoch 11709) -4871> 2014-09-18 18:08:30.904395 7f4c9f251700 5 -- op tracker -- , seq: 85987, time: 2014-09-18 18:08:30.904394, event: done, op: pg_notify(2.410(12) epoch 11709) -4870> 2014-09-18 18:08:30.904507 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3198 ==== pg_notify(2.186(12) epoch 11709) v5 ==== 1386+0+0 (2404324193 0 0) 0x24f78b40 con 0x1395ec60 -4869> 2014-09-18 18:08:30.904518 7f4c9f251700 5 -- op tracker -- , seq: 85988, time: 2014-09-18 18:08:30.904466, event: header_read, op: pg_notify(2.186(12) epoch 11709) -4868> 2014-09-18 18:08:30.904525 7f4c9f251700 5 -- op tracker -- , seq: 85988, time: 2014-09-18 18:08:30.904467, event: throttled, op: pg_notify(2.186(12) epoch 11709) -4867> 2014-09-18 18:08:30.904531 7f4c9f251700 5 -- op tracker -- , seq: 85988, time: 2014-09-18 18:08:30.904497, event: all_read, op: pg_notify(2.186(12) epoch 11709) -4866> 2014-09-18 18:08:30.904537 7f4c9f251700 5 -- op tracker -- , seq: 85988, time: 2014-09-18 18:08:30.904515, event: dispatched, op: pg_notify(2.186(12) epoch 11709) -4865> 2014-09-18 18:08:30.904543 7f4c9f251700 5 -- op tracker -- , seq: 85988, time: 2014-09-18 18:08:30.904542, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11709) -4864> 2014-09-18 18:08:30.904550 7f4c9f251700 5 -- op tracker -- , seq: 85988, time: 2014-09-18 18:08:30.904549, event: started, op: pg_notify(2.186(12) epoch 11709) -4863> 2014-09-18 18:08:30.904567 7f4c9f251700 5 -- op tracker -- , seq: 85988, time: 2014-09-18 18:08:30.904566, event: done, op: pg_notify(2.186(12) epoch 11709) -4862> 2014-09-18 18:08:30.936794 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 690 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0xeee9200 con 0x1f103e40 -4861> 2014-09-18 18:08:30.936980 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -4860> 2014-09-18 18:08:30.936991 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 691 ==== pg_notify(2.4f6(15) epoch 11709) v5 ==== 1543+0+0 (3096517688 0 0) 0x21e11a40 con 0x1f103e40 -4859> 2014-09-18 18:08:30.937006 7f4c9f251700 5 -- op tracker -- , seq: 85989, time: 2014-09-18 18:08:30.936793, event: header_read, op: pg_notify(2.4f6(15) epoch 11709) -4858> 2014-09-18 18:08:30.937020 7f4c9f251700 5 -- op tracker -- , seq: 85989, time: 2014-09-18 18:08:30.936793, event: throttled, op: pg_notify(2.4f6(15) epoch 11709) -4857> 2014-09-18 18:08:30.937028 7f4c9f251700 5 -- op tracker -- , seq: 85989, time: 2014-09-18 18:08:30.936825, event: all_read, op: pg_notify(2.4f6(15) epoch 11709) -4856> 2014-09-18 18:08:30.937037 7f4c9f251700 5 -- op tracker -- , seq: 85989, time: 2014-09-18 18:08:30.937001, event: dispatched, op: pg_notify(2.4f6(15) epoch 11709) -4855> 2014-09-18 18:08:30.937046 7f4c9f251700 5 -- op tracker -- , seq: 85989, time: 2014-09-18 18:08:30.937043, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11709) -4854> 2014-09-18 18:08:30.937056 7f4c9f251700 5 -- op tracker -- , seq: 85989, time: 2014-09-18 18:08:30.937054, event: started, op: pg_notify(2.4f6(15) epoch 11709) -4853> 2014-09-18 18:08:30.937084 7f4c9f251700 5 -- op tracker -- , seq: 85989, time: 2014-09-18 18:08:30.937082, event: done, op: pg_notify(2.4f6(15) epoch 11709) -4852> 2014-09-18 18:08:30.955957 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1726 ==== pg_info(1 pgs e11709:2.178) v4 ==== 718+0+0 (2999087865 0 0) 0xf1ea5a0 con 0x4900000 -4851> 2014-09-18 18:08:30.955974 7f4c9f251700 5 -- op tracker -- , seq: 85990, time: 2014-09-18 18:08:30.955914, event: header_read, op: pg_info(1 pgs e11709:2.178) -4850> 2014-09-18 18:08:30.955994 7f4c9f251700 5 -- op tracker -- , seq: 85990, time: 2014-09-18 18:08:30.955915, event: throttled, op: pg_info(1 pgs e11709:2.178) -4849> 2014-09-18 18:08:30.956000 7f4c9f251700 5 -- op tracker -- , seq: 85990, time: 2014-09-18 18:08:30.955944, event: all_read, op: pg_info(1 pgs e11709:2.178) -4848> 2014-09-18 18:08:30.956005 7f4c9f251700 5 -- op tracker -- , seq: 85990, time: 2014-09-18 18:08:30.955969, event: dispatched, op: pg_info(1 pgs e11709:2.178) -4847> 2014-09-18 18:08:30.956018 7f4c9f251700 5 -- op tracker -- , seq: 85990, time: 2014-09-18 18:08:30.956015, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.178) -4846> 2014-09-18 18:08:30.956028 7f4c9f251700 5 -- op tracker -- , seq: 85990, time: 2014-09-18 18:08:30.956025, event: started, op: pg_info(1 pgs e11709:2.178) -4845> 2014-09-18 18:08:30.956049 7f4c9f251700 5 -- op tracker -- , seq: 85990, time: 2014-09-18 18:08:30.956047, event: done, op: pg_info(1 pgs e11709:2.178) -4844> 2014-09-18 18:08:30.971842 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3199 ==== pg_notify(2.74e(11) epoch 11709) v5 ==== 1303+0+0 (2512653137 0 0) 0xddf2d20 con 0x1395ec60 -4843> 2014-09-18 18:08:30.971863 7f4c9f251700 5 -- op tracker -- , seq: 85991, time: 2014-09-18 18:08:30.971748, event: header_read, op: pg_notify(2.74e(11) epoch 11709) -4842> 2014-09-18 18:08:30.971871 7f4c9f251700 5 -- op tracker -- , seq: 85991, time: 2014-09-18 18:08:30.971749, event: throttled, op: pg_notify(2.74e(11) epoch 11709) -4841> 2014-09-18 18:08:30.971875 7f4c9f251700 5 -- op tracker -- , seq: 85991, time: 2014-09-18 18:08:30.971807, event: all_read, op: pg_notify(2.74e(11) epoch 11709) -4840> 2014-09-18 18:08:30.971879 7f4c9f251700 5 -- op tracker -- , seq: 85991, time: 2014-09-18 18:08:30.971858, event: dispatched, op: pg_notify(2.74e(11) epoch 11709) -4839> 2014-09-18 18:08:30.971884 7f4c9f251700 5 -- op tracker -- , seq: 85991, time: 2014-09-18 18:08:30.971882, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11709) -4838> 2014-09-18 18:08:30.971890 7f4c9f251700 5 -- op tracker -- , seq: 85991, time: 2014-09-18 18:08:30.971888, event: started, op: pg_notify(2.74e(11) epoch 11709) -4837> 2014-09-18 18:08:30.971915 7f4c9f251700 5 -- op tracker -- , seq: 85991, time: 2014-09-18 18:08:30.971913, event: done, op: pg_notify(2.74e(11) epoch 11709) -4836> 2014-09-18 18:08:30.975073 7f4ca853b700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=127 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3aec0).reader couldn't read tag, (0) Success -4835> 2014-09-18 18:08:30.975098 7f4ca853b700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=127 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3aec0).fault (0) Success -4834> 2014-09-18 18:08:30.975106 7f4ca853b700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=127 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3aec0).fault with nothing to send, going to standby -4833> 2014-09-18 18:08:30.975281 7f4c80b75700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x14b4f600 sd=54 :45434 s=2 pgs=5 cs=1 l=1 c=0x1d7c9180).reader couldn't read tag, (0) Success -4832> 2014-09-18 18:08:30.975322 7f4c80b75700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x14b4f600 sd=54 :45434 s=2 pgs=5 cs=1 l=1 c=0x1d7c9180).fault (0) Success -4831> 2014-09-18 18:08:30.975283 7f4c740ab700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x14b503c0 sd=93 :49381 s=2 pgs=5 cs=1 l=1 c=0x1d7c8100).reader couldn't read tag, (0) Success -4830> 2014-09-18 18:08:30.975351 7f4c740ab700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x14b503c0 sd=93 :49381 s=2 pgs=5 cs=1 l=1 c=0x1d7c8100).fault (0) Success -4829> 2014-09-18 18:08:30.975386 7f4c9da4e700 1 -- 10.242.42.178:0/2483 mark_down 0x1d7c8100 -- pipe dne -4828> 2014-09-18 18:08:30.975550 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=199 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).connect error 10.242.42.164:6856/39317, (111) Connection refused -4827> 2014-09-18 18:08:30.975577 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=199 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).fault (111) Connection refused -4826> 2014-09-18 18:08:30.975610 7f4c71984700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=199 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).fault -4825> 2014-09-18 18:08:30.975562 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).connect error 10.242.42.164:6855/39317, (111) Connection refused -4824> 2014-09-18 18:08:30.975622 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).fault (111) Connection refused -4823> 2014-09-18 18:08:30.975635 7f4c790fb700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).fault -4822> 2014-09-18 18:08:30.975670 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).connect error 10.242.42.164:6856/39317, (111) Connection refused -4821> 2014-09-18 18:08:30.975690 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).fault (111) Connection refused -4820> 2014-09-18 18:08:30.975707 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).connect error 10.242.42.164:6855/39317, (111) Connection refused -4819> 2014-09-18 18:08:30.975729 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).fault (111) Connection refused -4818> 2014-09-18 18:08:31.039254 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 692 ==== pg_notify(2.799(10),2.4c6(15) epoch 11709) v5 ==== 2779+0+0 (682050886 0 0) 0x21e12940 con 0x1f103e40 -4817> 2014-09-18 18:08:31.039275 7f4c9f251700 5 -- op tracker -- , seq: 85992, time: 2014-09-18 18:08:31.039144, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11709) -4816> 2014-09-18 18:08:31.039283 7f4c9f251700 5 -- op tracker -- , seq: 85992, time: 2014-09-18 18:08:31.039145, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11709) -4815> 2014-09-18 18:08:31.039288 7f4c9f251700 5 -- op tracker -- , seq: 85992, time: 2014-09-18 18:08:31.039214, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11709) -4814> 2014-09-18 18:08:31.039292 7f4c9f251700 5 -- op tracker -- , seq: 85992, time: 2014-09-18 18:08:31.039270, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11709) -4813> 2014-09-18 18:08:31.039297 7f4c9f251700 5 -- op tracker -- , seq: 85992, time: 2014-09-18 18:08:31.039295, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11709) -4812> 2014-09-18 18:08:31.039303 7f4c9f251700 5 -- op tracker -- , seq: 85992, time: 2014-09-18 18:08:31.039302, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11709) -4811> 2014-09-18 18:08:31.039338 7f4c9f251700 5 -- op tracker -- , seq: 85992, time: 2014-09-18 18:08:31.039337, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11709) -4810> 2014-09-18 18:08:31.045125 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3200 ==== pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11709) v5 ==== 6481+0+0 (120749879 0 0) 0xf099a20 con 0x1395ec60 -4809> 2014-09-18 18:08:31.045146 7f4c9f251700 5 -- op tracker -- , seq: 85993, time: 2014-09-18 18:08:31.044946, event: header_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11709) -4808> 2014-09-18 18:08:31.045157 7f4c9f251700 5 -- op tracker -- , seq: 85993, time: 2014-09-18 18:08:31.044947, event: throttled, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11709) -4807> 2014-09-18 18:08:31.045164 7f4c9f251700 5 -- op tracker -- , seq: 85993, time: 2014-09-18 18:08:31.045059, event: all_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11709) -4806> 2014-09-18 18:08:31.045172 7f4c9f251700 5 -- op tracker -- , seq: 85993, time: 2014-09-18 18:08:31.045141, event: dispatched, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11709) -4805> 2014-09-18 18:08:31.045180 7f4c9f251700 5 -- op tracker -- , seq: 85993, time: 2014-09-18 18:08:31.045178, event: waiting_for_osdmap, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11709) -4804> 2014-09-18 18:08:31.045189 7f4c9f251700 5 -- op tracker -- , seq: 85993, time: 2014-09-18 18:08:31.045187, event: started, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11709) -4803> 2014-09-18 18:08:31.045221 7f4c9f251700 5 -- op tracker -- , seq: 85993, time: 2014-09-18 18:08:31.045220, event: done, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11709) -4802> 2014-09-18 18:08:31.049082 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3201 ==== pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11709) v5 ==== 4008+0+0 (2600808111 0 0) 0x21cfa5a0 con 0x1395ec60 -4801> 2014-09-18 18:08:31.049104 7f4c9f251700 5 -- op tracker -- , seq: 85994, time: 2014-09-18 18:08:31.048987, event: header_read, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11709) -4800> 2014-09-18 18:08:31.049115 7f4c9f251700 5 -- op tracker -- , seq: 85994, time: 2014-09-18 18:08:31.048990, event: throttled, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11709) -4799> 2014-09-18 18:08:31.049121 7f4c9f251700 5 -- op tracker -- , seq: 85994, time: 2014-09-18 18:08:31.049043, event: all_read, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11709) -4798> 2014-09-18 18:08:31.049128 7f4c9f251700 5 -- op tracker -- , seq: 85994, time: 2014-09-18 18:08:31.049099, event: dispatched, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11709) -4797> 2014-09-18 18:08:31.049135 7f4c9f251700 5 -- op tracker -- , seq: 85994, time: 2014-09-18 18:08:31.049133, event: waiting_for_osdmap, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11709) -4796> 2014-09-18 18:08:31.049143 7f4c9f251700 5 -- op tracker -- , seq: 85994, time: 2014-09-18 18:08:31.049141, event: started, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11709) -4795> 2014-09-18 18:08:31.049177 7f4c9f251700 5 -- op tracker -- , seq: 85994, time: 2014-09-18 18:08:31.049175, event: done, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11709) -4794> 2014-09-18 18:08:31.087365 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.6 10.242.42.166:0/22481 440 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:31.088704) v2 ==== 47+0+0 (2602473747 0 0) 0x21f06940 con 0x10201700 -4793> 2014-09-18 18:08:31.087404 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/22481 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.088704) v2 -- ?+0 0x13041e00 con 0x10201700 -4792> 2014-09-18 18:08:31.087446 7f4c9ca4c700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x11bb9840 con 0x1088aec0 -4791> 2014-09-18 18:08:31.087667 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.6 10.242.42.166:0/22481 440 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:31.088704) v2 ==== 47+0+0 (2602473747 0 0) 0x100f0940 con 0x14393080 -4790> 2014-09-18 18:08:31.087691 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/22481 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.088704) v2 -- ?+0 0xf273860 con 0x14393080 -4789> 2014-09-18 18:08:31.121150 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 303 ==== pg_info(1 pgs e11709:2.756) v4 ==== 738+0+0 (1274150929 0 0) 0xec6b480 con 0x140f95a0 -4788> 2014-09-18 18:08:31.121180 7f4c9f251700 5 -- op tracker -- , seq: 85995, time: 2014-09-18 18:08:31.121036, event: header_read, op: pg_info(1 pgs e11709:2.756) -4787> 2014-09-18 18:08:31.121191 7f4c9f251700 5 -- op tracker -- , seq: 85995, time: 2014-09-18 18:08:31.121039, event: throttled, op: pg_info(1 pgs e11709:2.756) -4786> 2014-09-18 18:08:31.121195 7f4c9f251700 5 -- op tracker -- , seq: 85995, time: 2014-09-18 18:08:31.121105, event: all_read, op: pg_info(1 pgs e11709:2.756) -4785> 2014-09-18 18:08:31.121200 7f4c9f251700 5 -- op tracker -- , seq: 85995, time: 2014-09-18 18:08:31.121174, event: dispatched, op: pg_info(1 pgs e11709:2.756) -4784> 2014-09-18 18:08:31.121207 7f4c9f251700 5 -- op tracker -- , seq: 85995, time: 2014-09-18 18:08:31.121205, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.756) -4783> 2014-09-18 18:08:31.121223 7f4c9f251700 5 -- op tracker -- , seq: 85995, time: 2014-09-18 18:08:31.121222, event: started, op: pg_info(1 pgs e11709:2.756) -4782> 2014-09-18 18:08:31.121247 7f4c9f251700 5 -- op tracker -- , seq: 85995, time: 2014-09-18 18:08:31.121246, event: done, op: pg_info(1 pgs e11709:2.756) -4781> 2014-09-18 18:08:31.145721 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 341 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0xe2ecfc0 con 0x12562260 -4780> 2014-09-18 18:08:31.145742 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -4779> 2014-09-18 18:08:31.145748 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 342 ==== pg_notify(2.719(17) epoch 11709) v5 ==== 1601+0+0 (3965405738 0 0) 0x117383c0 con 0x12562260 -4778> 2014-09-18 18:08:31.145756 7f4c9f251700 5 -- op tracker -- , seq: 85996, time: 2014-09-18 18:08:31.145652, event: header_read, op: pg_notify(2.719(17) epoch 11709) -4777> 2014-09-18 18:08:31.145767 7f4c9f251700 5 -- op tracker -- , seq: 85996, time: 2014-09-18 18:08:31.145653, event: throttled, op: pg_notify(2.719(17) epoch 11709) -4776> 2014-09-18 18:08:31.145777 7f4c9f251700 5 -- op tracker -- , seq: 85996, time: 2014-09-18 18:08:31.145686, event: all_read, op: pg_notify(2.719(17) epoch 11709) -4775> 2014-09-18 18:08:31.145785 7f4c9f251700 5 -- op tracker -- , seq: 85996, time: 2014-09-18 18:08:31.145753, event: dispatched, op: pg_notify(2.719(17) epoch 11709) -4774> 2014-09-18 18:08:31.145792 7f4c9f251700 5 -- op tracker -- , seq: 85996, time: 2014-09-18 18:08:31.145791, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11709) -4773> 2014-09-18 18:08:31.145802 7f4c9f251700 5 -- op tracker -- , seq: 85996, time: 2014-09-18 18:08:31.145800, event: started, op: pg_notify(2.719(17) epoch 11709) -4772> 2014-09-18 18:08:31.145821 7f4c9f251700 5 -- op tracker -- , seq: 85996, time: 2014-09-18 18:08:31.145820, event: done, op: pg_notify(2.719(17) epoch 11709) -4771> 2014-09-18 18:08:31.171750 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3202 ==== pg_info(1 pgs e11709:2.67b(3)) v4 ==== 903+0+0 (297274314 0 0) 0x2b986780 con 0x1395ec60 -4770> 2014-09-18 18:08:31.171769 7f4c9f251700 5 -- op tracker -- , seq: 85997, time: 2014-09-18 18:08:31.171666, event: header_read, op: pg_info(1 pgs e11709:2.67b(3)) -4769> 2014-09-18 18:08:31.171781 7f4c9f251700 5 -- op tracker -- , seq: 85997, time: 2014-09-18 18:08:31.171667, event: throttled, op: pg_info(1 pgs e11709:2.67b(3)) -4768> 2014-09-18 18:08:31.171788 7f4c9f251700 5 -- op tracker -- , seq: 85997, time: 2014-09-18 18:08:31.171710, event: all_read, op: pg_info(1 pgs e11709:2.67b(3)) -4767> 2014-09-18 18:08:31.171796 7f4c9f251700 5 -- op tracker -- , seq: 85997, time: 2014-09-18 18:08:31.171765, event: dispatched, op: pg_info(1 pgs e11709:2.67b(3)) -4766> 2014-09-18 18:08:31.171804 7f4c9f251700 5 -- op tracker -- , seq: 85997, time: 2014-09-18 18:08:31.171803, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.67b(3)) -4765> 2014-09-18 18:08:31.171813 7f4c9f251700 5 -- op tracker -- , seq: 85997, time: 2014-09-18 18:08:31.171812, event: started, op: pg_info(1 pgs e11709:2.67b(3)) -4764> 2014-09-18 18:08:31.171831 7f4c9f251700 5 -- op tracker -- , seq: 85997, time: 2014-09-18 18:08:31.171830, event: done, op: pg_info(1 pgs e11709:2.67b(3)) -4763> 2014-09-18 18:08:31.171842 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] exit Started/Stray 1.449038 7 0.000318 -4762> 2014-09-18 18:08:31.171864 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -4761> 2014-09-18 18:08:31.171921 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11600 n=520 ec=188 les/c 11600/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 crt=6745'24728 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -4760> 2014-09-18 18:08:31.172474 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11709:2.67b) v4 -- ?+0 0xf29c1c0 con 0x1395ec60 -4759> 2014-09-18 18:08:31.175831 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).connect error 10.242.42.164:6856/39317, (111) Connection refused -4758> 2014-09-18 18:08:31.175852 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).fault (111) Connection refused -4757> 2014-09-18 18:08:31.175846 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).connect error 10.242.42.164:6855/39317, (111) Connection refused -4756> 2014-09-18 18:08:31.175863 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).fault (111) Connection refused -4755> 2014-09-18 18:08:31.177686 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3203 ==== pg_info(1 pgs e11709:2.13) v4 ==== 738+0+0 (2092362235 0 0) 0xf29c1c0 con 0x1395ec60 -4754> 2014-09-18 18:08:31.177705 7f4c9f251700 5 -- op tracker -- , seq: 85998, time: 2014-09-18 18:08:31.177613, event: header_read, op: pg_info(1 pgs e11709:2.13) -4753> 2014-09-18 18:08:31.177715 7f4c9f251700 5 -- op tracker -- , seq: 85998, time: 2014-09-18 18:08:31.177614, event: throttled, op: pg_info(1 pgs e11709:2.13) -4752> 2014-09-18 18:08:31.177723 7f4c9f251700 5 -- op tracker -- , seq: 85998, time: 2014-09-18 18:08:31.177640, event: all_read, op: pg_info(1 pgs e11709:2.13) -4751> 2014-09-18 18:08:31.177730 7f4c9f251700 5 -- op tracker -- , seq: 85998, time: 2014-09-18 18:08:31.177700, event: dispatched, op: pg_info(1 pgs e11709:2.13) -4750> 2014-09-18 18:08:31.177737 7f4c9f251700 5 -- op tracker -- , seq: 85998, time: 2014-09-18 18:08:31.177735, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.13) -4749> 2014-09-18 18:08:31.177745 7f4c9f251700 5 -- op tracker -- , seq: 85998, time: 2014-09-18 18:08:31.177743, event: started, op: pg_info(1 pgs e11709:2.13) -4748> 2014-09-18 18:08:31.177762 7f4c9f251700 5 -- op tracker -- , seq: 85998, time: 2014-09-18 18:08:31.177761, event: done, op: pg_info(1 pgs e11709:2.13) -4747> 2014-09-18 18:08:31.179841 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3204 ==== pg_info(1 pgs e11709:2.178) v4 ==== 738+0+0 (3210229753 0 0) 0x2b987860 con 0x1395ec60 -4746> 2014-09-18 18:08:31.179860 7f4c9f251700 5 -- op tracker -- , seq: 85999, time: 2014-09-18 18:08:31.179794, event: header_read, op: pg_info(1 pgs e11709:2.178) -4745> 2014-09-18 18:08:31.179871 7f4c9f251700 5 -- op tracker -- , seq: 85999, time: 2014-09-18 18:08:31.179795, event: throttled, op: pg_info(1 pgs e11709:2.178) -4744> 2014-09-18 18:08:31.179877 7f4c9f251700 5 -- op tracker -- , seq: 85999, time: 2014-09-18 18:08:31.179827, event: all_read, op: pg_info(1 pgs e11709:2.178) -4743> 2014-09-18 18:08:31.179883 7f4c9f251700 5 -- op tracker -- , seq: 85999, time: 2014-09-18 18:08:31.179856, event: dispatched, op: pg_info(1 pgs e11709:2.178) -4742> 2014-09-18 18:08:31.179889 7f4c9f251700 5 -- op tracker -- , seq: 85999, time: 2014-09-18 18:08:31.179888, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.178) -4741> 2014-09-18 18:08:31.179897 7f4c9f251700 5 -- op tracker -- , seq: 85999, time: 2014-09-18 18:08:31.179896, event: started, op: pg_info(1 pgs e11709:2.178) -4740> 2014-09-18 18:08:31.179915 7f4c9f251700 5 -- op tracker -- , seq: 85999, time: 2014-09-18 18:08:31.179914, event: done, op: pg_info(1 pgs e11709:2.178) -4739> 2014-09-18 18:08:31.179932 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11709:2.178) v4 -- ?+0 0xf487c20 con 0x1395ec60 -4738> 2014-09-18 18:08:31.179951 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6822/33871 -- pg_info(1 pgs e11709:2.178) v4 -- ?+0 0xf4865a0 con 0x4900000 -4737> 2014-09-18 18:08:31.179965 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_info(1 pgs e11709:2.178) v4 -- ?+0 0xf4e6d20 con 0x1395ef20 -4736> 2014-09-18 18:08:31.180033 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11709 n=501 ec=188 les/c 11709/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=20,32 from 0//0//-1 -4735> 2014-09-18 18:08:31.180047 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11709 n=501 ec=188 les/c 11709/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 20 from 0//0//-1 -4734> 2014-09-18 18:08:31.180056 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11709 n=501 ec=188 les/c 11709/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 32 from 0//0//-1 -4733> 2014-09-18 18:08:31.180072 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11709 n=501 ec=188 les/c 11709/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 0.498938 9 0.001315 -4732> 2014-09-18 18:08:31.180084 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.178( v 6874'18440 (4157'15439,6874'18440] local-les=11709 n=501 ec=188 les/c 11709/11571 11708/11708/9945) [42,32,20]/[42,9] r=0 lpr=11708 pi=11570-11707/3 bft=20,32 crt=6810'18431 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -4731> 2014-09-18 18:08:31.181118 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5081 ==== pg_notify(2.120(11),2.40a(162) epoch 11707) v5 ==== 11039+0+0 (515136951 0 0) 0x2b9863c0 con 0x1088aec0 -4730> 2014-09-18 18:08:31.181137 7f4c9f251700 5 -- op tracker -- , seq: 86000, time: 2014-09-18 18:08:31.180935, event: header_read, op: pg_notify(2.120(11),2.40a(162) epoch 11707) -4729> 2014-09-18 18:08:31.181148 7f4c9f251700 5 -- op tracker -- , seq: 86000, time: 2014-09-18 18:08:31.180936, event: throttled, op: pg_notify(2.120(11),2.40a(162) epoch 11707) -4728> 2014-09-18 18:08:31.181154 7f4c9f251700 5 -- op tracker -- , seq: 86000, time: 2014-09-18 18:08:31.181102, event: all_read, op: pg_notify(2.120(11),2.40a(162) epoch 11707) -4727> 2014-09-18 18:08:31.181161 7f4c9f251700 5 -- op tracker -- , seq: 86000, time: 2014-09-18 18:08:31.181132, event: dispatched, op: pg_notify(2.120(11),2.40a(162) epoch 11707) -4726> 2014-09-18 18:08:31.181167 7f4c9f251700 5 -- op tracker -- , seq: 86000, time: 2014-09-18 18:08:31.181165, event: waiting_for_osdmap, op: pg_notify(2.120(11),2.40a(162) epoch 11707) -4725> 2014-09-18 18:08:31.181175 7f4c9f251700 5 -- op tracker -- , seq: 86000, time: 2014-09-18 18:08:31.181174, event: started, op: pg_notify(2.120(11),2.40a(162) epoch 11707) -4724> 2014-09-18 18:08:31.181202 7f4c9f251700 5 -- op tracker -- , seq: 86000, time: 2014-09-18 18:08:31.181200, event: done, op: pg_notify(2.120(11),2.40a(162) epoch 11707) -4723> 2014-09-18 18:08:31.186015 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5082 ==== pg_notify(2.1a2(20) epoch 11707) v5 ==== 1798+0+0 (1463438387 0 0) 0x1368cd20 con 0x1088aec0 -4722> 2014-09-18 18:08:31.186029 7f4c9f251700 5 -- op tracker -- , seq: 86001, time: 2014-09-18 18:08:31.185956, event: header_read, op: pg_notify(2.1a2(20) epoch 11707) -4721> 2014-09-18 18:08:31.186039 7f4c9f251700 5 -- op tracker -- , seq: 86001, time: 2014-09-18 18:08:31.185956, event: throttled, op: pg_notify(2.1a2(20) epoch 11707) -4720> 2014-09-18 18:08:31.186048 7f4c9f251700 5 -- op tracker -- , seq: 86001, time: 2014-09-18 18:08:31.185991, event: all_read, op: pg_notify(2.1a2(20) epoch 11707) -4719> 2014-09-18 18:08:31.186059 7f4c9f251700 5 -- op tracker -- , seq: 86001, time: 2014-09-18 18:08:31.186025, event: dispatched, op: pg_notify(2.1a2(20) epoch 11707) -4718> 2014-09-18 18:08:31.186064 7f4c9f251700 5 -- op tracker -- , seq: 86001, time: 2014-09-18 18:08:31.186062, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11707) -4717> 2014-09-18 18:08:31.186078 7f4c9f251700 5 -- op tracker -- , seq: 86001, time: 2014-09-18 18:08:31.186076, event: started, op: pg_notify(2.1a2(20) epoch 11707) -4716> 2014-09-18 18:08:31.186095 7f4c9f251700 5 -- op tracker -- , seq: 86001, time: 2014-09-18 18:08:31.186094, event: done, op: pg_notify(2.1a2(20) epoch 11707) -4715> 2014-09-18 18:08:31.187358 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5083 ==== pg_notify(2.4c1(15) epoch 11707) v5 ==== 1507+0+0 (2904917893 0 0) 0xf0a1de0 con 0x1088aec0 -4714> 2014-09-18 18:08:31.187372 7f4c9f251700 5 -- op tracker -- , seq: 86002, time: 2014-09-18 18:08:31.187314, event: header_read, op: pg_notify(2.4c1(15) epoch 11707) -4713> 2014-09-18 18:08:31.187380 7f4c9f251700 5 -- op tracker -- , seq: 86002, time: 2014-09-18 18:08:31.187315, event: throttled, op: pg_notify(2.4c1(15) epoch 11707) -4712> 2014-09-18 18:08:31.187386 7f4c9f251700 5 -- op tracker -- , seq: 86002, time: 2014-09-18 18:08:31.187347, event: all_read, op: pg_notify(2.4c1(15) epoch 11707) -4711> 2014-09-18 18:08:31.187392 7f4c9f251700 5 -- op tracker -- , seq: 86002, time: 2014-09-18 18:08:31.187369, event: dispatched, op: pg_notify(2.4c1(15) epoch 11707) -4710> 2014-09-18 18:08:31.187398 7f4c9f251700 5 -- op tracker -- , seq: 86002, time: 2014-09-18 18:08:31.187397, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11707) -4709> 2014-09-18 18:08:31.187406 7f4c9f251700 5 -- op tracker -- , seq: 86002, time: 2014-09-18 18:08:31.187404, event: started, op: pg_notify(2.4c1(15) epoch 11707) -4708> 2014-09-18 18:08:31.187422 7f4c9f251700 5 -- op tracker -- , seq: 86002, time: 2014-09-18 18:08:31.187421, event: done, op: pg_notify(2.4c1(15) epoch 11707) -4707> 2014-09-18 18:08:31.187459 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3205 ==== pg_info(1 pgs e11709:2.11b) v4 ==== 718+0+0 (3161006999 0 0) 0xf487c20 con 0x1395ec60 -4706> 2014-09-18 18:08:31.187469 7f4c9f251700 5 -- op tracker -- , seq: 86003, time: 2014-09-18 18:08:31.187431, event: header_read, op: pg_info(1 pgs e11709:2.11b) -4705> 2014-09-18 18:08:31.187476 7f4c9f251700 5 -- op tracker -- , seq: 86003, time: 2014-09-18 18:08:31.187431, event: throttled, op: pg_info(1 pgs e11709:2.11b) -4704> 2014-09-18 18:08:31.187482 7f4c9f251700 5 -- op tracker -- , seq: 86003, time: 2014-09-18 18:08:31.187448, event: all_read, op: pg_info(1 pgs e11709:2.11b) -4703> 2014-09-18 18:08:31.187488 7f4c9f251700 5 -- op tracker -- , seq: 86003, time: 2014-09-18 18:08:31.187467, event: dispatched, op: pg_info(1 pgs e11709:2.11b) -4702> 2014-09-18 18:08:31.187494 7f4c9f251700 5 -- op tracker -- , seq: 86003, time: 2014-09-18 18:08:31.187493, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.11b) -4701> 2014-09-18 18:08:31.187501 7f4c9f251700 5 -- op tracker -- , seq: 86003, time: 2014-09-18 18:08:31.187500, event: started, op: pg_info(1 pgs e11709:2.11b) -4700> 2014-09-18 18:08:31.187518 7f4c9f251700 5 -- op tracker -- , seq: 86003, time: 2014-09-18 18:08:31.187517, event: done, op: pg_info(1 pgs e11709:2.11b) -4699> 2014-09-18 18:08:31.187607 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11709:2.11b) v4 -- ?+0 0xf93a1e0 con 0x1395ec60 -4698> 2014-09-18 18:08:31.187626 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11709:2.11b) v4 -- ?+0 0xf93d840 con 0x1395f8c0 -4697> 2014-09-18 18:08:31.187686 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11709 n=460 ec=188 les/c 11709/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=9,27 from 0//0//-1 -4696> 2014-09-18 18:08:31.187766 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11709 n=460 ec=188 les/c 11709/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 9 from 0//0//-1 -4695> 2014-09-18 18:08:31.187804 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11709 n=460 ec=188 les/c 11709/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 27 from 0//0//-1 -4694> 2014-09-18 18:08:31.187822 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11709 n=460 ec=188 les/c 11709/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 0.496495 7 0.003576 -4693> 2014-09-18 18:08:31.187834 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.11b( v 6775'28078 (4153'25079,6775'28078] local-les=11709 n=460 ec=188 les/c 11709/10662 11708/11708/11258) [27,9,42]/[42] r=0 lpr=11708 pi=5116-11707/27 bft=9,27 crt=6775'28075 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -4692> 2014-09-18 18:08:31.193025 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3206 ==== pg_info(1 pgs e11709:2.67b) v4 ==== 738+0+0 (916075805 0 0) 0xf93a1e0 con 0x1395ec60 -4691> 2014-09-18 18:08:31.193044 7f4c9f251700 5 -- op tracker -- , seq: 86004, time: 2014-09-18 18:08:31.192889, event: header_read, op: pg_info(1 pgs e11709:2.67b) -4690> 2014-09-18 18:08:31.193055 7f4c9f251700 5 -- op tracker -- , seq: 86004, time: 2014-09-18 18:08:31.192890, event: throttled, op: pg_info(1 pgs e11709:2.67b) -4689> 2014-09-18 18:08:31.193063 7f4c9f251700 5 -- op tracker -- , seq: 86004, time: 2014-09-18 18:08:31.192922, event: all_read, op: pg_info(1 pgs e11709:2.67b) -4688> 2014-09-18 18:08:31.193070 7f4c9f251700 5 -- op tracker -- , seq: 86004, time: 2014-09-18 18:08:31.193040, event: dispatched, op: pg_info(1 pgs e11709:2.67b) -4687> 2014-09-18 18:08:31.193076 7f4c9f251700 5 -- op tracker -- , seq: 86004, time: 2014-09-18 18:08:31.193074, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.67b) -4686> 2014-09-18 18:08:31.193084 7f4c9f251700 5 -- op tracker -- , seq: 86004, time: 2014-09-18 18:08:31.193082, event: started, op: pg_info(1 pgs e11709:2.67b) -4685> 2014-09-18 18:08:31.193100 7f4c9f251700 5 -- op tracker -- , seq: 86004, time: 2014-09-18 18:08:31.193099, event: done, op: pg_info(1 pgs e11709:2.67b) -4684> 2014-09-18 18:08:31.331913 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3638 ==== pg_notify(2.40a(162) epoch 11708) v5 ==== 9736+0+0 (1341861356 0 0) 0x19f25660 con 0xcbecec0 -4683> 2014-09-18 18:08:31.331935 7f4c9f251700 5 -- op tracker -- , seq: 86005, time: 2014-09-18 18:08:31.331686, event: header_read, op: pg_notify(2.40a(162) epoch 11708) -4682> 2014-09-18 18:08:31.331947 7f4c9f251700 5 -- op tracker -- , seq: 86005, time: 2014-09-18 18:08:31.331688, event: throttled, op: pg_notify(2.40a(162) epoch 11708) -4681> 2014-09-18 18:08:31.331959 7f4c9f251700 5 -- op tracker -- , seq: 86005, time: 2014-09-18 18:08:31.331895, event: all_read, op: pg_notify(2.40a(162) epoch 11708) -4680> 2014-09-18 18:08:31.331969 7f4c9f251700 5 -- op tracker -- , seq: 86005, time: 2014-09-18 18:08:31.331930, event: dispatched, op: pg_notify(2.40a(162) epoch 11708) -4679> 2014-09-18 18:08:31.331980 7f4c9f251700 5 -- op tracker -- , seq: 86005, time: 2014-09-18 18:08:31.331978, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11708) -4678> 2014-09-18 18:08:31.331993 7f4c9f251700 5 -- op tracker -- , seq: 86005, time: 2014-09-18 18:08:31.331992, event: started, op: pg_notify(2.40a(162) epoch 11708) -4677> 2014-09-18 18:08:31.332016 7f4c9f251700 5 -- op tracker -- , seq: 86005, time: 2014-09-18 18:08:31.332015, event: done, op: pg_notify(2.40a(162) epoch 11708) -4676> 2014-09-18 18:08:31.332039 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3639 ==== pg_notify(2.186(12),2.719(17) epoch 11708) v5 ==== 2975+0+0 (1477956231 0 0) 0xe1ad480 con 0xcbecec0 -4675> 2014-09-18 18:08:31.332049 7f4c9f251700 5 -- op tracker -- , seq: 86006, time: 2014-09-18 18:08:31.331914, event: header_read, op: pg_notify(2.186(12),2.719(17) epoch 11708) -4674> 2014-09-18 18:08:31.332065 7f4c9f251700 5 -- op tracker -- , seq: 86006, time: 2014-09-18 18:08:31.331915, event: throttled, op: pg_notify(2.186(12),2.719(17) epoch 11708) -4673> 2014-09-18 18:08:31.332072 7f4c9f251700 5 -- op tracker -- , seq: 86006, time: 2014-09-18 18:08:31.331958, event: all_read, op: pg_notify(2.186(12),2.719(17) epoch 11708) -4672> 2014-09-18 18:08:31.332078 7f4c9f251700 5 -- op tracker -- , seq: 86006, time: 2014-09-18 18:08:31.332047, event: dispatched, op: pg_notify(2.186(12),2.719(17) epoch 11708) -4671> 2014-09-18 18:08:31.332085 7f4c9f251700 5 -- op tracker -- , seq: 86006, time: 2014-09-18 18:08:31.332084, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17) epoch 11708) -4670> 2014-09-18 18:08:31.332093 7f4c9f251700 5 -- op tracker -- , seq: 86006, time: 2014-09-18 18:08:31.332091, event: started, op: pg_notify(2.186(12),2.719(17) epoch 11708) -4669> 2014-09-18 18:08:31.332125 7f4c9f251700 5 -- op tracker -- , seq: 86006, time: 2014-09-18 18:08:31.332123, event: done, op: pg_notify(2.186(12),2.719(17) epoch 11708) -4668> 2014-09-18 18:08:31.333901 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3640 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0xe2ec6c0 con 0xcbecec0 -4667> 2014-09-18 18:08:31.333915 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -4666> 2014-09-18 18:08:31.333940 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3641 ==== pg_notify(2.1bb(9) epoch 11709) v5 ==== 1189+0+0 (1138583293 0 0) 0xe1abfe0 con 0xcbecec0 -4665> 2014-09-18 18:08:31.333950 7f4c9f251700 5 -- op tracker -- , seq: 86007, time: 2014-09-18 18:08:31.333905, event: header_read, op: pg_notify(2.1bb(9) epoch 11709) -4664> 2014-09-18 18:08:31.333958 7f4c9f251700 5 -- op tracker -- , seq: 86007, time: 2014-09-18 18:08:31.333905, event: throttled, op: pg_notify(2.1bb(9) epoch 11709) -4663> 2014-09-18 18:08:31.333964 7f4c9f251700 5 -- op tracker -- , seq: 86007, time: 2014-09-18 18:08:31.333930, event: all_read, op: pg_notify(2.1bb(9) epoch 11709) -4662> 2014-09-18 18:08:31.333970 7f4c9f251700 5 -- op tracker -- , seq: 86007, time: 2014-09-18 18:08:31.333947, event: dispatched, op: pg_notify(2.1bb(9) epoch 11709) -4661> 2014-09-18 18:08:31.333976 7f4c9f251700 5 -- op tracker -- , seq: 86007, time: 2014-09-18 18:08:31.333974, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11709) -4660> 2014-09-18 18:08:31.333983 7f4c9f251700 5 -- op tracker -- , seq: 86007, time: 2014-09-18 18:08:31.333982, event: started, op: pg_notify(2.1bb(9) epoch 11709) -4659> 2014-09-18 18:08:31.334000 7f4c9f251700 5 -- op tracker -- , seq: 86007, time: 2014-09-18 18:08:31.333998, event: done, op: pg_notify(2.1bb(9) epoch 11709) -4658> 2014-09-18 18:08:31.334344 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3642 ==== pg_notify(2.71c(12),2.457(21) epoch 11709) v5 ==== 3211+0+0 (1030981751 0 0) 0xf93b860 con 0xcbecec0 -4657> 2014-09-18 18:08:31.334363 7f4c9f251700 5 -- op tracker -- , seq: 86008, time: 2014-09-18 18:08:31.334278, event: header_read, op: pg_notify(2.71c(12),2.457(21) epoch 11709) -4656> 2014-09-18 18:08:31.334374 7f4c9f251700 5 -- op tracker -- , seq: 86008, time: 2014-09-18 18:08:31.334279, event: throttled, op: pg_notify(2.71c(12),2.457(21) epoch 11709) -4655> 2014-09-18 18:08:31.334380 7f4c9f251700 5 -- op tracker -- , seq: 86008, time: 2014-09-18 18:08:31.334330, event: all_read, op: pg_notify(2.71c(12),2.457(21) epoch 11709) -4654> 2014-09-18 18:08:31.334386 7f4c9f251700 5 -- op tracker -- , seq: 86008, time: 2014-09-18 18:08:31.334359, event: dispatched, op: pg_notify(2.71c(12),2.457(21) epoch 11709) -4653> 2014-09-18 18:08:31.334393 7f4c9f251700 5 -- op tracker -- , seq: 86008, time: 2014-09-18 18:08:31.334392, event: waiting_for_osdmap, op: pg_notify(2.71c(12),2.457(21) epoch 11709) -4652> 2014-09-18 18:08:31.334401 7f4c9f251700 5 -- op tracker -- , seq: 86008, time: 2014-09-18 18:08:31.334400, event: started, op: pg_notify(2.71c(12),2.457(21) epoch 11709) -4651> 2014-09-18 18:08:31.334429 7f4c9f251700 5 -- op tracker -- , seq: 86008, time: 2014-09-18 18:08:31.334427, event: done, op: pg_notify(2.71c(12),2.457(21) epoch 11709) -4650> 2014-09-18 18:08:31.336557 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3643 ==== pg_notify(2.78f(56) epoch 11709) v5 ==== 3722+0+0 (3336787197 0 0) 0xf93a000 con 0xcbecec0 -4649> 2014-09-18 18:08:31.336576 7f4c9f251700 5 -- op tracker -- , seq: 86009, time: 2014-09-18 18:08:31.336441, event: header_read, op: pg_notify(2.78f(56) epoch 11709) -4648> 2014-09-18 18:08:31.336587 7f4c9f251700 5 -- op tracker -- , seq: 86009, time: 2014-09-18 18:08:31.336442, event: throttled, op: pg_notify(2.78f(56) epoch 11709) -4647> 2014-09-18 18:08:31.336593 7f4c9f251700 5 -- op tracker -- , seq: 86009, time: 2014-09-18 18:08:31.336543, event: all_read, op: pg_notify(2.78f(56) epoch 11709) -4646> 2014-09-18 18:08:31.336599 7f4c9f251700 5 -- op tracker -- , seq: 86009, time: 2014-09-18 18:08:31.336572, event: dispatched, op: pg_notify(2.78f(56) epoch 11709) -4645> 2014-09-18 18:08:31.336606 7f4c9f251700 5 -- op tracker -- , seq: 86009, time: 2014-09-18 18:08:31.336605, event: waiting_for_osdmap, op: pg_notify(2.78f(56) epoch 11709) -4644> 2014-09-18 18:08:31.336614 7f4c9f251700 5 -- op tracker -- , seq: 86009, time: 2014-09-18 18:08:31.336612, event: started, op: pg_notify(2.78f(56) epoch 11709) -4643> 2014-09-18 18:08:31.336632 7f4c9f251700 5 -- op tracker -- , seq: 86009, time: 2014-09-18 18:08:31.336631, event: done, op: pg_notify(2.78f(56) epoch 11709) -4642> 2014-09-18 18:08:31.336658 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 1.614726 13 0.000266 -4641> 2014-09-18 18:08:31.336685 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -4640> 2014-09-18 18:08:31.336928 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000242 0 0.000000 -4639> 2014-09-18 18:08:31.336946 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -4638> 2014-09-18 18:08:31.337043 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000097 0 0.000000 -4637> 2014-09-18 18:08:31.337057 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.615172 0 0.000000 -4636> 2014-09-18 18:08:31.337071 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11703 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -4635> 2014-09-18 18:08:31.337544 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -4634> 2014-09-18 18:08:31.337934 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6854/39317 -- osd_map(11709..11709 src has 1..11709) v3 -- ?+0 0x2db45b00 con 0x12b3aec0 -4633> 2014-09-18 18:08:31.338054 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3644 ==== pg_notify(2.78f(56) epoch 11709) v5 ==== 3722+0+0 (396383107 0 0) 0xf93b0e0 con 0xcbecec0 -4632> 2014-09-18 18:08:31.338081 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6854/39317 -- pg_info(1 pgs e11709:2.78f(155)) v4 -- ?+0 0x21e11a40 con 0x12b3aec0 -4631> 2014-09-18 18:08:31.338106 7f4c9f251700 5 -- op tracker -- , seq: 86010, time: 2014-09-18 18:08:31.337929, event: header_read, op: pg_notify(2.78f(56) epoch 11709) -4630> 2014-09-18 18:08:31.338135 7f4c9f251700 5 -- op tracker -- , seq: 86010, time: 2014-09-18 18:08:31.337930, event: throttled, op: pg_notify(2.78f(56) epoch 11709) -4629> 2014-09-18 18:08:31.338147 7f4c9f251700 5 -- op tracker -- , seq: 86010, time: 2014-09-18 18:08:31.338019, event: all_read, op: pg_notify(2.78f(56) epoch 11709) -4628> 2014-09-18 18:08:31.338156 7f4c9f251700 5 -- op tracker -- , seq: 86010, time: 2014-09-18 18:08:31.338070, event: dispatched, op: pg_notify(2.78f(56) epoch 11709) -4627> 2014-09-18 18:08:31.338165 7f4c9f251700 5 -- op tracker -- , seq: 86010, time: 2014-09-18 18:08:31.338162, event: waiting_for_osdmap, op: pg_notify(2.78f(56) epoch 11709) -4626> 2014-09-18 18:08:31.338176 7f4c9f251700 5 -- op tracker -- , seq: 86010, time: 2014-09-18 18:08:31.338173, event: started, op: pg_notify(2.78f(56) epoch 11709) -4625> 2014-09-18 18:08:31.338155 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=93 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).connect error 10.242.42.164:6854/39317, (111) Connection refused -4624> 2014-09-18 18:08:31.338200 7f4c9f251700 5 -- op tracker -- , seq: 86010, time: 2014-09-18 18:08:31.338198, event: done, op: pg_notify(2.78f(56) epoch 11709) -4623> 2014-09-18 18:08:31.338203 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11709:2.78f(155)) v4 -- ?+0 0xf1ea5a0 con 0x1395f8c0 -4622> 2014-09-18 18:08:31.338206 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=93 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).fault (111) Connection refused -4621> 2014-09-18 18:08:31.338216 7f4c773de700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=93 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).fault -4620> 2014-09-18 18:08:31.338268 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=93 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).connect error 10.242.42.164:6854/39317, (111) Connection refused -4619> 2014-09-18 18:08:31.338290 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=93 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).fault (111) Connection refused -4618> 2014-09-18 18:08:31.338293 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11709:2.78f(155)) v4 -- ?+0 0xf810b20 con 0xcbecec0 -4617> 2014-09-18 18:08:31.339966 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 937 ==== pg_info(1 pgs e11709:2.78f) v4 ==== 738+0+0 (2012454504 0 0) 0xf1ea5a0 con 0x1395f8c0 -4616> 2014-09-18 18:08:31.339985 7f4c9f251700 5 -- op tracker -- , seq: 86011, time: 2014-09-18 18:08:31.339902, event: header_read, op: pg_info(1 pgs e11709:2.78f) -4615> 2014-09-18 18:08:31.339995 7f4c9f251700 5 -- op tracker -- , seq: 86011, time: 2014-09-18 18:08:31.339903, event: throttled, op: pg_info(1 pgs e11709:2.78f) -4614> 2014-09-18 18:08:31.340002 7f4c9f251700 5 -- op tracker -- , seq: 86011, time: 2014-09-18 18:08:31.339928, event: all_read, op: pg_info(1 pgs e11709:2.78f) -4613> 2014-09-18 18:08:31.340008 7f4c9f251700 5 -- op tracker -- , seq: 86011, time: 2014-09-18 18:08:31.339981, event: dispatched, op: pg_info(1 pgs e11709:2.78f) -4612> 2014-09-18 18:08:31.340014 7f4c9f251700 5 -- op tracker -- , seq: 86011, time: 2014-09-18 18:08:31.340013, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.78f) -4611> 2014-09-18 18:08:31.340023 7f4c9f251700 5 -- op tracker -- , seq: 86011, time: 2014-09-18 18:08:31.340021, event: started, op: pg_info(1 pgs e11709:2.78f) -4610> 2014-09-18 18:08:31.340040 7f4c9f251700 5 -- op tracker -- , seq: 86011, time: 2014-09-18 18:08:31.340039, event: done, op: pg_info(1 pgs e11709:2.78f) -4609> 2014-09-18 18:08:31.375757 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 963 ==== osd_map(11709..11709 src has 1..11709) v3 ==== 1350+0+0 (3586054905 0 0) 0x11bb81c0 con 0x10a878c0 -4608> 2014-09-18 18:08:31.375786 7f4c9f251700 3 osd.42 11709 handle_osd_map epochs [11709,11709], i have 11709, src has [1,11709] -4607> 2014-09-18 18:08:31.375868 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 964 ==== pg_notify(2.517(11),2.4c6(15) epoch 11709) v5 ==== 2834+0+0 (3560189037 0 0) 0x1624ef00 con 0x10a878c0 -4606> 2014-09-18 18:08:31.375885 7f4c9f251700 5 -- op tracker -- , seq: 86012, time: 2014-09-18 18:08:31.375761, event: header_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11709) -4605> 2014-09-18 18:08:31.375898 7f4c9f251700 5 -- op tracker -- , seq: 86012, time: 2014-09-18 18:08:31.375761, event: throttled, op: pg_notify(2.517(11),2.4c6(15) epoch 11709) -4604> 2014-09-18 18:08:31.375909 7f4c9f251700 5 -- op tracker -- , seq: 86012, time: 2014-09-18 18:08:31.375852, event: all_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11709) -4603> 2014-09-18 18:08:31.375919 7f4c9f251700 5 -- op tracker -- , seq: 86012, time: 2014-09-18 18:08:31.375880, event: dispatched, op: pg_notify(2.517(11),2.4c6(15) epoch 11709) -4602> 2014-09-18 18:08:31.375924 7f4c9f251700 5 -- op tracker -- , seq: 86012, time: 2014-09-18 18:08:31.375923, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.4c6(15) epoch 11709) -4601> 2014-09-18 18:08:31.375932 7f4c9f251700 5 -- op tracker -- , seq: 86012, time: 2014-09-18 18:08:31.375930, event: started, op: pg_notify(2.517(11),2.4c6(15) epoch 11709) -4600> 2014-09-18 18:08:31.375971 7f4c9f251700 5 -- op tracker -- , seq: 86012, time: 2014-09-18 18:08:31.375968, event: done, op: pg_notify(2.517(11),2.4c6(15) epoch 11709) -4599> 2014-09-18 18:08:31.385137 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.22 10.242.42.172:0/37668 440 ==== osd_ping(ping e11709 stamp 2014-09-18 23:37:20.489420) v2 ==== 47+0+0 (401398823 0 0) 0xf160ee0 con 0x109caf20 -4598> 2014-09-18 18:08:31.385156 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.22 10.242.42.172:0/37668 440 ==== osd_ping(ping e11709 stamp 2014-09-18 23:37:20.489420) v2 ==== 47+0+0 (401398823 0 0) 0xf93a3c0 con 0x14393340 -4597> 2014-09-18 18:08:31.385165 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11709 stamp 2014-09-18 23:37:20.489420) v2 -- ?+0 0x21f06940 con 0x109caf20 -4596> 2014-09-18 18:08:31.385213 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11709 stamp 2014-09-18 23:37:20.489420) v2 -- ?+0 0x100f0940 con 0x14393340 -4595> 2014-09-18 18:08:31.398076 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.11 10.242.42.168:0/17298 127 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.264901) v2 ==== 47+0+0 (1293500640 0 0) 0xe76e940 con 0xcc49e40 -4594> 2014-09-18 18:08:31.398075 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.11 10.242.42.168:0/17298 127 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.264901) v2 ==== 47+0+0 (1293500640 0 0) 0xf09be00 con 0x101cd180 -4593> 2014-09-18 18:08:31.398122 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.264901) v2 -- ?+0 0xf160ee0 con 0xcc49e40 -4592> 2014-09-18 18:08:31.398150 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.264901) v2 -- ?+0 0xf93a3c0 con 0x101cd180 -4591> 2014-09-18 18:08:31.416632 7f4c8fa32700 5 osd.42 11709 heartbeat: osd_stat(228 GB used, 138 GB avail, 366 GB total, peers [0,1,2,3,4,6,9,11,20,22,23,24,27,30,32,35,39,43,47,48,53,55,56,57,58,59,63]/[] op hist []) -4590> 2014-09-18 18:08:31.416648 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6802/15878 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf6deb40 con 0x4751e40 -4589> 2014-09-18 18:08:31.416666 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6803/15878 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf6e0d00 con 0x149b0ec0 -4588> 2014-09-18 18:08:31.416677 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6855/39317 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf6e01c0 con 0x23a75760 -4587> 2014-09-18 18:08:31.416701 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6856/39317 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf6e0580 con 0x23a746e0 -4586> 2014-09-18 18:08:31.416712 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6818/19139 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf6de960 con 0x12b6a940 -4585> 2014-09-18 18:08:31.416727 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6820/19139 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf6dfa40 con 0x1d518aa0 -4584> 2014-09-18 18:08:31.416742 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6824/20623 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf6e1de0 con 0x1d662b00 -4583> 2014-09-18 18:08:31.416759 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6825/20623 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa261e0 con 0x1d6629a0 -4582> 2014-09-18 18:08:31.416778 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6826/22324 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa29480 con 0x1277db20 -4581> 2014-09-18 18:08:31.416803 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6828/22324 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa29a20 con 0x1277d180 -4580> 2014-09-18 18:08:31.416815 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6810/22481 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa27c20 con 0x1096fa20 -4579> 2014-09-18 18:08:31.416826 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6813/22481 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa29840 con 0x10970520 -4578> 2014-09-18 18:08:31.416836 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6827/24959 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa281c0 con 0x1db5c420 -4577> 2014-09-18 18:08:31.416840 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).connect error 10.242.42.164:6856/39317, (111) Connection refused -4576> 2014-09-18 18:08:31.416840 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).connect error 10.242.42.164:6855/39317, (111) Connection refused -4575> 2014-09-18 18:08:31.416857 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6840/24959 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa27860 con 0x1db5c160 -4574> 2014-09-18 18:08:31.416876 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).fault (111) Connection refused -4573> 2014-09-18 18:08:31.416880 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).fault (111) Connection refused -4572> 2014-09-18 18:08:31.416889 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6822/17298 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa28940 con 0x1217b760 -4571> 2014-09-18 18:08:31.416907 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6823/17298 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa29de0 con 0x1217c7e0 -4570> 2014-09-18 18:08:31.416927 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6823/33871 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa26780 con 0xe58a9a0 -4569> 2014-09-18 18:08:31.416937 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6825/33871 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xfa26f00 con 0xe58a840 -4568> 2014-09-18 18:08:31.416948 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6820/37668 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x100f12a0 con 0x10a407e0 -4567> 2014-09-18 18:08:31.416963 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6832/37668 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x100f0580 con 0x10a40260 -4566> 2014-09-18 18:08:31.416991 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6826/11463 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x100f0ee0 con 0xe417b20 -4565> 2014-09-18 18:08:31.417004 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6827/11463 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x100efa40 con 0xe58ab00 -4564> 2014-09-18 18:08:31.417016 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6803/5612 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x100eef00 con 0xeff6f20 -4563> 2014-09-18 18:08:31.417030 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6810/5612 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x100f1480 con 0xeff6c60 -4562> 2014-09-18 18:08:31.417035 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6825/20623 171 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf242b40 con 0x1d6629a0 -4561> 2014-09-18 18:08:31.417048 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6802/21548 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x100f1660 con 0x1da53a20 -4560> 2014-09-18 18:08:31.417096 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6803/21548 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x853cd20 con 0x1da53760 -4559> 2014-09-18 18:08:31.417134 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6830/1204 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x853f2a0 con 0x1db5cdc0 -4558> 2014-09-18 18:08:31.417162 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6831/1204 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x853c780 con 0x1db5cc60 -4557> 2014-09-18 18:08:31.417178 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6825/1099 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x19e48960 con 0x1db5da20 -4556> 2014-09-18 18:08:31.417200 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6826/1099 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x19e48780 con 0x1db5db80 -4555> 2014-09-18 18:08:31.417211 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6813/17421 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x19e4b840 con 0x125239c0 -4554> 2014-09-18 18:08:31.417222 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6814/17421 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x19e492c0 con 0x11d45860 -4553> 2014-09-18 18:08:31.417233 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6817/2660 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x2b94d0c0 con 0x238dc420 -4552> 2014-09-18 18:08:31.417243 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6818/2660 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x2b94af00 con 0x238dcb00 -4551> 2014-09-18 18:08:31.417253 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6843/4511 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x2b94d2a0 con 0x149b1de0 -4550> 2014-09-18 18:08:31.417263 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6845/4511 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x2b94b680 con 0x149b07e0 -4549> 2014-09-18 18:08:31.417274 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6832/25184 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x2b94bfe0 con 0x1217d180 -4548> 2014-09-18 18:08:31.417293 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6833/25184 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x2b94b4a0 con 0x1217caa0 -4547> 2014-09-18 18:08:31.417304 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6837/27583 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf810940 con 0x10db26e0 -4546> 2014-09-18 18:08:31.417322 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6838/27583 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf80f680 con 0x10db4ec0 -4545> 2014-09-18 18:08:31.417341 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6826/2318 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf810580 con 0x1db5c6e0 -4544> 2014-09-18 18:08:31.417353 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6832/2318 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xf099480 con 0x1db5e260 -4543> 2014-09-18 18:08:31.417365 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6830/19404 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x21e10000 con 0xe58d180 -4542> 2014-09-18 18:08:31.417377 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6831/19404 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x21e114a0 con 0xe58d020 -4541> 2014-09-18 18:08:31.417390 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6840/36458 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x1247d4a0 con 0x1364b700 -4540> 2014-09-18 18:08:31.417401 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6842/36458 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x1247c1e0 con 0x1364b860 -4539> 2014-09-18 18:08:31.417411 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6802/988 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x1247e3a0 con 0x1db5d600 -4538> 2014-09-18 18:08:31.417431 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6803/988 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xde5e3c0 con 0x1db5dfa0 -4537> 2014-09-18 18:08:31.417449 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6839/46756 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0x21d15c00 con 0x1d7c7a20 -4536> 2014-09-18 18:08:31.417466 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6842/46756 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xe7761c0 con 0x1da543c0 -4535> 2014-09-18 18:08:31.417483 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6842/4508 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xe775c20 con 0x1d7c6580 -4534> 2014-09-18 18:08:31.417497 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6844/4508 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xe7770c0 con 0x1d7c9440 -4533> 2014-09-18 18:08:31.417516 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6805/2487 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xe777480 con 0x1d996aa0 -4532> 2014-09-18 18:08:31.417528 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6809/2487 -- osd_ping(ping e11709 stamp 2014-09-18 18:08:31.416647) v2 -- ?+0 0xe7743c0 con 0x1d996c00 -4531> 2014-09-18 18:08:31.417559 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6826/22324 165 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x12ba30c0 con 0x1277db20 -4530> 2014-09-18 18:08:31.417643 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6828/22324 165 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x19f23c20 con 0x1277d180 -4529> 2014-09-18 18:08:31.417662 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6843/4511 163 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf6e0760 con 0x149b1de0 -4528> 2014-09-18 18:08:31.417678 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6845/4511 163 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x1390af00 con 0x149b07e0 -4527> 2014-09-18 18:08:31.417697 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6839/46756 161 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf7a9840 con 0x1d7c7a20 -4526> 2014-09-18 18:08:31.417710 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6842/46756 161 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf1e63c0 con 0x1da543c0 -4525> 2014-09-18 18:08:31.417727 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6802/21548 160 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf274940 con 0x1da53a20 -4524> 2014-09-18 18:08:31.417740 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6803/21548 160 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x12139860 con 0x1da53760 -4523> 2014-09-18 18:08:31.417751 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6822/17298 138 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x24f7a940 con 0x1217b760 -4522> 2014-09-18 18:08:31.417762 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6823/17298 138 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf80ed20 con 0x1217c7e0 -4521> 2014-09-18 18:08:31.417780 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6832/25184 138 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x1390be00 con 0x1217d180 -4520> 2014-09-18 18:08:31.417792 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6833/25184 138 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf1065a0 con 0x1217caa0 -4519> 2014-09-18 18:08:31.417803 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6802/988 126 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xe0c70e0 con 0x1db5d600 -4518> 2014-09-18 18:08:31.417815 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6803/988 126 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xe146f00 con 0x1db5dfa0 -4517> 2014-09-18 18:08:31.417826 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6825/1099 126 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xfae8940 con 0x1db5da20 -4516> 2014-09-18 18:08:31.417837 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6826/1099 126 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x128cb0e0 con 0x1db5db80 -4515> 2014-09-18 18:08:31.417848 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6830/1204 126 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xfc56000 con 0x1db5cdc0 -4514> 2014-09-18 18:08:31.417860 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6831/1204 126 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x10415480 con 0x1db5cc60 -4513> 2014-09-18 18:08:31.417870 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6827/24959 124 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x11b0e1e0 con 0x1db5c420 -4512> 2014-09-18 18:08:31.417882 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6840/24959 124 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xe1490c0 con 0x1db5c160 -4511> 2014-09-18 18:08:31.417892 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6826/2318 124 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf29ab40 con 0x1db5c6e0 -4510> 2014-09-18 18:08:31.417904 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6832/2318 124 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xe0c8ee0 con 0x1db5e260 -4509> 2014-09-18 18:08:31.417915 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6837/27583 104 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x1247cd20 con 0x10db26e0 -4508> 2014-09-18 18:08:31.417927 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6838/27583 104 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf0e3860 con 0x10db4ec0 -4507> 2014-09-18 18:08:31.417937 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.59 10.242.42.178:6842/4508 1 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x1393da20 con 0x1d7c6580 -4506> 2014-09-18 18:08:31.417949 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.59 10.242.42.178:6844/4508 1 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x2b930b20 con 0x1d7c9440 -4505> 2014-09-18 18:08:31.417961 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6826/11463 1577 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x2b94c580 con 0xe417b20 -4504> 2014-09-18 18:08:31.417972 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6827/11463 1577 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xe7752c0 con 0xe58ab00 -4503> 2014-09-18 18:08:31.417983 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6823/33871 1577 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xfa270e0 con 0xe58a9a0 -4502> 2014-09-18 18:08:31.417995 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6825/33871 1577 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf9732c0 con 0xe58a840 -4501> 2014-09-18 18:08:31.418005 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6830/19404 1577 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf1e9de0 con 0xe58d180 -4500> 2014-09-18 18:08:31.418017 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6831/19404 1577 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf7a9c00 con 0xe58d020 -4499> 2014-09-18 18:08:31.418027 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6840/36458 657 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x24fa10e0 con 0x1364b700 -4498> 2014-09-18 18:08:31.418039 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6842/36458 657 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf15ffe0 con 0x1364b860 -4497> 2014-09-18 18:08:31.418049 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6810/22481 457 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (143328964 0 0) 0x2b924760 con 0x1096fa20 -4496> 2014-09-18 18:08:31.418061 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6813/22481 457 ==== osd_ping(ping_reply e11707 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (143328964 0 0) 0xe76f0c0 con 0x10970520 -4495> 2014-09-18 18:08:31.418071 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6820/37668 451 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf975a20 con 0x10a407e0 -4494> 2014-09-18 18:08:31.418083 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6832/37668 451 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf243a40 con 0x10a40260 -4493> 2014-09-18 18:08:31.418093 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6803/5612 343 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf0b6780 con 0xeff6f20 -4492> 2014-09-18 18:08:31.418104 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6810/5612 343 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x24fa1680 con 0xeff6c60 -4491> 2014-09-18 18:08:31.418115 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6805/2487 332 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x1368e3a0 con 0x1d996aa0 -4490> 2014-09-18 18:08:31.418128 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6809/2487 332 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf3432c0 con 0x1d996c00 -4489> 2014-09-18 18:08:31.418138 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6802/15878 191 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf7283a0 con 0x4751e40 -4488> 2014-09-18 18:08:31.418150 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6803/15878 191 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x21f06d00 con 0x149b0ec0 -4487> 2014-09-18 18:08:31.418161 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6813/17421 188 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf93cb20 con 0x125239c0 -4486> 2014-09-18 18:08:31.418172 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6814/17421 188 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x10415de0 con 0x11d45860 -4485> 2014-09-18 18:08:31.418183 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6817/2660 186 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf2b01c0 con 0x238dc420 -4484> 2014-09-18 18:08:31.418195 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6818/2660 186 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xf91c940 con 0x238dcb00 -4483> 2014-09-18 18:08:31.418205 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6818/19139 182 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xfc581c0 con 0x12b6a940 -4482> 2014-09-18 18:08:31.418217 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6820/19139 182 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0xec683c0 con 0x1d518aa0 -4481> 2014-09-18 18:08:31.418227 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6824/20623 171 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.416647) v2 ==== 47+0+0 (515126043 0 0) 0x2b9292a0 con 0x1d662b00 -4480> 2014-09-18 18:08:31.487446 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 965 ==== MBackfillReserve GRANT pgid: 2.1bb, query_epoch: 11709 v3 ==== 30+0+0 (3112237280 0 0) 0x1624e960 con 0x10a878c0 -4479> 2014-09-18 18:08:31.487475 7f4c9f251700 5 -- op tracker -- , seq: 86013, time: 2014-09-18 18:08:31.487346, event: header_read, op: MBackfillReserve GRANT pgid: 2.1bb, query_epoch: 11709 -4478> 2014-09-18 18:08:31.487488 7f4c9f251700 5 -- op tracker -- , seq: 86013, time: 2014-09-18 18:08:31.487348, event: throttled, op: MBackfillReserve GRANT pgid: 2.1bb, query_epoch: 11709 -4477> 2014-09-18 18:08:31.487496 7f4c9f251700 5 -- op tracker -- , seq: 86013, time: 2014-09-18 18:08:31.487396, event: all_read, op: MBackfillReserve GRANT pgid: 2.1bb, query_epoch: 11709 -4476> 2014-09-18 18:08:31.487503 7f4c9f251700 5 -- op tracker -- , seq: 86013, time: 2014-09-18 18:08:31.487468, event: dispatched, op: MBackfillReserve GRANT pgid: 2.1bb, query_epoch: 11709 -4475> 2014-09-18 18:08:31.487511 7f4c9f251700 5 -- op tracker -- , seq: 86013, time: 2014-09-18 18:08:31.487510, event: waiting_for_osdmap, op: MBackfillReserve GRANT pgid: 2.1bb, query_epoch: 11709 -4474> 2014-09-18 18:08:31.487531 7f4c9f251700 5 -- op tracker -- , seq: 86013, time: 2014-09-18 18:08:31.487530, event: done, op: MBackfillReserve GRANT pgid: 2.1bb, query_epoch: 11709 -4473> 2014-09-18 18:08:31.487550 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- MBackfillReserve REQUEST pgid: 2.1bb, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf80e5a0 con 0x1395f8c0 -4472> 2014-09-18 18:08:31.488198 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 938 ==== MBackfillReserve REJECT pgid: 2.1bb, query_epoch: 11709 v3 ==== 30+0+0 (2687497417 0 0) 0xf80e5a0 con 0x1395f8c0 -4471> 2014-09-18 18:08:31.488218 7f4c9f251700 5 -- op tracker -- , seq: 86014, time: 2014-09-18 18:08:31.488100, event: header_read, op: MBackfillReserve REJECT pgid: 2.1bb, query_epoch: 11709 -4470> 2014-09-18 18:08:31.488228 7f4c9f251700 5 -- op tracker -- , seq: 86014, time: 2014-09-18 18:08:31.488101, event: throttled, op: MBackfillReserve REJECT pgid: 2.1bb, query_epoch: 11709 -4469> 2014-09-18 18:08:31.488234 7f4c9f251700 5 -- op tracker -- , seq: 86014, time: 2014-09-18 18:08:31.488135, event: all_read, op: MBackfillReserve REJECT pgid: 2.1bb, query_epoch: 11709 -4468> 2014-09-18 18:08:31.488240 7f4c9f251700 5 -- op tracker -- , seq: 86014, time: 2014-09-18 18:08:31.488213, event: dispatched, op: MBackfillReserve REJECT pgid: 2.1bb, query_epoch: 11709 -4467> 2014-09-18 18:08:31.488247 7f4c9f251700 5 -- op tracker -- , seq: 86014, time: 2014-09-18 18:08:31.488245, event: waiting_for_osdmap, op: MBackfillReserve REJECT pgid: 2.1bb, query_epoch: 11709 -4466> 2014-09-18 18:08:31.488261 7f4c9f251700 5 -- op tracker -- , seq: 86014, time: 2014-09-18 18:08:31.488260, event: done, op: MBackfillReserve REJECT pgid: 2.1bb, query_epoch: 11709 -4465> 2014-09-18 18:08:31.488309 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- MBackfillReserve REJECT pgid: 2.1bb, query_epoch: 11709 v3 -- ?+0 0x24fd1c20 con 0x10a878c0 -4464> 2014-09-18 18:08:31.488332 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.2c7( v 6805'24696 (4143'21695,6805'24696] local-les=11430 n=499 ec=188 les/c 11430/9946 11429/11429/10779) [27,42,0]/[42] r=0 lpr=11429 pi=3160-11428/139 bft=0,27 crt=6788'24687 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 141.695906 520 0.002473 -4463> 2014-09-18 18:08:31.488414 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.2c7( v 6805'24696 (4143'21695,6805'24696] local-les=11430 n=499 ec=188 les/c 11430/9946 11429/11429/10779) [27,42,0]/[42] r=0 lpr=11429 pi=3160-11428/139 bft=0,27 crt=6788'24687 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved -4462> 2014-09-18 18:08:31.488429 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- MBackfillReserve REQUEST pgid: 2.2c7, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf1e9a20 con 0x10a878c0 -4461> 2014-09-18 18:08:31.488328 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.1bb( v 6791'24086 (4155'21085,6791'24086] local-les=11407 n=551 ec=188 les/c 11407/9946 11406/11406/9945) [42,27,0]/[42] r=0 lpr=11406 pi=9945-11405/9 bft=0,27 crt=6752'24077 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 1.764584 8 0.000213 -4460> 2014-09-18 18:08:31.488477 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.1bb( v 6791'24086 (4155'21085,6791'24086] local-les=11407 n=551 ec=188 les/c 11407/9946 11406/11406/9945) [42,27,0]/[42] r=0 lpr=11406 pi=9945-11405/9 bft=0,27 crt=6752'24077 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling -4459> 2014-09-18 18:08:31.488816 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 966 ==== MBackfillReserve GRANT pgid: 2.e0, query_epoch: 11709 v3 ==== 30+0+0 (1837258881 0 0) 0xf1e9a20 con 0x10a878c0 -4458> 2014-09-18 18:08:31.488835 7f4c9f251700 5 -- op tracker -- , seq: 86015, time: 2014-09-18 18:08:31.488775, event: header_read, op: MBackfillReserve GRANT pgid: 2.e0, query_epoch: 11709 -4457> 2014-09-18 18:08:31.488845 7f4c9f251700 5 -- op tracker -- , seq: 86015, time: 2014-09-18 18:08:31.488776, event: throttled, op: MBackfillReserve GRANT pgid: 2.e0, query_epoch: 11709 -4456> 2014-09-18 18:08:31.488851 7f4c9f251700 5 -- op tracker -- , seq: 86015, time: 2014-09-18 18:08:31.488789, event: all_read, op: MBackfillReserve GRANT pgid: 2.e0, query_epoch: 11709 -4455> 2014-09-18 18:08:31.488857 7f4c9f251700 5 -- op tracker -- , seq: 86015, time: 2014-09-18 18:08:31.488831, event: dispatched, op: MBackfillReserve GRANT pgid: 2.e0, query_epoch: 11709 -4454> 2014-09-18 18:08:31.488863 7f4c9f251700 5 -- op tracker -- , seq: 86015, time: 2014-09-18 18:08:31.488862, event: waiting_for_osdmap, op: MBackfillReserve GRANT pgid: 2.e0, query_epoch: 11709 -4453> 2014-09-18 18:08:31.488877 7f4c9f251700 5 -- op tracker -- , seq: 86015, time: 2014-09-18 18:08:31.488876, event: done, op: MBackfillReserve GRANT pgid: 2.e0, query_epoch: 11709 -4452> 2014-09-18 18:08:31.488884 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- MBackfillReserve REQUEST pgid: 2.e0, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf1e9840 con 0x1395f8c0 -4451> 2014-09-18 18:08:31.489330 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 939 ==== MBackfillReserve REJECT pgid: 2.e0, query_epoch: 11709 v3 ==== 30+0+0 (1949127848 0 0) 0xf1e9840 con 0x1395f8c0 -4450> 2014-09-18 18:08:31.489349 7f4c9f251700 5 -- op tracker -- , seq: 86016, time: 2014-09-18 18:08:31.489265, event: header_read, op: MBackfillReserve REJECT pgid: 2.e0, query_epoch: 11709 -4449> 2014-09-18 18:08:31.489359 7f4c9f251700 5 -- op tracker -- , seq: 86016, time: 2014-09-18 18:08:31.489266, event: throttled, op: MBackfillReserve REJECT pgid: 2.e0, query_epoch: 11709 -4448> 2014-09-18 18:08:31.489365 7f4c9f251700 5 -- op tracker -- , seq: 86016, time: 2014-09-18 18:08:31.489283, event: all_read, op: MBackfillReserve REJECT pgid: 2.e0, query_epoch: 11709 -4447> 2014-09-18 18:08:31.489371 7f4c9f251700 5 -- op tracker -- , seq: 86016, time: 2014-09-18 18:08:31.489345, event: dispatched, op: MBackfillReserve REJECT pgid: 2.e0, query_epoch: 11709 -4446> 2014-09-18 18:08:31.489377 7f4c9f251700 5 -- op tracker -- , seq: 86016, time: 2014-09-18 18:08:31.489376, event: waiting_for_osdmap, op: MBackfillReserve REJECT pgid: 2.e0, query_epoch: 11709 -4445> 2014-09-18 18:08:31.489391 7f4c9f251700 5 -- op tracker -- , seq: 86016, time: 2014-09-18 18:08:31.489389, event: done, op: MBackfillReserve REJECT pgid: 2.e0, query_epoch: 11709 -4444> 2014-09-18 18:08:31.489439 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- MBackfillReserve REJECT pgid: 2.e0, query_epoch: 11709 v3 -- ?+0 0x21e103c0 con 0x10a878c0 -4443> 2014-09-18 18:08:31.489487 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.719( v 6842'29696 (4073'25919,6842'29696] local-les=11531 n=513 ec=188 les/c 11531/9946 11530/11530/9945) [42,27,48]/[42] r=0 lpr=11530 pi=9945-11529/17 bft=27,48 crt=6842'29696 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 141.696956 624 0.004440 -4442> 2014-09-18 18:08:31.489503 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.719( v 6842'29696 (4073'25919,6842'29696] local-les=11531 n=513 ec=188 les/c 11531/9946 11530/11530/9945) [42,27,48]/[42] r=0 lpr=11530 pi=9945-11529/17 bft=27,48 crt=6842'29696 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved -4441> 2014-09-18 18:08:31.489515 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- MBackfillReserve REQUEST pgid: 2.719, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf1e6d20 con 0x1395f8c0 -4440> 2014-09-18 18:08:31.489551 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.e0( v 6854'28782 (4157'25783,6854'28782] local-les=11407 n=509 ec=188 les/c 11407/9946 11406/11406/10777) [27,42,0]/[42] r=0 lpr=11406 pi=5117-11405/79 bft=0,27 crt=6854'28778 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 1.765443 6 0.000086 -4439> 2014-09-18 18:08:31.489570 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.e0( v 6854'28782 (4157'25783,6854'28782] local-les=11407 n=509 ec=188 les/c 11407/9946 11406/11406/10777) [27,42,0]/[42] r=0 lpr=11406 pi=5117-11405/79 bft=0,27 crt=6854'28778 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling -4438> 2014-09-18 18:08:31.490003 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 940 ==== MBackfillReserve REJECT pgid: 2.719, query_epoch: 11709 v3 ==== 30+0+0 (877081962 0 0) 0xf1e6d20 con 0x1395f8c0 -4437> 2014-09-18 18:08:31.490022 7f4c9f251700 5 -- op tracker -- , seq: 86017, time: 2014-09-18 18:08:31.489902, event: header_read, op: MBackfillReserve REJECT pgid: 2.719, query_epoch: 11709 -4436> 2014-09-18 18:08:31.490033 7f4c9f251700 5 -- op tracker -- , seq: 86017, time: 2014-09-18 18:08:31.489903, event: throttled, op: MBackfillReserve REJECT pgid: 2.719, query_epoch: 11709 -4435> 2014-09-18 18:08:31.490039 7f4c9f251700 5 -- op tracker -- , seq: 86017, time: 2014-09-18 18:08:31.489934, event: all_read, op: MBackfillReserve REJECT pgid: 2.719, query_epoch: 11709 -4434> 2014-09-18 18:08:31.490045 7f4c9f251700 5 -- op tracker -- , seq: 86017, time: 2014-09-18 18:08:31.490018, event: dispatched, op: MBackfillReserve REJECT pgid: 2.719, query_epoch: 11709 -4433> 2014-09-18 18:08:31.490051 7f4c9f251700 5 -- op tracker -- , seq: 86017, time: 2014-09-18 18:08:31.490050, event: waiting_for_osdmap, op: MBackfillReserve REJECT pgid: 2.719, query_epoch: 11709 -4432> 2014-09-18 18:08:31.490066 7f4c9f251700 5 -- op tracker -- , seq: 86017, time: 2014-09-18 18:08:31.490064, event: done, op: MBackfillReserve REJECT pgid: 2.719, query_epoch: 11709 -4431> 2014-09-18 18:08:31.490098 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.719( v 6842'29696 (4073'25919,6842'29696] local-les=11531 n=513 ec=188 les/c 11531/9946 11530/11530/9945) [42,27,48]/[42] r=0 lpr=11530 pi=9945-11529/17 bft=27,48 crt=6842'29696 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.000595 1 0.000039 -4430> 2014-09-18 18:08:31.490113 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.719( v 6842'29696 (4073'25919,6842'29696] local-les=11531 n=513 ec=188 les/c 11531/9946 11530/11530/9945) [42,27,48]/[42] r=0 lpr=11530 pi=9945-11529/17 bft=27,48 crt=6842'29696 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling -4429> 2014-09-18 18:08:31.490122 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 141.696720 521 0.002379 -4428> 2014-09-18 18:08:31.490146 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved -4427> 2014-09-18 18:08:31.490169 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- MBackfillReserve REQUEST pgid: 2.2d8, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf867c20 con 0x1395ec60 -4426> 2014-09-18 18:08:31.490621 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3207 ==== MBackfillReserve REJECT pgid: 2.2d8, query_epoch: 11709 v3 ==== 30+0+0 (117260603 0 0) 0xf867c20 con 0x1395ec60 -4425> 2014-09-18 18:08:31.490641 7f4c9f251700 5 -- op tracker -- , seq: 86018, time: 2014-09-18 18:08:31.490547, event: header_read, op: MBackfillReserve REJECT pgid: 2.2d8, query_epoch: 11709 -4424> 2014-09-18 18:08:31.490651 7f4c9f251700 5 -- op tracker -- , seq: 86018, time: 2014-09-18 18:08:31.490548, event: throttled, op: MBackfillReserve REJECT pgid: 2.2d8, query_epoch: 11709 -4423> 2014-09-18 18:08:31.490657 7f4c9f251700 5 -- op tracker -- , seq: 86018, time: 2014-09-18 18:08:31.490582, event: all_read, op: MBackfillReserve REJECT pgid: 2.2d8, query_epoch: 11709 -4422> 2014-09-18 18:08:31.490663 7f4c9f251700 5 -- op tracker -- , seq: 86018, time: 2014-09-18 18:08:31.490637, event: dispatched, op: MBackfillReserve REJECT pgid: 2.2d8, query_epoch: 11709 -4421> 2014-09-18 18:08:31.490670 7f4c9f251700 5 -- op tracker -- , seq: 86018, time: 2014-09-18 18:08:31.490668, event: waiting_for_osdmap, op: MBackfillReserve REJECT pgid: 2.2d8, query_epoch: 11709 -4420> 2014-09-18 18:08:31.490684 7f4c9f251700 5 -- op tracker -- , seq: 86018, time: 2014-09-18 18:08:31.490683, event: done, op: MBackfillReserve REJECT pgid: 2.2d8, query_epoch: 11709 -4419> 2014-09-18 18:08:31.490697 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.000551 1 0.000063 -4418> 2014-09-18 18:08:31.490711 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling -4417> 2014-09-18 18:08:31.490727 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.491( v 6842'30456 (4176'28325,6842'30456] local-les=11484 n=502 ec=188 les/c 11484/9946 11483/11483/9945) [42,20,32]/[42] r=0 lpr=11483 pi=9945-11482/7 bft=20,32 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 141.696891 313 0.001349 -4416> 2014-09-18 18:08:31.490745 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.491( v 6842'30456 (4176'28325,6842'30456] local-les=11484 n=502 ec=188 les/c 11484/9946 11483/11483/9945) [42,20,32]/[42] r=0 lpr=11483 pi=9945-11482/7 bft=20,32 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved -4415> 2014-09-18 18:08:31.490802 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6822/33871 -- MBackfillReserve REQUEST pgid: 2.491, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf868760 con 0x4900000 -4414> 2014-09-18 18:08:31.491302 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1727 ==== MBackfillReserve GRANT pgid: 2.491, query_epoch: 11709 v3 ==== 30+0+0 (2233994826 0 0) 0xf868760 con 0x4900000 -4413> 2014-09-18 18:08:31.491322 7f4c9f251700 5 -- op tracker -- , seq: 86019, time: 2014-09-18 18:08:31.491232, event: header_read, op: MBackfillReserve GRANT pgid: 2.491, query_epoch: 11709 -4412> 2014-09-18 18:08:31.491332 7f4c9f251700 5 -- op tracker -- , seq: 86019, time: 2014-09-18 18:08:31.491233, event: throttled, op: MBackfillReserve GRANT pgid: 2.491, query_epoch: 11709 -4411> 2014-09-18 18:08:31.491339 7f4c9f251700 5 -- op tracker -- , seq: 86019, time: 2014-09-18 18:08:31.491266, event: all_read, op: MBackfillReserve GRANT pgid: 2.491, query_epoch: 11709 -4410> 2014-09-18 18:08:31.491345 7f4c9f251700 5 -- op tracker -- , seq: 86019, time: 2014-09-18 18:08:31.491317, event: dispatched, op: MBackfillReserve GRANT pgid: 2.491, query_epoch: 11709 -4409> 2014-09-18 18:08:31.491351 7f4c9f251700 5 -- op tracker -- , seq: 86019, time: 2014-09-18 18:08:31.491350, event: waiting_for_osdmap, op: MBackfillReserve GRANT pgid: 2.491, query_epoch: 11709 -4408> 2014-09-18 18:08:31.491366 7f4c9f251700 5 -- op tracker -- , seq: 86019, time: 2014-09-18 18:08:31.491364, event: done, op: MBackfillReserve GRANT pgid: 2.491, query_epoch: 11709 -4407> 2014-09-18 18:08:31.491388 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- MBackfillReserve REQUEST pgid: 2.491, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf1e7680 con 0x1395ef20 -4406> 2014-09-18 18:08:31.491721 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 537 ==== MBackfillReserve REJECT pgid: 2.491, query_epoch: 11709 v3 ==== 30+0+0 (2626093667 0 0) 0xf1e7680 con 0x1395ef20 -4405> 2014-09-18 18:08:31.491740 7f4c9f251700 5 -- op tracker -- , seq: 86020, time: 2014-09-18 18:08:31.491667, event: header_read, op: MBackfillReserve REJECT pgid: 2.491, query_epoch: 11709 -4404> 2014-09-18 18:08:31.491763 7f4c9f251700 5 -- op tracker -- , seq: 86020, time: 2014-09-18 18:08:31.491668, event: throttled, op: MBackfillReserve REJECT pgid: 2.491, query_epoch: 11709 -4403> 2014-09-18 18:08:31.491770 7f4c9f251700 5 -- op tracker -- , seq: 86020, time: 2014-09-18 18:08:31.491703, event: all_read, op: MBackfillReserve REJECT pgid: 2.491, query_epoch: 11709 -4402> 2014-09-18 18:08:31.491775 7f4c9f251700 5 -- op tracker -- , seq: 86020, time: 2014-09-18 18:08:31.491735, event: dispatched, op: MBackfillReserve REJECT pgid: 2.491, query_epoch: 11709 -4401> 2014-09-18 18:08:31.491782 7f4c9f251700 5 -- op tracker -- , seq: 86020, time: 2014-09-18 18:08:31.491781, event: waiting_for_osdmap, op: MBackfillReserve REJECT pgid: 2.491, query_epoch: 11709 -4400> 2014-09-18 18:08:31.491797 7f4c9f251700 5 -- op tracker -- , seq: 86020, time: 2014-09-18 18:08:31.491795, event: done, op: MBackfillReserve REJECT pgid: 2.491, query_epoch: 11709 -4399> 2014-09-18 18:08:31.491814 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6822/33871 -- MBackfillReserve REJECT pgid: 2.491, query_epoch: 11709 v3 -- ?+0 0xf867fe0 con 0x4900000 -4398> 2014-09-18 18:08:31.491850 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.40a( v 6791'27848 (4153'24489,6791'27848] local-les=11485 n=526 ec=188 les/c 11485/9946 11483/11483/10766) [2,32,42]/[42] r=0 lpr=11483 pi=188-11482/162 bft=2,32 crt=6791'27848 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 141.013999 559 0.002152 -4397> 2014-09-18 18:08:31.491868 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.40a( v 6791'27848 (4153'24489,6791'27848] local-les=11485 n=526 ec=188 les/c 11485/9946 11483/11483/10766) [2,32,42]/[42] r=0 lpr=11483 pi=188-11482/162 bft=2,32 crt=6791'27848 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved -4396> 2014-09-18 18:08:31.491880 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6817/19139 -- MBackfillReserve REQUEST pgid: 2.40a, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf1e7e00 con 0x10f60520 -4395> 2014-09-18 18:08:31.491878 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.491( v 6842'30456 (4176'28325,6842'30456] local-les=11484 n=502 ec=188 les/c 11484/9946 11483/11483/9945) [42,20,32]/[42] r=0 lpr=11483 pi=9945-11482/7 bft=20,32 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.001133 2 0.000105 -4394> 2014-09-18 18:08:31.491894 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.491( v 6842'30456 (4176'28325,6842'30456] local-les=11484 n=502 ec=188 les/c 11484/9946 11483/11483/9945) [42,20,32]/[42] r=0 lpr=11483 pi=9945-11482/7 bft=20,32 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling -4393> 2014-09-18 18:08:31.493460 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 967 ==== MBackfillReserve GRANT pgid: 2.2c7, query_epoch: 11709 v3 ==== 30+0+0 (2224098601 0 0) 0x21e103c0 con 0x10a878c0 -4392> 2014-09-18 18:08:31.493479 7f4c9f251700 5 -- op tracker -- , seq: 86021, time: 2014-09-18 18:08:31.493414, event: header_read, op: MBackfillReserve GRANT pgid: 2.2c7, query_epoch: 11709 -4391> 2014-09-18 18:08:31.493490 7f4c9f251700 5 -- op tracker -- , seq: 86021, time: 2014-09-18 18:08:31.493415, event: throttled, op: MBackfillReserve GRANT pgid: 2.2c7, query_epoch: 11709 -4390> 2014-09-18 18:08:31.493496 7f4c9f251700 5 -- op tracker -- , seq: 86021, time: 2014-09-18 18:08:31.493430, event: all_read, op: MBackfillReserve GRANT pgid: 2.2c7, query_epoch: 11709 -4389> 2014-09-18 18:08:31.493501 7f4c9f251700 5 -- op tracker -- , seq: 86021, time: 2014-09-18 18:08:31.493475, event: dispatched, op: MBackfillReserve GRANT pgid: 2.2c7, query_epoch: 11709 -4388> 2014-09-18 18:08:31.493508 7f4c9f251700 5 -- op tracker -- , seq: 86021, time: 2014-09-18 18:08:31.493506, event: waiting_for_osdmap, op: MBackfillReserve GRANT pgid: 2.2c7, query_epoch: 11709 -4387> 2014-09-18 18:08:31.493522 7f4c9f251700 5 -- op tracker -- , seq: 86021, time: 2014-09-18 18:08:31.493520, event: done, op: MBackfillReserve GRANT pgid: 2.2c7, query_epoch: 11709 -4386> 2014-09-18 18:08:31.493561 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- MBackfillReserve REQUEST pgid: 2.2c7, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf1e65a0 con 0x1395f8c0 -4385> 2014-09-18 18:08:31.494174 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 941 ==== MBackfillReserve REJECT pgid: 2.2c7, query_epoch: 11709 v3 ==== 30+0+0 (2638086400 0 0) 0xf1e65a0 con 0x1395f8c0 -4384> 2014-09-18 18:08:31.494193 7f4c9f251700 5 -- op tracker -- , seq: 86022, time: 2014-09-18 18:08:31.494056, event: header_read, op: MBackfillReserve REJECT pgid: 2.2c7, query_epoch: 11709 -4383> 2014-09-18 18:08:31.494203 7f4c9f251700 5 -- op tracker -- , seq: 86022, time: 2014-09-18 18:08:31.494057, event: throttled, op: MBackfillReserve REJECT pgid: 2.2c7, query_epoch: 11709 -4382> 2014-09-18 18:08:31.494210 7f4c9f251700 5 -- op tracker -- , seq: 86022, time: 2014-09-18 18:08:31.494074, event: all_read, op: MBackfillReserve REJECT pgid: 2.2c7, query_epoch: 11709 -4381> 2014-09-18 18:08:31.494215 7f4c9f251700 5 -- op tracker -- , seq: 86022, time: 2014-09-18 18:08:31.494189, event: dispatched, op: MBackfillReserve REJECT pgid: 2.2c7, query_epoch: 11709 -4380> 2014-09-18 18:08:31.494222 7f4c9f251700 5 -- op tracker -- , seq: 86022, time: 2014-09-18 18:08:31.494220, event: waiting_for_osdmap, op: MBackfillReserve REJECT pgid: 2.2c7, query_epoch: 11709 -4379> 2014-09-18 18:08:31.494236 7f4c9f251700 5 -- op tracker -- , seq: 86022, time: 2014-09-18 18:08:31.494234, event: done, op: MBackfillReserve REJECT pgid: 2.2c7, query_epoch: 11709 -4378> 2014-09-18 18:08:31.494311 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- MBackfillReserve REJECT pgid: 2.2c7, query_epoch: 11709 v3 -- ?+0 0x1213ad00 con 0x10a878c0 -4377> 2014-09-18 18:08:31.494327 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.2c7( v 6805'24696 (4143'21695,6805'24696] local-les=11430 n=499 ec=188 les/c 11430/9946 11429/11429/10779) [27,42,0]/[42] r=0 lpr=11429 pi=3160-11428/139 bft=0,27 crt=6788'24687 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.005913 2 0.000162 -4376> 2014-09-18 18:08:31.494342 7f4c96a40700 5 osd.42 pg_epoch: 11709 pg[2.2c7( v 6805'24696 (4143'21695,6805'24696] local-les=11430 n=499 ec=188 les/c 11430/9946 11429/11429/10779) [27,42,0]/[42] r=0 lpr=11429 pi=3160-11428/139 bft=0,27 crt=6788'24687 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling -4375> 2014-09-18 18:08:31.494374 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.17e( v 6789'26580 (4153'23579,6789'26580] local-les=11485 n=538 ec=188 les/c 11485/9946 11483/11483/10661) [6,42,30]/[42] r=0 lpr=11483 pi=9903-11482/17 bft=6,30 crt=6775'26571 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 140.532001 521 0.001954 -4374> 2014-09-18 18:08:31.494396 7f4c97241700 5 osd.42 pg_epoch: 11709 pg[2.17e( v 6789'26580 (4153'23579,6789'26580] local-les=11485 n=538 ec=188 les/c 11485/9946 11483/11483/10661) [6,42,30]/[42] r=0 lpr=11483 pi=9903-11482/17 bft=6,30 crt=6775'26571 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved -4373> 2014-09-18 18:08:31.494411 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- MBackfillReserve REQUEST pgid: 2.17e, query_epoch: 11709, prio: 202 v3 -- ?+0 0xf9730e0 con 0x1088aec0 -4372> 2014-09-18 18:08:31.520813 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.52 10.242.42.168:0/16848 130 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.387484) v2 ==== 47+0+0 (3497654123 0 0) 0x12112780 con 0xcc49b80 -4371> 2014-09-18 18:08:31.520814 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.52 10.242.42.168:0/16848 130 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.387484) v2 ==== 47+0+0 (3497654123 0 0) 0xf106f00 con 0x101cc520 -4370> 2014-09-18 18:08:31.520855 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16848 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.387484) v2 -- ?+0 0xe76e940 con 0xcc49b80 -4369> 2014-09-18 18:08:31.520885 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16848 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.387484) v2 -- ?+0 0xf09be00 con 0x101cc520 -4368> 2014-09-18 18:08:31.538461 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=93 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).connect error 10.242.42.164:6854/39317, (111) Connection refused -4367> 2014-09-18 18:08:31.538486 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=93 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).fault (111) Connection refused -4366> 2014-09-18 18:08:31.588725 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.35 10.242.42.164:0/17421 165 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.587411) v2 ==== 47+0+0 (455333479 0 0) 0xf099840 con 0x1d5bcd60 -4365> 2014-09-18 18:08:31.588740 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.35 10.242.42.164:0/17421 165 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.587411) v2 ==== 47+0+0 (455333479 0 0) 0xf09c3a0 con 0x142e9700 -4364> 2014-09-18 18:08:31.588786 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/17421 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.587411) v2 -- ?+0 0xf106f00 con 0x1d5bcd60 -4363> 2014-09-18 18:08:31.588822 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/17421 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.587411) v2 -- ?+0 0x12112780 con 0x142e9700 -4362> 2014-09-18 18:08:31.589251 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.6 10.242.42.166:0/22481 441 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:31.591238) v2 ==== 47+0+0 (4251231948 0 0) 0x100ef4a0 con 0x14393080 -4361> 2014-09-18 18:08:31.589267 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.6 10.242.42.166:0/22481 441 ==== osd_ping(ping e11707 stamp 2014-09-18 18:08:31.591238) v2 ==== 47+0+0 (4251231948 0 0) 0x12c12d00 con 0x10201700 -4360> 2014-09-18 18:08:31.589273 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/22481 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.591238) v2 -- ?+0 0xf099840 con 0x14393080 -4359> 2014-09-18 18:08:31.589323 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/22481 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.591238) v2 -- ?+0 0xf09c3a0 con 0x10201700 -4358> 2014-09-18 18:08:31.649755 7f4c98243700 10 monclient: tick -4357> 2014-09-18 18:08:31.649776 7f4c98243700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-09-18 18:08:01.649774) -4356> 2014-09-18 18:08:31.649793 7f4c98243700 10 monclient: renew subs? (now: 2014-09-18 18:08:31.649793; renew after: 2014-09-18 18:10:11.726903) -- no -4355> 2014-09-18 18:08:31.657175 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.45 10.242.42.178:0/7086 106 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.656501) v2 ==== 47+0+0 (161731474 0 0) 0xf811c00 con 0x110bb700 -4354> 2014-09-18 18:08:31.657214 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.656501) v2 -- ?+0 0x12c12d00 con 0x110bb700 -4353> 2014-09-18 18:08:31.657174 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.45 10.242.42.178:0/7086 106 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.656501) v2 ==== 47+0+0 (161731474 0 0) 0xf1e8940 con 0x10a5d9c0 -4352> 2014-09-18 18:08:31.657250 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:31.656501) v2 -- ?+0 0x100ef4a0 con 0x10a5d9c0 -4351> 2014-09-18 18:08:31.673967 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 29 ==== osd_map(11709..11710 src has 1..11710) v3 ==== 1707+0+0 (1567767961 0 0) 0x24d1a480 con 0x111771e0 -4350> 2014-09-18 18:08:31.674043 7f4ca0a54700 3 osd.42 11709 handle_osd_map epochs [11709,11710], i have 11709, src has [1,11710] -4349> 2014-09-18 18:08:31.677332 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0xe937440 con 0x1395ec60 -4348> 2014-09-18 18:08:31.677341 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0x24dbef40 con 0x140f95a0 -4347> 2014-09-18 18:08:31.677351 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(0.12(27) epoch 11710) v5 -- ?+0 0x2b986780 con 0x1395ec60 -4346> 2014-09-18 18:08:31.677357 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11710) v5 -- ?+0 0x2b93d480 con 0x140f95a0 -4345> 2014-09-18 18:08:31.677402 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11710) v5 -- ?+0 0x142201c0 con 0x140f95a0 -4344> 2014-09-18 18:08:31.677451 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0x19df2f40 con 0x1395edc0 -4343> 2014-09-18 18:08:31.677469 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11710) v5 -- ?+0 0xf29c1c0 con 0x1395edc0 -4342> 2014-09-18 18:08:31.677507 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0x19df33c0 con 0x1395f1e0 -4341> 2014-09-18 18:08:31.677519 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11710) v5 -- ?+0 0x117383c0 con 0x1395f1e0 -4340> 2014-09-18 18:08:31.677991 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0x19df3180 con 0xcbecec0 -4339> 2014-09-18 18:08:31.678005 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(11) epoch 11710) v5 -- ?+0 0x21cfa5a0 con 0xcbecec0 -4338> 2014-09-18 18:08:31.678525 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0xe339180 con 0x23a74f20 -4337> 2014-09-18 18:08:31.678545 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11710) v5 -- ?+0 0x21e12940 con 0x23a74f20 -4336> 2014-09-18 18:08:31.679962 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1266 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x12bf2d00 con 0xe8db760 -4335> 2014-09-18 18:08:31.679979 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4334> 2014-09-18 18:08:31.680019 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1267 ==== pg_notify(2.17e(17) epoch 11710) v5 ==== 1593+0+0 (1636289277 0 0) 0xf0db680 con 0xe8db760 -4333> 2014-09-18 18:08:31.680038 7f4c9f251700 5 -- op tracker -- , seq: 86023, time: 2014-09-18 18:08:31.679960, event: header_read, op: pg_notify(2.17e(17) epoch 11710) -4332> 2014-09-18 18:08:31.680049 7f4c9f251700 5 -- op tracker -- , seq: 86023, time: 2014-09-18 18:08:31.679961, event: throttled, op: pg_notify(2.17e(17) epoch 11710) -4331> 2014-09-18 18:08:31.680058 7f4c9f251700 5 -- op tracker -- , seq: 86023, time: 2014-09-18 18:08:31.680006, event: all_read, op: pg_notify(2.17e(17) epoch 11710) -4330> 2014-09-18 18:08:31.680065 7f4c9f251700 5 -- op tracker -- , seq: 86023, time: 2014-09-18 18:08:31.680033, event: dispatched, op: pg_notify(2.17e(17) epoch 11710) -4329> 2014-09-18 18:08:31.680072 7f4c9f251700 5 -- op tracker -- , seq: 86023, time: 2014-09-18 18:08:31.680070, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11710) -4328> 2014-09-18 18:08:31.680089 7f4c9f251700 5 -- op tracker -- , seq: 86023, time: 2014-09-18 18:08:31.680087, event: started, op: pg_notify(2.17e(17) epoch 11710) -4327> 2014-09-18 18:08:31.680107 7f4c9f251700 5 -- op tracker -- , seq: 86023, time: 2014-09-18 18:08:31.680106, event: done, op: pg_notify(2.17e(17) epoch 11710) -4326> 2014-09-18 18:08:31.683612 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 538 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x1533bb00 con 0x1395ef20 -4325> 2014-09-18 18:08:31.683628 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4324> 2014-09-18 18:08:31.683658 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 539 ==== pg_notify(2.4d0(10) epoch 11710) v5 ==== 1292+0+0 (1653278981 0 0) 0xf4e6d20 con 0x1395ef20 -4323> 2014-09-18 18:08:31.683676 7f4c9f251700 5 -- op tracker -- , seq: 86024, time: 2014-09-18 18:08:31.683612, event: header_read, op: pg_notify(2.4d0(10) epoch 11710) -4322> 2014-09-18 18:08:31.683691 7f4c9f251700 5 -- op tracker -- , seq: 86024, time: 2014-09-18 18:08:31.683612, event: throttled, op: pg_notify(2.4d0(10) epoch 11710) -4321> 2014-09-18 18:08:31.683703 7f4c9f251700 5 -- op tracker -- , seq: 86024, time: 2014-09-18 18:08:31.683644, event: all_read, op: pg_notify(2.4d0(10) epoch 11710) -4320> 2014-09-18 18:08:31.683714 7f4c9f251700 5 -- op tracker -- , seq: 86024, time: 2014-09-18 18:08:31.683671, event: dispatched, op: pg_notify(2.4d0(10) epoch 11710) -4319> 2014-09-18 18:08:31.683726 7f4c9f251700 5 -- op tracker -- , seq: 86024, time: 2014-09-18 18:08:31.683723, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11710) -4318> 2014-09-18 18:08:31.683748 7f4c9f251700 5 -- op tracker -- , seq: 86024, time: 2014-09-18 18:08:31.683745, event: started, op: pg_notify(2.4d0(10) epoch 11710) -4317> 2014-09-18 18:08:31.683774 7f4c9f251700 5 -- op tracker -- , seq: 86024, time: 2014-09-18 18:08:31.683772, event: done, op: pg_notify(2.4d0(10) epoch 11710) -4316> 2014-09-18 18:08:31.684536 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1198 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x1182b200 con 0x1088b440 -4315> 2014-09-18 18:08:31.684553 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4314> 2014-09-18 18:08:31.684622 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1042 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0xd7c4d00 con 0x14358260 -4313> 2014-09-18 18:08:31.684632 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4312> 2014-09-18 18:08:31.684704 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1043 ==== pg_notify(2.719(17) epoch 11710) v5 ==== 1601+0+0 (2494209801 0 0) 0xf815660 con 0x14358260 -4311> 2014-09-18 18:08:31.684719 7f4c9f251700 5 -- op tracker -- , seq: 86025, time: 2014-09-18 18:08:31.684634, event: header_read, op: pg_notify(2.719(17) epoch 11710) -4310> 2014-09-18 18:08:31.684728 7f4c9f251700 5 -- op tracker -- , seq: 86025, time: 2014-09-18 18:08:31.684635, event: throttled, op: pg_notify(2.719(17) epoch 11710) -4309> 2014-09-18 18:08:31.684735 7f4c9f251700 5 -- op tracker -- , seq: 86025, time: 2014-09-18 18:08:31.684690, event: all_read, op: pg_notify(2.719(17) epoch 11710) -4308> 2014-09-18 18:08:31.684741 7f4c9f251700 5 -- op tracker -- , seq: 86025, time: 2014-09-18 18:08:31.684714, event: dispatched, op: pg_notify(2.719(17) epoch 11710) -4307> 2014-09-18 18:08:31.684747 7f4c9f251700 5 -- op tracker -- , seq: 86025, time: 2014-09-18 18:08:31.684746, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11710) -4306> 2014-09-18 18:08:31.684755 7f4c9f251700 5 -- op tracker -- , seq: 86025, time: 2014-09-18 18:08:31.684753, event: started, op: pg_notify(2.719(17) epoch 11710) -4305> 2014-09-18 18:08:31.684774 7f4c9f251700 5 -- op tracker -- , seq: 86025, time: 2014-09-18 18:08:31.684772, event: done, op: pg_notify(2.719(17) epoch 11710) -4304> 2014-09-18 18:08:31.684790 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1199 ==== pg_notify(2.2d8(173) epoch 11710) v5 ==== 10757+0+0 (2589711571 0 0) 0xec6bde0 con 0x1088b440 -4303> 2014-09-18 18:08:31.684802 7f4c9f251700 5 -- op tracker -- , seq: 86026, time: 2014-09-18 18:08:31.684612, event: header_read, op: pg_notify(2.2d8(173) epoch 11710) -4302> 2014-09-18 18:08:31.684809 7f4c9f251700 5 -- op tracker -- , seq: 86026, time: 2014-09-18 18:08:31.684613, event: throttled, op: pg_notify(2.2d8(173) epoch 11710) -4301> 2014-09-18 18:08:31.684816 7f4c9f251700 5 -- op tracker -- , seq: 86026, time: 2014-09-18 18:08:31.684771, event: all_read, op: pg_notify(2.2d8(173) epoch 11710) -4300> 2014-09-18 18:08:31.684822 7f4c9f251700 5 -- op tracker -- , seq: 86026, time: 2014-09-18 18:08:31.684799, event: dispatched, op: pg_notify(2.2d8(173) epoch 11710) -4299> 2014-09-18 18:08:31.684828 7f4c9f251700 5 -- op tracker -- , seq: 86026, time: 2014-09-18 18:08:31.684827, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11710) -4298> 2014-09-18 18:08:31.684836 7f4c9f251700 5 -- op tracker -- , seq: 86026, time: 2014-09-18 18:08:31.684834, event: started, op: pg_notify(2.2d8(173) epoch 11710) -4297> 2014-09-18 18:08:31.684852 7f4c9f251700 5 -- op tracker -- , seq: 86026, time: 2014-09-18 18:08:31.684851, event: done, op: pg_notify(2.2d8(173) epoch 11710) -4296> 2014-09-18 18:08:31.686349 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.59 10.242.42.178:6840/4508 4 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0xef578c0 con 0x23af51e0 -4295> 2014-09-18 18:08:31.686363 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4294> 2014-09-18 18:08:31.686388 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.59 10.242.42.178:6840/4508 5 ==== pg_notify(2.4e8(7) epoch 11710) v5 ==== 1139+0+0 (414186613 0 0) 0xf6dffe0 con 0x23af51e0 -4293> 2014-09-18 18:08:31.686399 7f4c9f251700 5 -- op tracker -- , seq: 86027, time: 2014-09-18 18:08:31.686348, event: header_read, op: pg_notify(2.4e8(7) epoch 11710) -4292> 2014-09-18 18:08:31.686407 7f4c9f251700 5 -- op tracker -- , seq: 86027, time: 2014-09-18 18:08:31.686348, event: throttled, op: pg_notify(2.4e8(7) epoch 11710) -4291> 2014-09-18 18:08:31.686414 7f4c9f251700 5 -- op tracker -- , seq: 86027, time: 2014-09-18 18:08:31.686378, event: all_read, op: pg_notify(2.4e8(7) epoch 11710) -4290> 2014-09-18 18:08:31.686420 7f4c9f251700 5 -- op tracker -- , seq: 86027, time: 2014-09-18 18:08:31.686396, event: dispatched, op: pg_notify(2.4e8(7) epoch 11710) -4289> 2014-09-18 18:08:31.686426 7f4c9f251700 5 -- op tracker -- , seq: 86027, time: 2014-09-18 18:08:31.686424, event: waiting_for_osdmap, op: pg_notify(2.4e8(7) epoch 11710) -4288> 2014-09-18 18:08:31.686433 7f4c9f251700 5 -- op tracker -- , seq: 86027, time: 2014-09-18 18:08:31.686432, event: started, op: pg_notify(2.4e8(7) epoch 11710) -4287> 2014-09-18 18:08:31.686450 7f4c9f251700 5 -- op tracker -- , seq: 86027, time: 2014-09-18 18:08:31.686449, event: done, op: pg_notify(2.4e8(7) epoch 11710) -4286> 2014-09-18 18:08:31.686695 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2357 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0xe266480 con 0xdd8c940 -4285> 2014-09-18 18:08:31.686711 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4284> 2014-09-18 18:08:31.686775 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2358 ==== pg_notify(2.457(21) epoch 11710) v5 ==== 1875+0+0 (395544065 0 0) 0xf2b92a0 con 0xdd8c940 -4283> 2014-09-18 18:08:31.686786 7f4c9f251700 5 -- op tracker -- , seq: 86028, time: 2014-09-18 18:08:31.686711, event: header_read, op: pg_notify(2.457(21) epoch 11710) -4282> 2014-09-18 18:08:31.686795 7f4c9f251700 5 -- op tracker -- , seq: 86028, time: 2014-09-18 18:08:31.686712, event: throttled, op: pg_notify(2.457(21) epoch 11710) -4281> 2014-09-18 18:08:31.686802 7f4c9f251700 5 -- op tracker -- , seq: 86028, time: 2014-09-18 18:08:31.686761, event: all_read, op: pg_notify(2.457(21) epoch 11710) -4280> 2014-09-18 18:08:31.686808 7f4c9f251700 5 -- op tracker -- , seq: 86028, time: 2014-09-18 18:08:31.686783, event: dispatched, op: pg_notify(2.457(21) epoch 11710) -4279> 2014-09-18 18:08:31.686814 7f4c9f251700 5 -- op tracker -- , seq: 86028, time: 2014-09-18 18:08:31.686813, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11710) -4278> 2014-09-18 18:08:31.686822 7f4c9f251700 5 -- op tracker -- , seq: 86028, time: 2014-09-18 18:08:31.686821, event: started, op: pg_notify(2.457(21) epoch 11710) -4277> 2014-09-18 18:08:31.686841 7f4c9f251700 5 -- op tracker -- , seq: 86028, time: 2014-09-18 18:08:31.686839, event: done, op: pg_notify(2.457(21) epoch 11710) -4276> 2014-09-18 18:08:31.687097 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3298 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x123321c0 con 0x4900580 -4275> 2014-09-18 18:08:31.687118 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4274> 2014-09-18 18:08:31.687548 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2359 ==== pg_notify(2.2c7(139) epoch 11710) v5 ==== 8651+0+0 (1779841100 0 0) 0xf2b8b20 con 0xdd8c940 -4273> 2014-09-18 18:08:31.687569 7f4c9f251700 5 -- op tracker -- , seq: 86029, time: 2014-09-18 18:08:31.687424, event: header_read, op: pg_notify(2.2c7(139) epoch 11710) -4272> 2014-09-18 18:08:31.687582 7f4c9f251700 5 -- op tracker -- , seq: 86029, time: 2014-09-18 18:08:31.687424, event: throttled, op: pg_notify(2.2c7(139) epoch 11710) -4271> 2014-09-18 18:08:31.687592 7f4c9f251700 5 -- op tracker -- , seq: 86029, time: 2014-09-18 18:08:31.687534, event: all_read, op: pg_notify(2.2c7(139) epoch 11710) -4270> 2014-09-18 18:08:31.687599 7f4c9f251700 5 -- op tracker -- , seq: 86029, time: 2014-09-18 18:08:31.687564, event: dispatched, op: pg_notify(2.2c7(139) epoch 11710) -4269> 2014-09-18 18:08:31.687606 7f4c9f251700 5 -- op tracker -- , seq: 86029, time: 2014-09-18 18:08:31.687604, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11710) -4268> 2014-09-18 18:08:31.687615 7f4c9f251700 5 -- op tracker -- , seq: 86029, time: 2014-09-18 18:08:31.687612, event: started, op: pg_notify(2.2c7(139) epoch 11710) -4267> 2014-09-18 18:08:31.687651 7f4c9f251700 5 -- op tracker -- , seq: 86029, time: 2014-09-18 18:08:31.687649, event: done, op: pg_notify(2.2c7(139) epoch 11710) -4266> 2014-09-18 18:08:31.687679 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3299 ==== pg_notify(2.4f6(184) epoch 11710) v5 ==== 11296+0+0 (145109358 0 0) 0x1247e1c0 con 0x4900580 -4265> 2014-09-18 18:08:31.687711 7f4c9f251700 5 -- op tracker -- , seq: 86030, time: 2014-09-18 18:08:31.687288, event: header_read, op: pg_notify(2.4f6(184) epoch 11710) -4264> 2014-09-18 18:08:31.687726 7f4c9f251700 5 -- op tracker -- , seq: 86030, time: 2014-09-18 18:08:31.687289, event: throttled, op: pg_notify(2.4f6(184) epoch 11710) -4263> 2014-09-18 18:08:31.687733 7f4c9f251700 5 -- op tracker -- , seq: 86030, time: 2014-09-18 18:08:31.687571, event: all_read, op: pg_notify(2.4f6(184) epoch 11710) -4262> 2014-09-18 18:08:31.687738 7f4c9f251700 5 -- op tracker -- , seq: 86030, time: 2014-09-18 18:08:31.687707, event: dispatched, op: pg_notify(2.4f6(184) epoch 11710) -4261> 2014-09-18 18:08:31.687745 7f4c9f251700 5 -- op tracker -- , seq: 86030, time: 2014-09-18 18:08:31.687743, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11710) -4260> 2014-09-18 18:08:31.687752 7f4c9f251700 5 -- op tracker -- , seq: 86030, time: 2014-09-18 18:08:31.687751, event: started, op: pg_notify(2.4f6(184) epoch 11710) -4259> 2014-09-18 18:08:31.687770 7f4c9f251700 5 -- op tracker -- , seq: 86030, time: 2014-09-18 18:08:31.687768, event: done, op: pg_notify(2.4f6(184) epoch 11710) -4258> 2014-09-18 18:08:31.687905 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2013 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x2db47180 con 0xe2d9a20 -4257> 2014-09-18 18:08:31.687918 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4256> 2014-09-18 18:08:31.687945 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2014 ==== pg_notify(2.26d(9) epoch 11710) v5 ==== 1257+0+0 (3029420509 0 0) 0xec6a580 con 0xe2d9a20 -4255> 2014-09-18 18:08:31.687960 7f4c9f251700 5 -- op tracker -- , seq: 86031, time: 2014-09-18 18:08:31.687907, event: header_read, op: pg_notify(2.26d(9) epoch 11710) -4254> 2014-09-18 18:08:31.687972 7f4c9f251700 5 -- op tracker -- , seq: 86031, time: 2014-09-18 18:08:31.687908, event: throttled, op: pg_notify(2.26d(9) epoch 11710) -4253> 2014-09-18 18:08:31.687981 7f4c9f251700 5 -- op tracker -- , seq: 86031, time: 2014-09-18 18:08:31.687935, event: all_read, op: pg_notify(2.26d(9) epoch 11710) -4252> 2014-09-18 18:08:31.687990 7f4c9f251700 5 -- op tracker -- , seq: 86031, time: 2014-09-18 18:08:31.687955, event: dispatched, op: pg_notify(2.26d(9) epoch 11710) -4251> 2014-09-18 18:08:31.687997 7f4c9f251700 5 -- op tracker -- , seq: 86031, time: 2014-09-18 18:08:31.687996, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11710) -4250> 2014-09-18 18:08:31.688005 7f4c9f251700 5 -- op tracker -- , seq: 86031, time: 2014-09-18 18:08:31.688004, event: started, op: pg_notify(2.26d(9) epoch 11710) -4249> 2014-09-18 18:08:31.688031 7f4c9f251700 5 -- op tracker -- , seq: 86031, time: 2014-09-18 18:08:31.688030, event: done, op: pg_notify(2.26d(9) epoch 11710) -4248> 2014-09-18 18:08:31.688333 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2598 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x1440b600 con 0x4900b00 -4247> 2014-09-18 18:08:31.688364 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4246> 2014-09-18 18:08:31.688395 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2599 ==== pg_notify(2.71c(12) epoch 11710) v5 ==== 1424+0+0 (88326887 0 0) 0xf7a74a0 con 0x4900b00 -4245> 2014-09-18 18:08:31.688409 7f4c9f251700 5 -- op tracker -- , seq: 86032, time: 2014-09-18 18:08:31.688345, event: header_read, op: pg_notify(2.71c(12) epoch 11710) -4244> 2014-09-18 18:08:31.688418 7f4c9f251700 5 -- op tracker -- , seq: 86032, time: 2014-09-18 18:08:31.688346, event: throttled, op: pg_notify(2.71c(12) epoch 11710) -4243> 2014-09-18 18:08:31.688424 7f4c9f251700 5 -- op tracker -- , seq: 86032, time: 2014-09-18 18:08:31.688383, event: all_read, op: pg_notify(2.71c(12) epoch 11710) -4242> 2014-09-18 18:08:31.688435 7f4c9f251700 5 -- op tracker -- , seq: 86032, time: 2014-09-18 18:08:31.688406, event: dispatched, op: pg_notify(2.71c(12) epoch 11710) -4241> 2014-09-18 18:08:31.688445 7f4c9f251700 5 -- op tracker -- , seq: 86032, time: 2014-09-18 18:08:31.688444, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11710) -4240> 2014-09-18 18:08:31.688457 7f4c9f251700 5 -- op tracker -- , seq: 86032, time: 2014-09-18 18:08:31.688452, event: started, op: pg_notify(2.71c(12) epoch 11710) -4239> 2014-09-18 18:08:31.688479 7f4c9f251700 5 -- op tracker -- , seq: 86032, time: 2014-09-18 18:08:31.688477, event: done, op: pg_notify(2.71c(12) epoch 11710) -4238> 2014-09-18 18:08:31.689865 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3300 ==== pg_notify(2.17e(17) epoch 11710) v5 ==== 1593+0+0 (1636289277 0 0) 0x1247c960 con 0x4900580 -4237> 2014-09-18 18:08:31.689880 7f4c9f251700 5 -- op tracker -- , seq: 86033, time: 2014-09-18 18:08:31.689821, event: header_read, op: pg_notify(2.17e(17) epoch 11710) -4236> 2014-09-18 18:08:31.689889 7f4c9f251700 5 -- op tracker -- , seq: 86033, time: 2014-09-18 18:08:31.689822, event: throttled, op: pg_notify(2.17e(17) epoch 11710) -4235> 2014-09-18 18:08:31.689895 7f4c9f251700 5 -- op tracker -- , seq: 86033, time: 2014-09-18 18:08:31.689853, event: all_read, op: pg_notify(2.17e(17) epoch 11710) -4234> 2014-09-18 18:08:31.689901 7f4c9f251700 5 -- op tracker -- , seq: 86033, time: 2014-09-18 18:08:31.689876, event: dispatched, op: pg_notify(2.17e(17) epoch 11710) -4233> 2014-09-18 18:08:31.689907 7f4c9f251700 5 -- op tracker -- , seq: 86033, time: 2014-09-18 18:08:31.689906, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11710) -4232> 2014-09-18 18:08:31.689915 7f4c9f251700 5 -- op tracker -- , seq: 86033, time: 2014-09-18 18:08:31.689913, event: started, op: pg_notify(2.17e(17) epoch 11710) -4231> 2014-09-18 18:08:31.689932 7f4c9f251700 5 -- op tracker -- , seq: 86033, time: 2014-09-18 18:08:31.689931, event: done, op: pg_notify(2.17e(17) epoch 11710) -4230> 2014-09-18 18:08:31.690441 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 128 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0xeee8d80 con 0x1395f080 -4229> 2014-09-18 18:08:31.690455 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4228> 2014-09-18 18:08:31.690479 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 129 ==== pg_notify(2.519(4) epoch 11710) v5 ==== 962+0+0 (862339893 0 0) 0x21e112c0 con 0x1395f080 -4227> 2014-09-18 18:08:31.690496 7f4c9f251700 5 -- op tracker -- , seq: 86034, time: 2014-09-18 18:08:31.690445, event: header_read, op: pg_notify(2.519(4) epoch 11710) -4226> 2014-09-18 18:08:31.690506 7f4c9f251700 5 -- op tracker -- , seq: 86034, time: 2014-09-18 18:08:31.690445, event: throttled, op: pg_notify(2.519(4) epoch 11710) -4225> 2014-09-18 18:08:31.690512 7f4c9f251700 5 -- op tracker -- , seq: 86034, time: 2014-09-18 18:08:31.690468, event: all_read, op: pg_notify(2.519(4) epoch 11710) -4224> 2014-09-18 18:08:31.690518 7f4c9f251700 5 -- op tracker -- , seq: 86034, time: 2014-09-18 18:08:31.690492, event: dispatched, op: pg_notify(2.519(4) epoch 11710) -4223> 2014-09-18 18:08:31.690574 7f4c9f251700 5 -- op tracker -- , seq: 86034, time: 2014-09-18 18:08:31.690572, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11710) -4222> 2014-09-18 18:08:31.690583 7f4c9f251700 5 -- op tracker -- , seq: 86034, time: 2014-09-18 18:08:31.690581, event: started, op: pg_notify(2.519(4) epoch 11710) -4221> 2014-09-18 18:08:31.690601 7f4c9f251700 5 -- op tracker -- , seq: 86034, time: 2014-09-18 18:08:31.690600, event: done, op: pg_notify(2.519(4) epoch 11710) -4220> 2014-09-18 18:08:31.690765 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 45 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x14408d80 con 0x1395f340 -4219> 2014-09-18 18:08:31.690778 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4218> 2014-09-18 18:08:31.690831 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 46 ==== pg_notify(2.1d8(9) epoch 11710) v5 ==== 1261+0+0 (3255979292 0 0) 0x19f25de0 con 0x1395f340 -4217> 2014-09-18 18:08:31.690850 7f4c9f251700 5 -- op tracker -- , seq: 86035, time: 2014-09-18 18:08:31.690774, event: header_read, op: pg_notify(2.1d8(9) epoch 11710) -4216> 2014-09-18 18:08:31.690863 7f4c9f251700 5 -- op tracker -- , seq: 86035, time: 2014-09-18 18:08:31.690774, event: throttled, op: pg_notify(2.1d8(9) epoch 11710) -4215> 2014-09-18 18:08:31.690873 7f4c9f251700 5 -- op tracker -- , seq: 86035, time: 2014-09-18 18:08:31.690817, event: all_read, op: pg_notify(2.1d8(9) epoch 11710) -4214> 2014-09-18 18:08:31.690885 7f4c9f251700 5 -- op tracker -- , seq: 86035, time: 2014-09-18 18:08:31.690845, event: dispatched, op: pg_notify(2.1d8(9) epoch 11710) -4213> 2014-09-18 18:08:31.690896 7f4c9f251700 5 -- op tracker -- , seq: 86035, time: 2014-09-18 18:08:31.690893, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11710) -4212> 2014-09-18 18:08:31.690909 7f4c9f251700 5 -- op tracker -- , seq: 86035, time: 2014-09-18 18:08:31.690906, event: started, op: pg_notify(2.1d8(9) epoch 11710) -4211> 2014-09-18 18:08:31.690937 7f4c9f251700 5 -- op tracker -- , seq: 86035, time: 2014-09-18 18:08:31.690934, event: done, op: pg_notify(2.1d8(9) epoch 11710) -4210> 2014-09-18 18:08:31.690990 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 130 ==== pg_notify(2.4e8(37) epoch 11710) v5 ==== 2849+0+0 (1686808677 0 0) 0xf345480 con 0x1395f080 -4209> 2014-09-18 18:08:31.691010 7f4c9f251700 5 -- op tracker -- , seq: 86036, time: 2014-09-18 18:08:31.690906, event: header_read, op: pg_notify(2.4e8(37) epoch 11710) -4208> 2014-09-18 18:08:31.691023 7f4c9f251700 5 -- op tracker -- , seq: 86036, time: 2014-09-18 18:08:31.690906, event: throttled, op: pg_notify(2.4e8(37) epoch 11710) -4207> 2014-09-18 18:08:31.691033 7f4c9f251700 5 -- op tracker -- , seq: 86036, time: 2014-09-18 18:08:31.690976, event: all_read, op: pg_notify(2.4e8(37) epoch 11710) -4206> 2014-09-18 18:08:31.691043 7f4c9f251700 5 -- op tracker -- , seq: 86036, time: 2014-09-18 18:08:31.691005, event: dispatched, op: pg_notify(2.4e8(37) epoch 11710) -4205> 2014-09-18 18:08:31.691054 7f4c9f251700 5 -- op tracker -- , seq: 86036, time: 2014-09-18 18:08:31.691052, event: waiting_for_osdmap, op: pg_notify(2.4e8(37) epoch 11710) -4204> 2014-09-18 18:08:31.691068 7f4c9f251700 5 -- op tracker -- , seq: 86036, time: 2014-09-18 18:08:31.691065, event: started, op: pg_notify(2.4e8(37) epoch 11710) -4203> 2014-09-18 18:08:31.691095 7f4c9f251700 5 -- op tracker -- , seq: 86036, time: 2014-09-18 18:08:31.691092, event: done, op: pg_notify(2.4e8(37) epoch 11710) -4202> 2014-09-18 18:08:31.691410 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3301 ==== pg_notify(2.1bb(9) epoch 11710) v5 ==== 1189+0+0 (2605749068 0 0) 0x1247f660 con 0x4900580 -4201> 2014-09-18 18:08:31.691427 7f4c9f251700 5 -- op tracker -- , seq: 86037, time: 2014-09-18 18:08:31.691359, event: header_read, op: pg_notify(2.1bb(9) epoch 11710) -4200> 2014-09-18 18:08:31.691437 7f4c9f251700 5 -- op tracker -- , seq: 86037, time: 2014-09-18 18:08:31.691360, event: throttled, op: pg_notify(2.1bb(9) epoch 11710) -4199> 2014-09-18 18:08:31.691443 7f4c9f251700 5 -- op tracker -- , seq: 86037, time: 2014-09-18 18:08:31.691398, event: all_read, op: pg_notify(2.1bb(9) epoch 11710) -4198> 2014-09-18 18:08:31.691449 7f4c9f251700 5 -- op tracker -- , seq: 86037, time: 2014-09-18 18:08:31.691423, event: dispatched, op: pg_notify(2.1bb(9) epoch 11710) -4197> 2014-09-18 18:08:31.691456 7f4c9f251700 5 -- op tracker -- , seq: 86037, time: 2014-09-18 18:08:31.691454, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11710) -4196> 2014-09-18 18:08:31.691463 7f4c9f251700 5 -- op tracker -- , seq: 86037, time: 2014-09-18 18:08:31.691462, event: started, op: pg_notify(2.1bb(9) epoch 11710) -4195> 2014-09-18 18:08:31.691481 7f4c9f251700 5 -- op tracker -- , seq: 86037, time: 2014-09-18 18:08:31.691480, event: done, op: pg_notify(2.1bb(9) epoch 11710) -4194> 2014-09-18 18:08:31.699528 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 788 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x2d7fcac0 con 0xe9a27e0 -4193> 2014-09-18 18:08:31.699543 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4192> 2014-09-18 18:08:31.699776 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 789 ==== pg_notify(2.2d8(173) epoch 11710) v5 ==== 10757+0+0 (2589711571 0 0) 0x1421fa40 con 0xe9a27e0 -4191> 2014-09-18 18:08:31.699786 7f4c9f251700 5 -- op tracker -- , seq: 86038, time: 2014-09-18 18:08:31.699627, event: header_read, op: pg_notify(2.2d8(173) epoch 11710) -4190> 2014-09-18 18:08:31.699795 7f4c9f251700 5 -- op tracker -- , seq: 86038, time: 2014-09-18 18:08:31.699627, event: throttled, op: pg_notify(2.2d8(173) epoch 11710) -4189> 2014-09-18 18:08:31.699800 7f4c9f251700 5 -- op tracker -- , seq: 86038, time: 2014-09-18 18:08:31.699765, event: all_read, op: pg_notify(2.2d8(173) epoch 11710) -4188> 2014-09-18 18:08:31.699806 7f4c9f251700 5 -- op tracker -- , seq: 86038, time: 2014-09-18 18:08:31.699784, event: dispatched, op: pg_notify(2.2d8(173) epoch 11710) -4187> 2014-09-18 18:08:31.699813 7f4c9f251700 5 -- op tracker -- , seq: 86038, time: 2014-09-18 18:08:31.699811, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11710) -4186> 2014-09-18 18:08:31.699820 7f4c9f251700 5 -- op tracker -- , seq: 86038, time: 2014-09-18 18:08:31.699819, event: started, op: pg_notify(2.2d8(173) epoch 11710) -4185> 2014-09-18 18:08:31.699836 7f4c9f251700 5 -- op tracker -- , seq: 86038, time: 2014-09-18 18:08:31.699835, event: done, op: pg_notify(2.2d8(173) epoch 11710) -4184> 2014-09-18 18:08:31.734771 7f4caae41700 5 osd.42 11710 tick -4183> 2014-09-18 18:08:31.748205 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.9 10.242.42.166:0/24959 110 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.750402) v2 ==== 47+0+0 (2889785760 0 0) 0xfc58ee0 con 0x10a5c520 -4182> 2014-09-18 18:08:31.748237 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:31.750402) v2 -- ?+0 0xf1e8940 con 0x10a5c520 -4181> 2014-09-18 18:08:31.748265 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.9 10.242.42.166:0/24959 110 ==== osd_ping(ping e11709 stamp 2014-09-18 18:08:31.750402) v2 ==== 47+0+0 (2889785760 0 0) 0xf6de000 con 0x110bb9c0 -4180> 2014-09-18 18:08:31.748286 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:31.750402) v2 -- ?+0 0xf811c00 con 0x110bb9c0 -4179> 2014-09-18 18:08:31.771545 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3208 ==== pg_notify(2.71(19) epoch 11710) v5 ==== 1795+0+0 (808487776 0 0) 0x2b986780 con 0x1395ec60 -4178> 2014-09-18 18:08:31.771564 7f4c9f251700 5 -- op tracker -- , seq: 86039, time: 2014-09-18 18:08:31.771496, event: header_read, op: pg_notify(2.71(19) epoch 11710) -4177> 2014-09-18 18:08:31.771578 7f4c9f251700 5 -- op tracker -- , seq: 86039, time: 2014-09-18 18:08:31.771497, event: throttled, op: pg_notify(2.71(19) epoch 11710) -4176> 2014-09-18 18:08:31.771586 7f4c9f251700 5 -- op tracker -- , seq: 86039, time: 2014-09-18 18:08:31.771531, event: all_read, op: pg_notify(2.71(19) epoch 11710) -4175> 2014-09-18 18:08:31.771594 7f4c9f251700 5 -- op tracker -- , seq: 86039, time: 2014-09-18 18:08:31.771560, event: dispatched, op: pg_notify(2.71(19) epoch 11710) -4174> 2014-09-18 18:08:31.771603 7f4c9f251700 5 -- op tracker -- , seq: 86039, time: 2014-09-18 18:08:31.771602, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11710) -4173> 2014-09-18 18:08:31.771612 7f4c9f251700 5 -- op tracker -- , seq: 86039, time: 2014-09-18 18:08:31.771610, event: started, op: pg_notify(2.71(19) epoch 11710) -4172> 2014-09-18 18:08:31.771630 7f4c9f251700 5 -- op tracker -- , seq: 86039, time: 2014-09-18 18:08:31.771629, event: done, op: pg_notify(2.71(19) epoch 11710) -4171> 2014-09-18 18:08:31.781274 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3209 ==== pg_notify(2.519(4) epoch 11710) v5 ==== 962+0+0 (862339893 0 0) 0x2b987e00 con 0x1395ec60 -4170> 2014-09-18 18:08:31.781304 7f4c9f251700 5 -- op tracker -- , seq: 86040, time: 2014-09-18 18:08:31.781203, event: header_read, op: pg_notify(2.519(4) epoch 11710) -4169> 2014-09-18 18:08:31.781317 7f4c9f251700 5 -- op tracker -- , seq: 86040, time: 2014-09-18 18:08:31.781204, event: throttled, op: pg_notify(2.519(4) epoch 11710) -4168> 2014-09-18 18:08:31.781325 7f4c9f251700 5 -- op tracker -- , seq: 86040, time: 2014-09-18 18:08:31.781232, event: all_read, op: pg_notify(2.519(4) epoch 11710) -4167> 2014-09-18 18:08:31.781333 7f4c9f251700 5 -- op tracker -- , seq: 86040, time: 2014-09-18 18:08:31.781300, event: dispatched, op: pg_notify(2.519(4) epoch 11710) -4166> 2014-09-18 18:08:31.781342 7f4c9f251700 5 -- op tracker -- , seq: 86040, time: 2014-09-18 18:08:31.781340, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11710) -4165> 2014-09-18 18:08:31.781350 7f4c9f251700 5 -- op tracker -- , seq: 86040, time: 2014-09-18 18:08:31.781349, event: started, op: pg_notify(2.519(4) epoch 11710) -4164> 2014-09-18 18:08:31.781383 7f4c9f251700 5 -- op tracker -- , seq: 86040, time: 2014-09-18 18:08:31.781381, event: done, op: pg_notify(2.519(4) epoch 11710) -4163> 2014-09-18 18:08:31.784190 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3210 ==== pg_notify(2.410(12) epoch 11710) v5 ==== 1376+0+0 (1809252380 0 0) 0xf0b9840 con 0x1395ec60 -4162> 2014-09-18 18:08:31.784210 7f4c9f251700 5 -- op tracker -- , seq: 86041, time: 2014-09-18 18:08:31.784088, event: header_read, op: pg_notify(2.410(12) epoch 11710) -4161> 2014-09-18 18:08:31.784227 7f4c9f251700 5 -- op tracker -- , seq: 86041, time: 2014-09-18 18:08:31.784089, event: throttled, op: pg_notify(2.410(12) epoch 11710) -4160> 2014-09-18 18:08:31.784235 7f4c9f251700 5 -- op tracker -- , seq: 86041, time: 2014-09-18 18:08:31.784139, event: all_read, op: pg_notify(2.410(12) epoch 11710) -4159> 2014-09-18 18:08:31.784243 7f4c9f251700 5 -- op tracker -- , seq: 86041, time: 2014-09-18 18:08:31.784206, event: dispatched, op: pg_notify(2.410(12) epoch 11710) -4158> 2014-09-18 18:08:31.784250 7f4c9f251700 5 -- op tracker -- , seq: 86041, time: 2014-09-18 18:08:31.784248, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11710) -4157> 2014-09-18 18:08:31.784259 7f4c9f251700 5 -- op tracker -- , seq: 86041, time: 2014-09-18 18:08:31.784257, event: started, op: pg_notify(2.410(12) epoch 11710) -4156> 2014-09-18 18:08:31.784287 7f4c9f251700 5 -- op tracker -- , seq: 86041, time: 2014-09-18 18:08:31.784285, event: done, op: pg_notify(2.410(12) epoch 11710) -4155> 2014-09-18 18:08:31.787909 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3211 ==== pg_notify(2.186(12) epoch 11710) v5 ==== 1386+0+0 (1709412671 0 0) 0xf243e00 con 0x1395ec60 -4154> 2014-09-18 18:08:31.787929 7f4c9f251700 5 -- op tracker -- , seq: 86042, time: 2014-09-18 18:08:31.787837, event: header_read, op: pg_notify(2.186(12) epoch 11710) -4153> 2014-09-18 18:08:31.787943 7f4c9f251700 5 -- op tracker -- , seq: 86042, time: 2014-09-18 18:08:31.787838, event: throttled, op: pg_notify(2.186(12) epoch 11710) -4152> 2014-09-18 18:08:31.787955 7f4c9f251700 5 -- op tracker -- , seq: 86042, time: 2014-09-18 18:08:31.787873, event: all_read, op: pg_notify(2.186(12) epoch 11710) -4151> 2014-09-18 18:08:31.787966 7f4c9f251700 5 -- op tracker -- , seq: 86042, time: 2014-09-18 18:08:31.787925, event: dispatched, op: pg_notify(2.186(12) epoch 11710) -4150> 2014-09-18 18:08:31.787972 7f4c9f251700 5 -- op tracker -- , seq: 86042, time: 2014-09-18 18:08:31.787971, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11710) -4149> 2014-09-18 18:08:31.787980 7f4c9f251700 5 -- op tracker -- , seq: 86042, time: 2014-09-18 18:08:31.787979, event: started, op: pg_notify(2.186(12) epoch 11710) -4148> 2014-09-18 18:08:31.787999 7f4c9f251700 5 -- op tracker -- , seq: 86042, time: 2014-09-18 18:08:31.787998, event: done, op: pg_notify(2.186(12) epoch 11710) -4147> 2014-09-18 18:08:31.816090 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3212 ==== pg_notify(2.74e(11) epoch 11710) v5 ==== 1303+0+0 (726470239 0 0) 0xf242d20 con 0x1395ec60 -4146> 2014-09-18 18:08:31.816109 7f4c9f251700 5 -- op tracker -- , seq: 86043, time: 2014-09-18 18:08:31.815958, event: header_read, op: pg_notify(2.74e(11) epoch 11710) -4145> 2014-09-18 18:08:31.816120 7f4c9f251700 5 -- op tracker -- , seq: 86043, time: 2014-09-18 18:08:31.815959, event: throttled, op: pg_notify(2.74e(11) epoch 11710) -4144> 2014-09-18 18:08:31.816128 7f4c9f251700 5 -- op tracker -- , seq: 86043, time: 2014-09-18 18:08:31.816024, event: all_read, op: pg_notify(2.74e(11) epoch 11710) -4143> 2014-09-18 18:08:31.816138 7f4c9f251700 5 -- op tracker -- , seq: 86043, time: 2014-09-18 18:08:31.816105, event: dispatched, op: pg_notify(2.74e(11) epoch 11710) -4142> 2014-09-18 18:08:31.816144 7f4c9f251700 5 -- op tracker -- , seq: 86043, time: 2014-09-18 18:08:31.816143, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11710) -4141> 2014-09-18 18:08:31.816154 7f4c9f251700 5 -- op tracker -- , seq: 86043, time: 2014-09-18 18:08:31.816153, event: started, op: pg_notify(2.74e(11) epoch 11710) -4140> 2014-09-18 18:08:31.816177 7f4c9f251700 5 -- op tracker -- , seq: 86043, time: 2014-09-18 18:08:31.816175, event: done, op: pg_notify(2.74e(11) epoch 11710) -4139> 2014-09-18 18:08:31.816998 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3213 ==== pg_notify(2.4b1(13) epoch 11710) v5 ==== 1441+0+0 (2475966872 0 0) 0xf242000 con 0x1395ec60 -4138> 2014-09-18 18:08:31.817017 7f4c9f251700 5 -- op tracker -- , seq: 86044, time: 2014-09-18 18:08:31.816919, event: header_read, op: pg_notify(2.4b1(13) epoch 11710) -4137> 2014-09-18 18:08:31.817027 7f4c9f251700 5 -- op tracker -- , seq: 86044, time: 2014-09-18 18:08:31.816921, event: throttled, op: pg_notify(2.4b1(13) epoch 11710) -4136> 2014-09-18 18:08:31.817033 7f4c9f251700 5 -- op tracker -- , seq: 86044, time: 2014-09-18 18:08:31.816957, event: all_read, op: pg_notify(2.4b1(13) epoch 11710) -4135> 2014-09-18 18:08:31.817039 7f4c9f251700 5 -- op tracker -- , seq: 86044, time: 2014-09-18 18:08:31.817013, event: dispatched, op: pg_notify(2.4b1(13) epoch 11710) -4134> 2014-09-18 18:08:31.817045 7f4c9f251700 5 -- op tracker -- , seq: 86044, time: 2014-09-18 18:08:31.817044, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11710) -4133> 2014-09-18 18:08:31.817053 7f4c9f251700 5 -- op tracker -- , seq: 86044, time: 2014-09-18 18:08:31.817051, event: started, op: pg_notify(2.4b1(13) epoch 11710) -4132> 2014-09-18 18:08:31.817084 7f4c9f251700 5 -- op tracker -- , seq: 86044, time: 2014-09-18 18:08:31.817082, event: done, op: pg_notify(2.4b1(13) epoch 11710) -4131> 2014-09-18 18:08:31.818133 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3214 ==== pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11710) v5 ==== 9048+0+0 (2046275484 0 0) 0xf2443a0 con 0x1395ec60 -4130> 2014-09-18 18:08:31.818152 7f4c9f251700 5 -- op tracker -- , seq: 86045, time: 2014-09-18 18:08:31.818007, event: header_read, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11710) -4129> 2014-09-18 18:08:31.818163 7f4c9f251700 5 -- op tracker -- , seq: 86045, time: 2014-09-18 18:08:31.818008, event: throttled, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11710) -4128> 2014-09-18 18:08:31.818170 7f4c9f251700 5 -- op tracker -- , seq: 86045, time: 2014-09-18 18:08:31.818119, event: all_read, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11710) -4127> 2014-09-18 18:08:31.818177 7f4c9f251700 5 -- op tracker -- , seq: 86045, time: 2014-09-18 18:08:31.818148, event: dispatched, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11710) -4126> 2014-09-18 18:08:31.818184 7f4c9f251700 5 -- op tracker -- , seq: 86045, time: 2014-09-18 18:08:31.818182, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11710) -4125> 2014-09-18 18:08:31.818193 7f4c9f251700 5 -- op tracker -- , seq: 86045, time: 2014-09-18 18:08:31.818191, event: started, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11710) -4124> 2014-09-18 18:08:31.818334 7f4c9f251700 5 -- op tracker -- , seq: 86045, time: 2014-09-18 18:08:31.818332, event: done, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15),2.799(9),2.4d0(10) epoch 11710) -4123> 2014-09-18 18:08:31.886498 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.22 10.242.42.172:0/37668 441 ==== osd_ping(ping e11710 stamp 2014-09-18 23:37:20.990675) v2 ==== 47+0+0 (3011827871 0 0) 0x2b987fe0 con 0x14393340 -4122> 2014-09-18 18:08:31.886563 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11710 stamp 2014-09-18 23:37:20.990675) v2 -- ?+0 0xfc58ee0 con 0x14393340 -4121> 2014-09-18 18:08:31.886991 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.22 10.242.42.172:0/37668 441 ==== osd_ping(ping e11710 stamp 2014-09-18 23:37:20.990675) v2 ==== 47+0+0 (3011827871 0 0) 0xf577c20 con 0x109caf20 -4120> 2014-09-18 18:08:31.887015 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11710 stamp 2014-09-18 23:37:20.990675) v2 -- ?+0 0xf6de000 con 0x109caf20 -4119> 2014-09-18 18:08:31.931210 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1728 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x10d19180 con 0x4900000 -4118> 2014-09-18 18:08:31.931236 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4117> 2014-09-18 18:08:31.931293 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1729 ==== pg_notify(2.e0(79) epoch 11710) v5 ==== 5119+0+0 (1386502659 0 0) 0xf867fe0 con 0x4900000 -4116> 2014-09-18 18:08:31.931306 7f4c9f251700 5 -- op tracker -- , seq: 86046, time: 2014-09-18 18:08:31.931210, event: header_read, op: pg_notify(2.e0(79) epoch 11710) -4115> 2014-09-18 18:08:31.931316 7f4c9f251700 5 -- op tracker -- , seq: 86046, time: 2014-09-18 18:08:31.931210, event: throttled, op: pg_notify(2.e0(79) epoch 11710) -4114> 2014-09-18 18:08:31.931325 7f4c9f251700 5 -- op tracker -- , seq: 86046, time: 2014-09-18 18:08:31.931282, event: all_read, op: pg_notify(2.e0(79) epoch 11710) -4113> 2014-09-18 18:08:31.931333 7f4c9f251700 5 -- op tracker -- , seq: 86046, time: 2014-09-18 18:08:31.931301, event: dispatched, op: pg_notify(2.e0(79) epoch 11710) -4112> 2014-09-18 18:08:31.931342 7f4c9f251700 5 -- op tracker -- , seq: 86046, time: 2014-09-18 18:08:31.931340, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11710) -4111> 2014-09-18 18:08:31.931351 7f4c9f251700 5 -- op tracker -- , seq: 86046, time: 2014-09-18 18:08:31.931349, event: started, op: pg_notify(2.e0(79) epoch 11710) -4110> 2014-09-18 18:08:31.931376 7f4c9f251700 5 -- op tracker -- , seq: 86046, time: 2014-09-18 18:08:31.931374, event: done, op: pg_notify(2.e0(79) epoch 11710) -4109> 2014-09-18 18:08:31.938641 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=82 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).connect error 10.242.42.164:6854/39317, (111) Connection refused -4108> 2014-09-18 18:08:31.938669 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=82 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).fault (111) Connection refused -4107> 2014-09-18 18:08:32.001263 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 968 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x12bf2400 con 0x10a878c0 -4106> 2014-09-18 18:08:32.001300 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4105> 2014-09-18 18:08:32.001383 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 969 ==== pg_notify(2.517(11),2.4c6(15) epoch 11710) v5 ==== 2834+0+0 (122357870 0 0) 0x1213ad00 con 0x10a878c0 -4104> 2014-09-18 18:08:32.001404 7f4c9f251700 5 -- op tracker -- , seq: 86047, time: 2014-09-18 18:08:32.001264, event: header_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11710) -4103> 2014-09-18 18:08:32.001416 7f4c9f251700 5 -- op tracker -- , seq: 86047, time: 2014-09-18 18:08:32.001264, event: throttled, op: pg_notify(2.517(11),2.4c6(15) epoch 11710) -4102> 2014-09-18 18:08:32.001426 7f4c9f251700 5 -- op tracker -- , seq: 86047, time: 2014-09-18 18:08:32.001346, event: all_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11710) -4101> 2014-09-18 18:08:32.001434 7f4c9f251700 5 -- op tracker -- , seq: 86047, time: 2014-09-18 18:08:32.001398, event: dispatched, op: pg_notify(2.517(11),2.4c6(15) epoch 11710) -4100> 2014-09-18 18:08:32.001443 7f4c9f251700 5 -- op tracker -- , seq: 86047, time: 2014-09-18 18:08:32.001441, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.4c6(15) epoch 11710) -4099> 2014-09-18 18:08:32.001452 7f4c9f251700 5 -- op tracker -- , seq: 86047, time: 2014-09-18 18:08:32.001451, event: started, op: pg_notify(2.517(11),2.4c6(15) epoch 11710) -4098> 2014-09-18 18:08:32.001486 7f4c9f251700 5 -- op tracker -- , seq: 86047, time: 2014-09-18 18:08:32.001484, event: done, op: pg_notify(2.517(11),2.4c6(15) epoch 11710) -4097> 2014-09-18 18:08:32.072411 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3645 ==== pg_notify(2.4f6(12) epoch 11709) v5 ==== 1382+0+0 (2056121563 0 0) 0x21cfa5a0 con 0xcbecec0 -4096> 2014-09-18 18:08:32.072436 7f4c9f251700 5 -- op tracker -- , seq: 86048, time: 2014-09-18 18:08:32.072323, event: header_read, op: pg_notify(2.4f6(12) epoch 11709) -4095> 2014-09-18 18:08:32.072446 7f4c9f251700 5 -- op tracker -- , seq: 86048, time: 2014-09-18 18:08:32.072325, event: throttled, op: pg_notify(2.4f6(12) epoch 11709) -4094> 2014-09-18 18:08:32.072451 7f4c9f251700 5 -- op tracker -- , seq: 86048, time: 2014-09-18 18:08:32.072393, event: all_read, op: pg_notify(2.4f6(12) epoch 11709) -4093> 2014-09-18 18:08:32.072455 7f4c9f251700 5 -- op tracker -- , seq: 86048, time: 2014-09-18 18:08:32.072430, event: dispatched, op: pg_notify(2.4f6(12) epoch 11709) -4092> 2014-09-18 18:08:32.072459 7f4c9f251700 5 -- op tracker -- , seq: 86048, time: 2014-09-18 18:08:32.072458, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11709) -4091> 2014-09-18 18:08:32.072466 7f4c9f251700 5 -- op tracker -- , seq: 86048, time: 2014-09-18 18:08:32.072465, event: started, op: pg_notify(2.4f6(12) epoch 11709) -4090> 2014-09-18 18:08:32.072486 7f4c9f251700 5 -- op tracker -- , seq: 86048, time: 2014-09-18 18:08:32.072484, event: done, op: pg_notify(2.4f6(12) epoch 11709) -4089> 2014-09-18 18:08:32.078334 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3646 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x19df3180 con 0xcbecec0 -4088> 2014-09-18 18:08:32.078353 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4087> 2014-09-18 18:08:32.078369 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3647 ==== pg_notify(2.3b6(10) epoch 11710) v5 ==== 1260+0+0 (2236055950 0 0) 0xf810b20 con 0xcbecec0 -4086> 2014-09-18 18:08:32.078380 7f4c9f251700 5 -- op tracker -- , seq: 86049, time: 2014-09-18 18:08:32.078334, event: header_read, op: pg_notify(2.3b6(10) epoch 11710) -4085> 2014-09-18 18:08:32.078390 7f4c9f251700 5 -- op tracker -- , seq: 86049, time: 2014-09-18 18:08:32.078335, event: throttled, op: pg_notify(2.3b6(10) epoch 11710) -4084> 2014-09-18 18:08:32.078397 7f4c9f251700 5 -- op tracker -- , seq: 86049, time: 2014-09-18 18:08:32.078359, event: all_read, op: pg_notify(2.3b6(10) epoch 11710) -4083> 2014-09-18 18:08:32.078403 7f4c9f251700 5 -- op tracker -- , seq: 86049, time: 2014-09-18 18:08:32.078377, event: dispatched, op: pg_notify(2.3b6(10) epoch 11710) -4082> 2014-09-18 18:08:32.078410 7f4c9f251700 5 -- op tracker -- , seq: 86049, time: 2014-09-18 18:08:32.078408, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11710) -4081> 2014-09-18 18:08:32.078417 7f4c9f251700 5 -- op tracker -- , seq: 86049, time: 2014-09-18 18:08:32.078416, event: started, op: pg_notify(2.3b6(10) epoch 11710) -4080> 2014-09-18 18:08:32.078436 7f4c9f251700 5 -- op tracker -- , seq: 86049, time: 2014-09-18 18:08:32.078434, event: done, op: pg_notify(2.3b6(10) epoch 11710) -4079> 2014-09-18 18:08:32.081241 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3648 ==== pg_notify(2.517(11) epoch 11710) v5 ==== 1343+0+0 (1985774812 0 0) 0xf93d660 con 0xcbecec0 -4078> 2014-09-18 18:08:32.081260 7f4c9f251700 5 -- op tracker -- , seq: 86050, time: 2014-09-18 18:08:32.081163, event: header_read, op: pg_notify(2.517(11) epoch 11710) -4077> 2014-09-18 18:08:32.081271 7f4c9f251700 5 -- op tracker -- , seq: 86050, time: 2014-09-18 18:08:32.081164, event: throttled, op: pg_notify(2.517(11) epoch 11710) -4076> 2014-09-18 18:08:32.081277 7f4c9f251700 5 -- op tracker -- , seq: 86050, time: 2014-09-18 18:08:32.081207, event: all_read, op: pg_notify(2.517(11) epoch 11710) -4075> 2014-09-18 18:08:32.081283 7f4c9f251700 5 -- op tracker -- , seq: 86050, time: 2014-09-18 18:08:32.081256, event: dispatched, op: pg_notify(2.517(11) epoch 11710) -4074> 2014-09-18 18:08:32.081290 7f4c9f251700 5 -- op tracker -- , seq: 86050, time: 2014-09-18 18:08:32.081288, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11710) -4073> 2014-09-18 18:08:32.081298 7f4c9f251700 5 -- op tracker -- , seq: 86050, time: 2014-09-18 18:08:32.081296, event: started, op: pg_notify(2.517(11) epoch 11710) -4072> 2014-09-18 18:08:32.081416 7f4c9f251700 5 -- op tracker -- , seq: 86050, time: 2014-09-18 18:08:32.081414, event: done, op: pg_notify(2.517(11) epoch 11710) -4071> 2014-09-18 18:08:32.082547 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3649 ==== pg_notify(2.2c7(139) epoch 11710) v5 ==== 8651+0+0 (2879506343 0 0) 0xf93a5a0 con 0xcbecec0 -4070> 2014-09-18 18:08:32.082566 7f4c9f251700 5 -- op tracker -- , seq: 86051, time: 2014-09-18 18:08:32.082377, event: header_read, op: pg_notify(2.2c7(139) epoch 11710) -4069> 2014-09-18 18:08:32.082577 7f4c9f251700 5 -- op tracker -- , seq: 86051, time: 2014-09-18 18:08:32.082378, event: throttled, op: pg_notify(2.2c7(139) epoch 11710) -4068> 2014-09-18 18:08:32.082583 7f4c9f251700 5 -- op tracker -- , seq: 86051, time: 2014-09-18 18:08:32.082526, event: all_read, op: pg_notify(2.2c7(139) epoch 11710) -4067> 2014-09-18 18:08:32.082589 7f4c9f251700 5 -- op tracker -- , seq: 86051, time: 2014-09-18 18:08:32.082562, event: dispatched, op: pg_notify(2.2c7(139) epoch 11710) -4066> 2014-09-18 18:08:32.082596 7f4c9f251700 5 -- op tracker -- , seq: 86051, time: 2014-09-18 18:08:32.082594, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11710) -4065> 2014-09-18 18:08:32.082604 7f4c9f251700 5 -- op tracker -- , seq: 86051, time: 2014-09-18 18:08:32.082602, event: started, op: pg_notify(2.2c7(139) epoch 11710) -4064> 2014-09-18 18:08:32.082623 7f4c9f251700 5 -- op tracker -- , seq: 86051, time: 2014-09-18 18:08:32.082621, event: done, op: pg_notify(2.2c7(139) epoch 11710) -4063> 2014-09-18 18:08:32.181288 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1853 ==== osd_map(11710..11710 src has 1..11710) v3 ==== 389+0+0 (2044594543 0 0) 0x1182bd40 con 0x10f60520 -4062> 2014-09-18 18:08:32.181321 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11710,11710], i have 11710, src has [1,11710] -4061> 2014-09-18 18:08:32.181341 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1854 ==== pg_notify(2.4f6(15) epoch 11710) v5 ==== 1581+0+0 (2226304985 0 0) 0xf1e7e00 con 0x10f60520 -4060> 2014-09-18 18:08:32.181353 7f4c9f251700 5 -- op tracker -- , seq: 86052, time: 2014-09-18 18:08:32.181273, event: header_read, op: pg_notify(2.4f6(15) epoch 11710) -4059> 2014-09-18 18:08:32.181365 7f4c9f251700 5 -- op tracker -- , seq: 86052, time: 2014-09-18 18:08:32.181273, event: throttled, op: pg_notify(2.4f6(15) epoch 11710) -4058> 2014-09-18 18:08:32.181374 7f4c9f251700 5 -- op tracker -- , seq: 86052, time: 2014-09-18 18:08:32.181310, event: all_read, op: pg_notify(2.4f6(15) epoch 11710) -4057> 2014-09-18 18:08:32.181382 7f4c9f251700 5 -- op tracker -- , seq: 86052, time: 2014-09-18 18:08:32.181349, event: dispatched, op: pg_notify(2.4f6(15) epoch 11710) -4056> 2014-09-18 18:08:32.181391 7f4c9f251700 5 -- op tracker -- , seq: 86052, time: 2014-09-18 18:08:32.181390, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11710) -4055> 2014-09-18 18:08:32.181401 7f4c9f251700 5 -- op tracker -- , seq: 86052, time: 2014-09-18 18:08:32.181399, event: started, op: pg_notify(2.4f6(15) epoch 11710) -4054> 2014-09-18 18:08:32.181425 7f4c9f251700 5 -- op tracker -- , seq: 86052, time: 2014-09-18 18:08:32.181423, event: done, op: pg_notify(2.4f6(15) epoch 11710) -4053> 2014-09-18 18:08:32.184143 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1855 ==== pg_notify(2.517(11) epoch 11710) v5 ==== 1335+0+0 (920037143 0 0) 0xf12dc00 con 0x10f60520 -4052> 2014-09-18 18:08:32.184341 7f4c9f251700 5 -- op tracker -- , seq: 86053, time: 2014-09-18 18:08:32.184074, event: header_read, op: pg_notify(2.517(11) epoch 11710) -4051> 2014-09-18 18:08:32.184352 7f4c9f251700 5 -- op tracker -- , seq: 86053, time: 2014-09-18 18:08:32.184074, event: throttled, op: pg_notify(2.517(11) epoch 11710) -4050> 2014-09-18 18:08:32.184359 7f4c9f251700 5 -- op tracker -- , seq: 86053, time: 2014-09-18 18:08:32.184107, event: all_read, op: pg_notify(2.517(11) epoch 11710) -4049> 2014-09-18 18:08:32.184365 7f4c9f251700 5 -- op tracker -- , seq: 86053, time: 2014-09-18 18:08:32.184336, event: dispatched, op: pg_notify(2.517(11) epoch 11710) -4048> 2014-09-18 18:08:32.184371 7f4c9f251700 5 -- op tracker -- , seq: 86053, time: 2014-09-18 18:08:32.184370, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11710) -4047> 2014-09-18 18:08:32.184379 7f4c9f251700 5 -- op tracker -- , seq: 86053, time: 2014-09-18 18:08:32.184378, event: started, op: pg_notify(2.517(11) epoch 11710) -4046> 2014-09-18 18:08:32.184399 7f4c9f251700 5 -- op tracker -- , seq: 86053, time: 2014-09-18 18:08:32.184397, event: done, op: pg_notify(2.517(11) epoch 11710) -4045> 2014-09-18 18:08:32.184459 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1856 ==== pg_notify(2.1d8(9) epoch 11710) v5 ==== 1261+0+0 (3255979292 0 0) 0xf12da20 con 0x10f60520 -4044> 2014-09-18 18:08:32.184472 7f4c9f251700 5 -- op tracker -- , seq: 86054, time: 2014-09-18 18:08:32.184410, event: header_read, op: pg_notify(2.1d8(9) epoch 11710) -4043> 2014-09-18 18:08:32.184479 7f4c9f251700 5 -- op tracker -- , seq: 86054, time: 2014-09-18 18:08:32.184411, event: throttled, op: pg_notify(2.1d8(9) epoch 11710) -4042> 2014-09-18 18:08:32.184485 7f4c9f251700 5 -- op tracker -- , seq: 86054, time: 2014-09-18 18:08:32.184446, event: all_read, op: pg_notify(2.1d8(9) epoch 11710) -4041> 2014-09-18 18:08:32.184491 7f4c9f251700 5 -- op tracker -- , seq: 86054, time: 2014-09-18 18:08:32.184469, event: dispatched, op: pg_notify(2.1d8(9) epoch 11710) -4040> 2014-09-18 18:08:32.184497 7f4c9f251700 5 -- op tracker -- , seq: 86054, time: 2014-09-18 18:08:32.184496, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11710) -4039> 2014-09-18 18:08:32.184505 7f4c9f251700 5 -- op tracker -- , seq: 86054, time: 2014-09-18 18:08:32.184503, event: started, op: pg_notify(2.1d8(9) epoch 11710) -4038> 2014-09-18 18:08:32.184521 7f4c9f251700 5 -- op tracker -- , seq: 86054, time: 2014-09-18 18:08:32.184520, event: done, op: pg_notify(2.1d8(9) epoch 11710) -4037> 2014-09-18 18:08:32.204455 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3650 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11710) v5 ==== 6174+0+0 (1656334779 0 0) 0xf973fe0 con 0xcbecec0 -4036> 2014-09-18 18:08:32.204482 7f4c9f251700 5 -- op tracker -- , seq: 86055, time: 2014-09-18 18:08:32.204362, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11710) -4035> 2014-09-18 18:08:32.204498 7f4c9f251700 5 -- op tracker -- , seq: 86055, time: 2014-09-18 18:08:32.204364, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11710) -4034> 2014-09-18 18:08:32.204509 7f4c9f251700 5 -- op tracker -- , seq: 86055, time: 2014-09-18 18:08:32.204437, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11710) -4033> 2014-09-18 18:08:32.204520 7f4c9f251700 5 -- op tracker -- , seq: 86055, time: 2014-09-18 18:08:32.204475, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11710) -4032> 2014-09-18 18:08:32.204531 7f4c9f251700 5 -- op tracker -- , seq: 86055, time: 2014-09-18 18:08:32.204528, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11710) -4031> 2014-09-18 18:08:32.204544 7f4c9f251700 5 -- op tracker -- , seq: 86055, time: 2014-09-18 18:08:32.204540, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11710) -4030> 2014-09-18 18:08:32.204630 7f4c9f251700 5 -- op tracker -- , seq: 86055, time: 2014-09-18 18:08:32.204628, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11710) -4029> 2014-09-18 18:08:32.206669 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3651 ==== pg_notify(2.1bb(9) epoch 11710) v5 ==== 1189+0+0 (936976735 0 0) 0xf972d20 con 0xcbecec0 -4028> 2014-09-18 18:08:32.206722 7f4c9f251700 5 -- op tracker -- , seq: 86056, time: 2014-09-18 18:08:32.206618, event: header_read, op: pg_notify(2.1bb(9) epoch 11710) -4027> 2014-09-18 18:08:32.206733 7f4c9f251700 5 -- op tracker -- , seq: 86056, time: 2014-09-18 18:08:32.206619, event: throttled, op: pg_notify(2.1bb(9) epoch 11710) -4026> 2014-09-18 18:08:32.206739 7f4c9f251700 5 -- op tracker -- , seq: 86056, time: 2014-09-18 18:08:32.206653, event: all_read, op: pg_notify(2.1bb(9) epoch 11710) -4025> 2014-09-18 18:08:32.206745 7f4c9f251700 5 -- op tracker -- , seq: 86056, time: 2014-09-18 18:08:32.206717, event: dispatched, op: pg_notify(2.1bb(9) epoch 11710) -4024> 2014-09-18 18:08:32.206752 7f4c9f251700 5 -- op tracker -- , seq: 86056, time: 2014-09-18 18:08:32.206750, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11710) -4023> 2014-09-18 18:08:32.206760 7f4c9f251700 5 -- op tracker -- , seq: 86056, time: 2014-09-18 18:08:32.206758, event: started, op: pg_notify(2.1bb(9) epoch 11710) -4022> 2014-09-18 18:08:32.206778 7f4c9f251700 5 -- op tracker -- , seq: 86056, time: 2014-09-18 18:08:32.206777, event: done, op: pg_notify(2.1bb(9) epoch 11710) -4021> 2014-09-18 18:08:32.217097 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).connect error 10.242.42.164:6856/39317, (111) Connection refused -4020> 2014-09-18 18:08:32.217097 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).connect error 10.242.42.164:6855/39317, (111) Connection refused -4019> 2014-09-18 18:08:32.217131 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).fault (111) Connection refused -4018> 2014-09-18 18:08:32.217142 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).fault (111) Connection refused -4017> 2014-09-18 18:08:32.240337 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.14 10.242.42.168:0/16237 142 ==== osd_ping(ping e11710 stamp 2014-09-18 18:08:32.106950) v2 ==== 47+0+0 (3609621221 0 0) 0xe0bc760 con 0x101cc3c0 -4016> 2014-09-18 18:08:32.240363 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.106950) v2 -- ?+0 0x2b987fe0 con 0x101cc3c0 -4015> 2014-09-18 18:08:32.240403 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.14 10.242.42.168:0/16237 142 ==== osd_ping(ping e11710 stamp 2014-09-18 18:08:32.106950) v2 ==== 47+0+0 (3609621221 0 0) 0xf7a7fe0 con 0xcc49a20 -4014> 2014-09-18 18:08:32.240421 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.106950) v2 -- ?+0 0xf577c20 con 0xcc49a20 -4013> 2014-09-18 18:08:32.320540 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.32 10.242.42.176:0/1099 113 ==== osd_ping(ping e11710 stamp 2014-09-18 23:37:37.068416) v2 ==== 47+0+0 (342611302 0 0) 0x2b930760 con 0x12abd080 -4012> 2014-09-18 18:08:32.320564 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/1099 -- osd_ping(ping_reply e11710 stamp 2014-09-18 23:37:37.068416) v2 -- ?+0 0xe0bc760 con 0x12abd080 -4011> 2014-09-18 18:08:32.320972 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.32 10.242.42.176:0/1099 113 ==== osd_ping(ping e11710 stamp 2014-09-18 23:37:37.068416) v2 ==== 47+0+0 (342611302 0 0) 0xe1a5660 con 0x110bb2e0 -4010> 2014-09-18 18:08:32.320995 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/1099 -- osd_ping(ping_reply e11710 stamp 2014-09-18 23:37:37.068416) v2 -- ?+0 0xf7a7fe0 con 0x110bb2e0 -4009> 2014-09-18 18:08:32.351355 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3215 ==== pg_log(0.12 epoch 11710 log log((0'0,0'0], crt=0'0) query_epoch 11710) v4 ==== 802+0+0 (4110525626 0 0) 0x21fad800 con 0x1395ec60 -4008> 2014-09-18 18:08:32.351375 7f4c9f251700 5 -- op tracker -- , seq: 86057, time: 2014-09-18 18:08:32.351299, event: header_read, op: pg_log(0.12 epoch 11710 log log((0'0,0'0], crt=0'0) query_epoch 11710) -4007> 2014-09-18 18:08:32.351384 7f4c9f251700 5 -- op tracker -- , seq: 86057, time: 2014-09-18 18:08:32.351300, event: throttled, op: pg_log(0.12 epoch 11710 log log((0'0,0'0], crt=0'0) query_epoch 11710) -4006> 2014-09-18 18:08:32.351388 7f4c9f251700 5 -- op tracker -- , seq: 86057, time: 2014-09-18 18:08:32.351331, event: all_read, op: pg_log(0.12 epoch 11710 log log((0'0,0'0], crt=0'0) query_epoch 11710) -4005> 2014-09-18 18:08:32.351402 7f4c9f251700 5 -- op tracker -- , seq: 86057, time: 2014-09-18 18:08:32.351370, event: dispatched, op: pg_log(0.12 epoch 11710 log log((0'0,0'0], crt=0'0) query_epoch 11710) -4004> 2014-09-18 18:08:32.351407 7f4c9f251700 5 -- op tracker -- , seq: 86057, time: 2014-09-18 18:08:32.351405, event: waiting_for_osdmap, op: pg_log(0.12 epoch 11710 log log((0'0,0'0], crt=0'0) query_epoch 11710) -4003> 2014-09-18 18:08:32.351420 7f4c9f251700 5 -- op tracker -- , seq: 86057, time: 2014-09-18 18:08:32.351418, event: started, op: pg_log(0.12 epoch 11710 log log((0'0,0'0], crt=0'0) query_epoch 11710) -4002> 2014-09-18 18:08:32.351436 7f4c9f251700 5 -- op tracker -- , seq: 86057, time: 2014-09-18 18:08:32.351434, event: done, op: pg_log(0.12 epoch 11710 log log((0'0,0'0], crt=0'0) query_epoch 11710) -4001> 2014-09-18 18:08:32.351570 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5084 ==== pg_notify(2.4e8(1) epoch 11708) v5 ==== 805+0+0 (1386005939 0 0) 0xf9730e0 con 0x1088aec0 -4000> 2014-09-18 18:08:32.351589 7f4c9f251700 5 -- op tracker -- , seq: 86058, time: 2014-09-18 18:08:32.351499, event: header_read, op: pg_notify(2.4e8(1) epoch 11708) -3999> 2014-09-18 18:08:32.351449 7f4c96a40700 5 osd.42 pg_epoch: 11710 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] exit Started/Stray 2.633851 10 0.000745 -3998> 2014-09-18 18:08:32.351606 7f4c9f251700 5 -- op tracker -- , seq: 86058, time: 2014-09-18 18:08:32.351500, event: throttled, op: pg_notify(2.4e8(1) epoch 11708) -3997> 2014-09-18 18:08:32.351625 7f4c9f251700 5 -- op tracker -- , seq: 86058, time: 2014-09-18 18:08:32.351556, event: all_read, op: pg_notify(2.4e8(1) epoch 11708) -3996> 2014-09-18 18:08:32.351632 7f4c9f251700 5 -- op tracker -- , seq: 86058, time: 2014-09-18 18:08:32.351585, event: dispatched, op: pg_notify(2.4e8(1) epoch 11708) -3995> 2014-09-18 18:08:32.351640 7f4c9f251700 5 -- op tracker -- , seq: 86058, time: 2014-09-18 18:08:32.351639, event: waiting_for_osdmap, op: pg_notify(2.4e8(1) epoch 11708) -3994> 2014-09-18 18:08:32.351650 7f4c9f251700 5 -- op tracker -- , seq: 86058, time: 2014-09-18 18:08:32.351648, event: started, op: pg_notify(2.4e8(1) epoch 11708) -3993> 2014-09-18 18:08:32.351621 7f4c96a40700 5 osd.42 pg_epoch: 11710 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive -3992> 2014-09-18 18:08:32.351668 7f4c9f251700 5 -- op tracker -- , seq: 86058, time: 2014-09-18 18:08:32.351666, event: done, op: pg_notify(2.4e8(1) epoch 11708) -3991> 2014-09-18 18:08:32.351687 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 2.624475 16 0.000134 -3990> 2014-09-18 18:08:32.351706 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -3989> 2014-09-18 18:08:32.351749 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000043 0 0.000000 -3988> 2014-09-18 18:08:32.351760 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 peering] exit Started/Primary/Peering 2.624556 0 0.000000 -3987> 2014-09-18 18:08:32.351771 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] exit Started/Primary 2.624574 0 0.000000 -3986> 2014-09-18 18:08:32.351780 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] enter Started/Primary -3985> 2014-09-18 18:08:32.351789 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] enter Started/Primary/Peering/WaitActingChange -3984> 2014-09-18 18:08:32.351695 7f4c96a40700 5 osd.42 pg_epoch: 11710 pg[0.12( empty local-les=11409 n=0 ec=1 les/c 11409/11411 11708/11708/11708) [9,42,27] r=1 lpr=11708 pi=3487-11707/27 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -3983> 2014-09-18 18:08:32.351825 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3982> 2014-09-18 18:08:32.351832 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x1236fe00 con 0x111771e0 -3981> 2014-09-18 18:08:32.351926 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3980> 2014-09-18 18:08:32.351933 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x14f14600 con 0x111771e0 -3979> 2014-09-18 18:08:32.352242 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11710:0.12) v4 -- ?+0 0xf097c20 con 0x1395ec60 -3978> 2014-09-18 18:08:32.352288 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3977> 2014-09-18 18:08:32.352296 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x14f12000 con 0x111771e0 -3976> 2014-09-18 18:08:32.352782 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3216 ==== pg_info(1 pgs e11710:0.12) v4 ==== 742+0+0 (3364766492 0 0) 0xf097c20 con 0x1395ec60 -3975> 2014-09-18 18:08:32.352796 7f4c9f251700 5 -- op tracker -- , seq: 86059, time: 2014-09-18 18:08:32.352744, event: header_read, op: pg_info(1 pgs e11710:0.12) -3974> 2014-09-18 18:08:32.352804 7f4c9f251700 5 -- op tracker -- , seq: 86059, time: 2014-09-18 18:08:32.352745, event: throttled, op: pg_info(1 pgs e11710:0.12) -3973> 2014-09-18 18:08:32.352810 7f4c9f251700 5 -- op tracker -- , seq: 86059, time: 2014-09-18 18:08:32.352770, event: all_read, op: pg_info(1 pgs e11710:0.12) -3972> 2014-09-18 18:08:32.352816 7f4c9f251700 5 -- op tracker -- , seq: 86059, time: 2014-09-18 18:08:32.352793, event: dispatched, op: pg_info(1 pgs e11710:0.12) -3971> 2014-09-18 18:08:32.352822 7f4c9f251700 5 -- op tracker -- , seq: 86059, time: 2014-09-18 18:08:32.352821, event: waiting_for_osdmap, op: pg_info(1 pgs e11710:0.12) -3970> 2014-09-18 18:08:32.352830 7f4c9f251700 5 -- op tracker -- , seq: 86059, time: 2014-09-18 18:08:32.352829, event: started, op: pg_info(1 pgs e11710:0.12) -3969> 2014-09-18 18:08:32.352845 7f4c9f251700 5 -- op tracker -- , seq: 86059, time: 2014-09-18 18:08:32.352844, event: done, op: pg_info(1 pgs e11710:0.12) -3968> 2014-09-18 18:08:32.352901 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3967> 2014-09-18 18:08:32.352908 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x1236f000 con 0x111771e0 -3966> 2014-09-18 18:08:32.352935 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3217 ==== pg_info(1 pgs e11710:0.12) v4 ==== 742+0+0 (1847026886 0 0) 0xf245a20 con 0x1395ec60 -3965> 2014-09-18 18:08:32.352948 7f4c9f251700 5 -- op tracker -- , seq: 86060, time: 2014-09-18 18:08:32.352892, event: header_read, op: pg_info(1 pgs e11710:0.12) -3964> 2014-09-18 18:08:32.352957 7f4c9f251700 5 -- op tracker -- , seq: 86060, time: 2014-09-18 18:08:32.352894, event: throttled, op: pg_info(1 pgs e11710:0.12) -3963> 2014-09-18 18:08:32.352961 7f4c9f251700 5 -- op tracker -- , seq: 86060, time: 2014-09-18 18:08:32.352921, event: all_read, op: pg_info(1 pgs e11710:0.12) -3962> 2014-09-18 18:08:32.352973 7f4c9f251700 5 -- op tracker -- , seq: 86060, time: 2014-09-18 18:08:32.352945, event: dispatched, op: pg_info(1 pgs e11710:0.12) -3961> 2014-09-18 18:08:32.352979 7f4c9f251700 5 -- op tracker -- , seq: 86060, time: 2014-09-18 18:08:32.352977, event: waiting_for_osdmap, op: pg_info(1 pgs e11710:0.12) -3960> 2014-09-18 18:08:32.352990 7f4c9f251700 5 -- op tracker -- , seq: 86060, time: 2014-09-18 18:08:32.352988, event: started, op: pg_info(1 pgs e11710:0.12) -3959> 2014-09-18 18:08:32.353014 7f4c9f251700 5 -- op tracker -- , seq: 86060, time: 2014-09-18 18:08:32.353012, event: done, op: pg_info(1 pgs e11710:0.12) -3958> 2014-09-18 18:08:32.353067 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3957> 2014-09-18 18:08:32.353100 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x14f12e00 con 0x111771e0 -3956> 2014-09-18 18:08:32.354812 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5085 ==== pg_notify(2.62(44) epoch 11708) v5 ==== 3220+0+0 (1958159800 0 0) 0xf486f00 con 0x1088aec0 -3955> 2014-09-18 18:08:32.354826 7f4c9f251700 5 -- op tracker -- , seq: 86061, time: 2014-09-18 18:08:32.354747, event: header_read, op: pg_notify(2.62(44) epoch 11708) -3954> 2014-09-18 18:08:32.354838 7f4c9f251700 5 -- op tracker -- , seq: 86061, time: 2014-09-18 18:08:32.354748, event: throttled, op: pg_notify(2.62(44) epoch 11708) -3953> 2014-09-18 18:08:32.354845 7f4c9f251700 5 -- op tracker -- , seq: 86061, time: 2014-09-18 18:08:32.354800, event: all_read, op: pg_notify(2.62(44) epoch 11708) -3952> 2014-09-18 18:08:32.354851 7f4c9f251700 5 -- op tracker -- , seq: 86061, time: 2014-09-18 18:08:32.354823, event: dispatched, op: pg_notify(2.62(44) epoch 11708) -3951> 2014-09-18 18:08:32.354857 7f4c9f251700 5 -- op tracker -- , seq: 86061, time: 2014-09-18 18:08:32.354856, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11708) -3950> 2014-09-18 18:08:32.354865 7f4c9f251700 5 -- op tracker -- , seq: 86061, time: 2014-09-18 18:08:32.354863, event: started, op: pg_notify(2.62(44) epoch 11708) -3949> 2014-09-18 18:08:32.354988 7f4c9f251700 5 -- op tracker -- , seq: 86061, time: 2014-09-18 18:08:32.354986, event: done, op: pg_notify(2.62(44) epoch 11708) -3948> 2014-09-18 18:08:32.355001 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3947> 2014-09-18 18:08:32.355011 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x1236f800 con 0x111771e0 -3946> 2014-09-18 18:08:32.360097 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5086 ==== pg_notify(2.3b6(10) epoch 11708) v5 ==== 1260+0+0 (3653555031 0 0) 0xf4870e0 con 0x1088aec0 -3945> 2014-09-18 18:08:32.360111 7f4c9f251700 5 -- op tracker -- , seq: 86062, time: 2014-09-18 18:08:32.360038, event: header_read, op: pg_notify(2.3b6(10) epoch 11708) -3944> 2014-09-18 18:08:32.360118 7f4c9f251700 5 -- op tracker -- , seq: 86062, time: 2014-09-18 18:08:32.360039, event: throttled, op: pg_notify(2.3b6(10) epoch 11708) -3943> 2014-09-18 18:08:32.360122 7f4c9f251700 5 -- op tracker -- , seq: 86062, time: 2014-09-18 18:08:32.360072, event: all_read, op: pg_notify(2.3b6(10) epoch 11708) -3942> 2014-09-18 18:08:32.360126 7f4c9f251700 5 -- op tracker -- , seq: 86062, time: 2014-09-18 18:08:32.360108, event: dispatched, op: pg_notify(2.3b6(10) epoch 11708) -3941> 2014-09-18 18:08:32.360130 7f4c9f251700 5 -- op tracker -- , seq: 86062, time: 2014-09-18 18:08:32.360129, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11708) -3940> 2014-09-18 18:08:32.360135 7f4c9f251700 5 -- op tracker -- , seq: 86062, time: 2014-09-18 18:08:32.360134, event: started, op: pg_notify(2.3b6(10) epoch 11708) -3939> 2014-09-18 18:08:32.360150 7f4c9f251700 5 -- op tracker -- , seq: 86062, time: 2014-09-18 18:08:32.360149, event: done, op: pg_notify(2.3b6(10) epoch 11708) -3938> 2014-09-18 18:08:32.360165 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3937> 2014-09-18 18:08:32.360174 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x14f14800 con 0x111771e0 -3936> 2014-09-18 18:08:32.389606 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5087 ==== pg_notify(2.6f8(10) epoch 11708) v5 ==== 1276+0+0 (1983548898 0 0) 0xf4883a0 con 0x1088aec0 -3935> 2014-09-18 18:08:32.389622 7f4c9f251700 5 -- op tracker -- , seq: 86063, time: 2014-09-18 18:08:32.389532, event: header_read, op: pg_notify(2.6f8(10) epoch 11708) -3934> 2014-09-18 18:08:32.389629 7f4c9f251700 5 -- op tracker -- , seq: 86063, time: 2014-09-18 18:08:32.389537, event: throttled, op: pg_notify(2.6f8(10) epoch 11708) -3933> 2014-09-18 18:08:32.389633 7f4c9f251700 5 -- op tracker -- , seq: 86063, time: 2014-09-18 18:08:32.389591, event: all_read, op: pg_notify(2.6f8(10) epoch 11708) -3932> 2014-09-18 18:08:32.389637 7f4c9f251700 5 -- op tracker -- , seq: 86063, time: 2014-09-18 18:08:32.389618, event: dispatched, op: pg_notify(2.6f8(10) epoch 11708) -3931> 2014-09-18 18:08:32.389641 7f4c9f251700 5 -- op tracker -- , seq: 86063, time: 2014-09-18 18:08:32.389640, event: waiting_for_osdmap, op: pg_notify(2.6f8(10) epoch 11708) -3930> 2014-09-18 18:08:32.389647 7f4c9f251700 5 -- op tracker -- , seq: 86063, time: 2014-09-18 18:08:32.389645, event: started, op: pg_notify(2.6f8(10) epoch 11708) -3929> 2014-09-18 18:08:32.389670 7f4c9f251700 5 -- op tracker -- , seq: 86063, time: 2014-09-18 18:08:32.389668, event: done, op: pg_notify(2.6f8(10) epoch 11708) -3928> 2014-09-18 18:08:32.389698 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3927> 2014-09-18 18:08:32.389708 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x1236da00 con 0x111771e0 -3926> 2014-09-18 18:08:32.415332 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.0 10.242.42.164:0/15878 160 ==== osd_ping(ping e11710 stamp 2014-09-18 18:08:32.414737) v2 ==== 47+0+0 (2996101923 0 0) 0xf576780 con 0x1d5bc7e0 -3925> 2014-09-18 18:08:32.415360 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.414737) v2 -- ?+0 0x2b930760 con 0x1d5bc7e0 -3924> 2014-09-18 18:08:32.415377 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.0 10.242.42.164:0/15878 160 ==== osd_ping(ping e11710 stamp 2014-09-18 18:08:32.414737) v2 ==== 47+0+0 (2996101923 0 0) 0xde5f860 con 0x142e8260 -3923> 2014-09-18 18:08:32.415399 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.414737) v2 -- ?+0 0xe1a5660 con 0x142e8260 -3922> 2014-09-18 18:08:32.434684 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5088 ==== pg_notify(2.517(11),2.519(23) epoch 11708) v5 ==== 3274+0+0 (2521192838 0 0) 0xf4890c0 con 0x1088aec0 -3921> 2014-09-18 18:08:32.434701 7f4c9f251700 5 -- op tracker -- , seq: 86064, time: 2014-09-18 18:08:32.434610, event: header_read, op: pg_notify(2.517(11),2.519(23) epoch 11708) -3920> 2014-09-18 18:08:32.434709 7f4c9f251700 5 -- op tracker -- , seq: 86064, time: 2014-09-18 18:08:32.434611, event: throttled, op: pg_notify(2.517(11),2.519(23) epoch 11708) -3919> 2014-09-18 18:08:32.434714 7f4c9f251700 5 -- op tracker -- , seq: 86064, time: 2014-09-18 18:08:32.434664, event: all_read, op: pg_notify(2.517(11),2.519(23) epoch 11708) -3918> 2014-09-18 18:08:32.434718 7f4c9f251700 5 -- op tracker -- , seq: 86064, time: 2014-09-18 18:08:32.434697, event: dispatched, op: pg_notify(2.517(11),2.519(23) epoch 11708) -3917> 2014-09-18 18:08:32.434723 7f4c9f251700 5 -- op tracker -- , seq: 86064, time: 2014-09-18 18:08:32.434721, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.519(23) epoch 11708) -3916> 2014-09-18 18:08:32.434731 7f4c9f251700 5 -- op tracker -- , seq: 86064, time: 2014-09-18 18:08:32.434729, event: started, op: pg_notify(2.517(11),2.519(23) epoch 11708) -3915> 2014-09-18 18:08:32.434762 7f4c9f251700 5 -- op tracker -- , seq: 86064, time: 2014-09-18 18:08:32.434760, event: done, op: pg_notify(2.517(11),2.519(23) epoch 11708) -3914> 2014-09-18 18:08:32.434770 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3913> 2014-09-18 18:08:32.434779 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x12eeb800 con 0x111771e0 -3912> 2014-09-18 18:08:32.434779 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 2.712198 16 0.000160 -3911> 2014-09-18 18:08:32.434928 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -3910> 2014-09-18 18:08:32.435004 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 bft=4,57 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000076 0 0.000000 -3909> 2014-09-18 18:08:32.435018 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 bft=4,57 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -3908> 2014-09-18 18:08:32.435091 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 bft=4,57 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000072 0 0.000000 -3907> 2014-09-18 18:08:32.435104 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 bft=4,57 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 2.712534 0 0.000000 -3906> 2014-09-18 18:08:32.435117 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11696 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 bft=4,57 crt=6868'25546 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -3905> 2014-09-18 18:08:32.435228 7f4c97241700 0 log_channel(default) log [INF] : 2.519 restarting backfill on osd.4 from (0'0,0'0] MAX to 6868'25546 -3904> 2014-09-18 18:08:32.435731 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6819/22324 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0x2fba0640 con 0x1f102840 -3903> 2014-09-18 18:08:32.435745 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6819/22324 -- pg_log(2.519 epoch 11710 log log((4121'22075,6868'25546], crt=6868'25546) query_epoch 11710) v4 -- ?+0 0x10f71500 con 0x1f102840 -3902> 2014-09-18 18:08:32.435893 7f4c97241700 5 osd.42 pg_epoch: 11710 pg[2.519( v 6868'25546 (4121'22075,6868'25546] local-les=11710 n=504 ec=188 les/c 11696/11586 11708/11708/9945) [42,57,4]/[42,6] r=0 lpr=11708 pi=11585-11707/4 bft=4,57 crt=6868'25546 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -3901> 2014-09-18 18:08:32.437975 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0xf85afc0 con 0x1088aec0 -3900> 2014-09-18 18:08:32.438190 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_info(1 pgs e11710:2.519(4)) v4 -- ?+0 0x2b927e00 con 0x1088aec0 -3899> 2014-09-18 18:08:32.438392 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0xe266480 con 0x1395f080 -3898> 2014-09-18 18:08:32.438429 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- pg_info(1 pgs e11710:2.519(4)) v4 -- ?+0 0xea00940 con 0x1395f080 -3897> 2014-09-18 18:08:32.438463 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3896> 2014-09-18 18:08:32.438475 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x12ee9600 con 0x111771e0 -3895> 2014-09-18 18:08:32.438529 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3894> 2014-09-18 18:08:32.438535 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x12ee9000 con 0x111771e0 -3893> 2014-09-18 18:08:32.440112 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.57 10.242.42.176:6801/988 131 ==== pg_info(1 pgs e11710:2.519) v4 ==== 738+0+0 (600668572 0 0) 0xea00940 con 0x1395f080 -3892> 2014-09-18 18:08:32.440129 7f4c9f251700 5 -- op tracker -- , seq: 86065, time: 2014-09-18 18:08:32.439999, event: header_read, op: pg_info(1 pgs e11710:2.519) -3891> 2014-09-18 18:08:32.440140 7f4c9f251700 5 -- op tracker -- , seq: 86065, time: 2014-09-18 18:08:32.440000, event: throttled, op: pg_info(1 pgs e11710:2.519) -3890> 2014-09-18 18:08:32.440145 7f4c9f251700 5 -- op tracker -- , seq: 86065, time: 2014-09-18 18:08:32.440036, event: all_read, op: pg_info(1 pgs e11710:2.519) -3889> 2014-09-18 18:08:32.440151 7f4c9f251700 5 -- op tracker -- , seq: 86065, time: 2014-09-18 18:08:32.440125, event: dispatched, op: pg_info(1 pgs e11710:2.519) -3888> 2014-09-18 18:08:32.440157 7f4c9f251700 5 -- op tracker -- , seq: 86065, time: 2014-09-18 18:08:32.440156, event: waiting_for_osdmap, op: pg_info(1 pgs e11710:2.519) -3887> 2014-09-18 18:08:32.440164 7f4c9f251700 5 -- op tracker -- , seq: 86065, time: 2014-09-18 18:08:32.440163, event: started, op: pg_info(1 pgs e11710:2.519) -3886> 2014-09-18 18:08:32.440181 7f4c9f251700 5 -- op tracker -- , seq: 86065, time: 2014-09-18 18:08:32.440180, event: done, op: pg_info(1 pgs e11710:2.519) -3885> 2014-09-18 18:08:32.440231 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3884> 2014-09-18 18:08:32.440241 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x12eeb200 con 0x111771e0 -3883> 2014-09-18 18:08:32.499114 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.11 10.242.42.168:0/17298 128 ==== osd_ping(ping e11710 stamp 2014-09-18 18:08:32.365912) v2 ==== 47+0+0 (694256223 0 0) 0xf09c1c0 con 0x101cd180 -3882> 2014-09-18 18:08:32.499136 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.365912) v2 -- ?+0 0xf576780 con 0x101cd180 -3881> 2014-09-18 18:08:32.499164 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.11 10.242.42.168:0/17298 128 ==== osd_ping(ping e11710 stamp 2014-09-18 18:08:32.365912) v2 ==== 47+0+0 (694256223 0 0) 0xe76f660 con 0xcc49e40 -3880> 2014-09-18 18:08:32.499184 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.365912) v2 -- ?+0 0xde5f860 con 0xcc49e40 -3879> 2014-09-18 18:08:32.517666 7f4c8fa32700 5 osd.42 11710 heartbeat: osd_stat(228 GB used, 138 GB avail, 366 GB total, peers [0,1,2,3,4,6,9,11,20,22,23,24,27,30,32,35,39,43,47,48,53,55,56,57,58,59,63]/[] op hist []) -3878> 2014-09-18 18:08:32.517681 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6802/15878 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x2b927fe0 con 0x4751e40 -3877> 2014-09-18 18:08:32.517697 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6803/15878 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf09a1e0 con 0x149b0ec0 -3876> 2014-09-18 18:08:32.517721 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6855/39317 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf09b680 con 0x23a75760 -3875> 2014-09-18 18:08:32.517731 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6856/39317 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf812780 con 0x23a746e0 -3874> 2014-09-18 18:08:32.517743 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6818/19139 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf106960 con 0x12b6a940 -3873> 2014-09-18 18:08:32.517760 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6820/19139 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf1f0760 con 0x1d518aa0 -3872> 2014-09-18 18:08:32.517777 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6824/20623 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf1eef00 con 0x1d662b00 -3871> 2014-09-18 18:08:32.517795 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6825/20623 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf1ef0e0 con 0x1d6629a0 -3870> 2014-09-18 18:08:32.517812 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6826/22324 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x2b989c00 con 0x1277db20 -3869> 2014-09-18 18:08:32.517825 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6828/22324 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x2b988d00 con 0x1277d180 -3868> 2014-09-18 18:08:32.517835 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6810/22481 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf578940 con 0x1096fa20 -3867> 2014-09-18 18:08:32.517847 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6813/22481 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf578b20 con 0x10970520 -3866> 2014-09-18 18:08:32.517857 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).connect error 10.242.42.164:6856/39317, (111) Connection refused -3865> 2014-09-18 18:08:32.517882 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6827/24959 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xfae9840 con 0x1db5c420 -3864> 2014-09-18 18:08:32.517865 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).connect error 10.242.42.164:6855/39317, (111) Connection refused -3863> 2014-09-18 18:08:32.517891 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.166:6840/24959 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xfae6780 con 0x1db5c160 -3862> 2014-09-18 18:08:32.517893 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6856/39317 pipe(0x13d50680 sd=29 :0 s=1 pgs=0 cs=0 l=1 c=0x23a746e0).fault (111) Connection refused -3861> 2014-09-18 18:08:32.517895 7f4c790fb700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6855/39317 pipe(0x10b64680 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a75760).fault (111) Connection refused -3860> 2014-09-18 18:08:32.517913 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6822/17298 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xfae6d20 con 0x1217b760 -3859> 2014-09-18 18:08:32.517926 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.168:6823/17298 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xfae8760 con 0x1217c7e0 -3858> 2014-09-18 18:08:32.517936 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6823/33871 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xea01de0 con 0xe58a9a0 -3857> 2014-09-18 18:08:32.517947 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6825/33871 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xe9fe000 con 0xe58a840 -3856> 2014-09-18 18:08:32.517962 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6820/37668 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x19f22f00 con 0x10a407e0 -3855> 2014-09-18 18:08:32.517974 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6832/37668 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x19f23860 con 0x10a40260 -3854> 2014-09-18 18:08:32.517986 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6826/11463 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xedfbfe0 con 0xe417b20 -3853> 2014-09-18 18:08:32.518019 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6827/11463 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xedfbc20 con 0xe58ab00 -3852> 2014-09-18 18:08:32.518045 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6803/5612 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x2b92ffe0 con 0xeff6f20 -3851> 2014-09-18 18:08:32.518045 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6818/19139 183 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe9ff680 con 0x12b6a940 -3850> 2014-09-18 18:08:32.518067 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6810/5612 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x2b92f0e0 con 0xeff6c60 -3849> 2014-09-18 18:08:32.518079 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6802/21548 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xe2865a0 con 0x1da53a20 -3848> 2014-09-18 18:08:32.518096 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.174:6803/21548 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xe289660 con 0x1da53760 -3847> 2014-09-18 18:08:32.518107 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6830/1204 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x24fd0f00 con 0x1db5cdc0 -3846> 2014-09-18 18:08:32.518118 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6831/1204 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x146e03a0 con 0x1db5cc60 -3845> 2014-09-18 18:08:32.518128 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6825/1099 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x12138000 con 0x1db5da20 -3844> 2014-09-18 18:08:32.518148 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6826/1099 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xe956b40 con 0x1db5db80 -3843> 2014-09-18 18:08:32.518186 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6813/17421 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14183860 con 0x125239c0 -3842> 2014-09-18 18:08:32.518206 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6814/17421 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x141841c0 con 0x11d45860 -3841> 2014-09-18 18:08:32.518225 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6817/2660 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14184940 con 0x238dc420 -3840> 2014-09-18 18:08:32.518242 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6818/2660 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x141850c0 con 0x238dcb00 -3839> 2014-09-18 18:08:32.518258 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6843/4511 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14184d00 con 0x149b1de0 -3838> 2014-09-18 18:08:32.518274 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6845/4511 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x141825a0 con 0x149b07e0 -3837> 2014-09-18 18:08:32.518291 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6832/25184 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14184760 con 0x1217d180 -3836> 2014-09-18 18:08:32.518303 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6833/25184 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14184ee0 con 0x1217caa0 -3835> 2014-09-18 18:08:32.518313 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6837/27583 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x141852a0 con 0x10db26e0 -3834> 2014-09-18 18:08:32.518341 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.164:6838/27583 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14185480 con 0x10db4ec0 -3833> 2014-09-18 18:08:32.518356 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6826/2318 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14183c20 con 0x1db5c6e0 -3832> 2014-09-18 18:08:32.518370 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.170:6832/2318 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14182b40 con 0x1db5e260 -3831> 2014-09-18 18:08:32.518403 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6830/19404 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14184b20 con 0xe58d180 -3830> 2014-09-18 18:08:32.518416 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6831/19404 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14183680 con 0xe58d020 -3829> 2014-09-18 18:08:32.518426 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6840/36458 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14185a20 con 0x1364b700 -3828> 2014-09-18 18:08:32.518442 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.172:6842/36458 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14183a40 con 0x1364b860 -3827> 2014-09-18 18:08:32.518460 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6802/988 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0x14182780 con 0x1db5d600 -3826> 2014-09-18 18:08:32.518483 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6803/988 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf8674a0 con 0x1db5dfa0 -3825> 2014-09-18 18:08:32.518496 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6839/46756 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf8670e0 con 0x1d7c7a20 -3824> 2014-09-18 18:08:32.518507 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.176:6842/46756 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf8661e0 con 0x1da543c0 -3823> 2014-09-18 18:08:32.518524 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6842/4508 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf866d20 con 0x1d7c6580 -3822> 2014-09-18 18:08:32.518546 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6844/4508 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf866b40 con 0x1d7c9440 -3821> 2014-09-18 18:08:32.518565 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6805/2487 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf868580 con 0x1d996aa0 -3820> 2014-09-18 18:08:32.518589 7f4c8fa32700 1 -- 10.242.42.178:0/2483 --> 10.242.42.178:6809/2487 -- osd_ping(ping e11710 stamp 2014-09-18 18:08:32.517679) v2 -- ?+0 0xf869c00 con 0x1d996c00 -3819> 2014-09-18 18:08:32.518649 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.2 10.242.42.164:6820/19139 183 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xec69e00 con 0x1d518aa0 -3818> 2014-09-18 18:08:32.518698 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6824/20623 172 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (3888733480 0 0) 0xf272d20 con 0x1d662b00 -3817> 2014-09-18 18:08:32.518724 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6823/20623 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0xcc03cc0 con 0x1f103e40 -3816> 2014-09-18 18:08:32.518747 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.3 10.242.42.164:6825/20623 172 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (3888733480 0 0) 0x1247d0e0 con 0x1d6629a0 -3815> 2014-09-18 18:08:32.518779 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6826/22324 166 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (3888733480 0 0) 0xeb7a3a0 con 0x1277db20 -3814> 2014-09-18 18:08:32.518794 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.4 10.242.42.164:6828/22324 166 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (3888733480 0 0) 0xf72fa40 con 0x1277d180 -3813> 2014-09-18 18:08:32.518807 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6843/4511 164 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe776940 con 0x149b1de0 -3812> 2014-09-18 18:08:32.518819 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.43 10.242.42.178:6845/4511 164 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe0bb2c0 con 0x149b07e0 -3811> 2014-09-18 18:08:32.518830 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6839/46756 162 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf7a63c0 con 0x1d7c7a20 -3810> 2014-09-18 18:08:32.518842 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.58 10.242.42.176:6842/46756 162 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf12d660 con 0x1da543c0 -3809> 2014-09-18 18:08:32.518854 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6802/21548 161 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf2750c0 con 0x1da53a20 -3808> 2014-09-18 18:08:32.518866 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.27 10.242.42.174:6803/21548 161 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0x2b94cb20 con 0x1da53760 -3807> 2014-09-18 18:08:32.518877 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6822/17298 139 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0x24f79680 con 0x1217b760 -3806> 2014-09-18 18:08:32.518888 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.11 10.242.42.168:6823/17298 139 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf811480 con 0x1217c7e0 -3805> 2014-09-18 18:08:32.518899 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6832/25184 139 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (3888733480 0 0) 0x1390c940 con 0x1217d180 -3804> 2014-09-18 18:08:32.518912 7f4c9da4e700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6831/25184 -- osd_map(11710..11710 src has 1..11710) v3 -- ?+0 0xef593c0 con 0x12562260 -3803> 2014-09-18 18:08:32.518924 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.47 10.242.42.164:6833/25184 139 ==== osd_ping(ping_reply e11709 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (3888733480 0 0) 0xf109480 con 0x1217caa0 -3802> 2014-09-18 18:08:32.518958 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6802/988 127 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe0c9de0 con 0x1db5d600 -3801> 2014-09-18 18:08:32.518971 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.57 10.242.42.176:6803/988 127 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe148760 con 0x1db5dfa0 -3800> 2014-09-18 18:08:32.518982 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6825/1099 127 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xec69860 con 0x1db5da20 -3799> 2014-09-18 18:08:32.518998 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.32 10.242.42.176:6826/1099 127 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf29c760 con 0x1db5db80 -3798> 2014-09-18 18:08:32.519012 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6830/1204 127 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xfc57680 con 0x1db5cdc0 -3797> 2014-09-18 18:08:32.519024 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.30 10.242.42.176:6831/1204 127 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf8665a0 con 0x1db5cc60 -3796> 2014-09-18 18:08:32.519035 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6827/24959 125 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0x24f6fde0 con 0x1db5c420 -3795> 2014-09-18 18:08:32.519046 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.9 10.242.42.166:6840/24959 125 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe148940 con 0x1db5c160 -3794> 2014-09-18 18:08:32.519057 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6826/2318 125 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf726b40 con 0x1db5c6e0 -3793> 2014-09-18 18:08:32.519070 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.53 10.242.42.170:6832/2318 125 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe0c8d00 con 0x1db5e260 -3792> 2014-09-18 18:08:32.519081 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6837/27583 105 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf0981c0 con 0x10db26e0 -3791> 2014-09-18 18:08:32.519092 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.48 10.242.42.164:6838/27583 105 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe0bb0e0 con 0x10db4ec0 -3790> 2014-09-18 18:08:32.519103 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.59 10.242.42.178:6842/4508 2 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf866d20 con 0x1d7c6580 -3789> 2014-09-18 18:08:32.519144 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.59 10.242.42.178:6844/4508 2 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe775c20 con 0x1d7c9440 -3788> 2014-09-18 18:08:32.519157 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6826/11463 1578 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf1921e0 con 0xe417b20 -3787> 2014-09-18 18:08:32.519239 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.23 10.242.42.172:6827/11463 1578 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe7750e0 con 0xe58ab00 -3786> 2014-09-18 18:08:32.519252 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6823/33871 1578 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xda31de0 con 0xe58a9a0 -3785> 2014-09-18 18:08:32.519264 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.20 10.242.42.172:6825/33871 1578 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf974940 con 0xe58a840 -3784> 2014-09-18 18:08:32.519275 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6830/19404 1578 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf1e7860 con 0xe58d180 -3783> 2014-09-18 18:08:32.519286 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.55 10.242.42.172:6831/19404 1578 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf7a92a0 con 0xe58d020 -3782> 2014-09-18 18:08:32.519297 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6840/36458 658 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0x24fa0d20 con 0x1364b700 -3781> 2014-09-18 18:08:32.519309 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.56 10.242.42.172:6842/36458 658 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf1610c0 con 0x1364b860 -3780> 2014-09-18 18:08:32.519320 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6810/22481 458 ==== osd_ping(ping_reply e11708 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (2674802085 0 0) 0x2b922780 con 0x1096fa20 -3779> 2014-09-18 18:08:32.519331 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.6 10.242.42.166:6813/22481 458 ==== osd_ping(ping_reply e11708 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (2674802085 0 0) 0x100ef2c0 con 0x10970520 -3778> 2014-09-18 18:08:32.519342 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6820/37668 452 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf972780 con 0x10a407e0 -3777> 2014-09-18 18:08:32.519353 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.22 10.242.42.172:6832/37668 452 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf1870e0 con 0x10a40260 -3776> 2014-09-18 18:08:32.519364 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6803/5612 344 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf0b8940 con 0xeff6f20 -3775> 2014-09-18 18:08:32.519375 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.24 10.242.42.172:6810/5612 344 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe289c00 con 0xeff6c60 -3774> 2014-09-18 18:08:32.519386 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6805/2487 333 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0x1368d2c0 con 0x1d996aa0 -3773> 2014-09-18 18:08:32.519398 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.63 10.242.42.178:6809/2487 333 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf342780 con 0x1d996c00 -3772> 2014-09-18 18:08:32.519408 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6802/15878 192 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf728760 con 0x4751e40 -3771> 2014-09-18 18:08:32.519419 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.0 10.242.42.164:6803/15878 192 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xe0bda20 con 0x149b0ec0 -3770> 2014-09-18 18:08:32.519430 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6813/17421 189 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0x1310f0c0 con 0x125239c0 -3769> 2014-09-18 18:08:32.519441 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.35 10.242.42.164:6814/17421 189 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0x2b930940 con 0x11d45860 -3768> 2014-09-18 18:08:32.519451 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6817/2660 187 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0xf2ae780 con 0x238dc420 -3767> 2014-09-18 18:08:32.519463 7f4c9da4e700 1 -- 10.242.42.178:0/2483 <== osd.39 10.242.42.172:6818/2660 187 ==== osd_ping(ping_reply e11710 stamp 2014-09-18 18:08:32.517679) v2 ==== 47+0+0 (1847655615 0 0) 0x2b9252a0 con 0x238dcb00 -3766> 2014-09-18 18:08:32.529551 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3652 ==== pg_notify(2.40a(162) epoch 11710) v5 ==== 9736+0+0 (528107569 0 0) 0xf972000 con 0xcbecec0 -3765> 2014-09-18 18:08:32.529568 7f4c9f251700 5 -- op tracker -- , seq: 86066, time: 2014-09-18 18:08:32.529416, event: header_read, op: pg_notify(2.40a(162) epoch 11710) -3764> 2014-09-18 18:08:32.529578 7f4c9f251700 5 -- op tracker -- , seq: 86066, time: 2014-09-18 18:08:32.529417, event: throttled, op: pg_notify(2.40a(162) epoch 11710) -3763> 2014-09-18 18:08:32.529585 7f4c9f251700 5 -- op tracker -- , seq: 86066, time: 2014-09-18 18:08:32.529536, event: all_read, op: pg_notify(2.40a(162) epoch 11710) -3762> 2014-09-18 18:08:32.529591 7f4c9f251700 5 -- op tracker -- , seq: 86066, time: 2014-09-18 18:08:32.529564, event: dispatched, op: pg_notify(2.40a(162) epoch 11710) -3761> 2014-09-18 18:08:32.529597 7f4c9f251700 5 -- op tracker -- , seq: 86066, time: 2014-09-18 18:08:32.529596, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11710) -3760> 2014-09-18 18:08:32.529605 7f4c9f251700 5 -- op tracker -- , seq: 86066, time: 2014-09-18 18:08:32.529604, event: started, op: pg_notify(2.40a(162) epoch 11710) -3759> 2014-09-18 18:08:32.529625 7f4c9f251700 5 -- op tracker -- , seq: 86066, time: 2014-09-18 18:08:32.529623, event: done, op: pg_notify(2.40a(162) epoch 11710) -3758> 2014-09-18 18:08:32.529645 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3757> 2014-09-18 18:08:32.529654 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x12eeb000 con 0x111771e0 -3756> 2014-09-18 18:08:32.538266 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3653 ==== pg_notify(2.78f(56) epoch 11710) v5 ==== 3722+0+0 (1281029204 0 0) 0xf9734a0 con 0xcbecec0 -3755> 2014-09-18 18:08:32.538279 7f4c9f251700 5 -- op tracker -- , seq: 86067, time: 2014-09-18 18:08:32.538197, event: header_read, op: pg_notify(2.78f(56) epoch 11710) -3754> 2014-09-18 18:08:32.538286 7f4c9f251700 5 -- op tracker -- , seq: 86067, time: 2014-09-18 18:08:32.538198, event: throttled, op: pg_notify(2.78f(56) epoch 11710) -3753> 2014-09-18 18:08:32.538294 7f4c9f251700 5 -- op tracker -- , seq: 86067, time: 2014-09-18 18:08:32.538253, event: all_read, op: pg_notify(2.78f(56) epoch 11710) -3752> 2014-09-18 18:08:32.538309 7f4c9f251700 5 -- op tracker -- , seq: 86067, time: 2014-09-18 18:08:32.538275, event: dispatched, op: pg_notify(2.78f(56) epoch 11710) -3751> 2014-09-18 18:08:32.538317 7f4c9f251700 5 -- op tracker -- , seq: 86067, time: 2014-09-18 18:08:32.538315, event: waiting_for_osdmap, op: pg_notify(2.78f(56) epoch 11710) -3750> 2014-09-18 18:08:32.538324 7f4c9f251700 5 -- op tracker -- , seq: 86067, time: 2014-09-18 18:08:32.538323, event: started, op: pg_notify(2.78f(56) epoch 11710) -3749> 2014-09-18 18:08:32.538341 7f4c9f251700 5 -- op tracker -- , seq: 86067, time: 2014-09-18 18:08:32.538339, event: done, op: pg_notify(2.78f(56) epoch 11710) -3748> 2014-09-18 18:08:32.538373 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3747> 2014-09-18 18:08:32.538382 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11710 {2.4e8=[6,57,59]} v11710) v1 -- ?+0 0x12eea200 con 0x111771e0 -3746> 2014-09-18 18:08:32.681173 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 47 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x19df3840 con 0x1395f340 -3745> 2014-09-18 18:08:32.681243 7f4c9f251700 3 osd.42 11710 handle_osd_map epochs [11711,11711], i have 11710, src has [1,11711] -3744> 2014-09-18 18:08:32.682307 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 30 ==== osd_map(11710..11711 src has 1..11711) v3 ==== 833+0+0 (390400427 0 0) 0x24d1c400 con 0x111771e0 -3743> 2014-09-18 18:08:32.683007 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.56 10.242.42.172:0/36458 653 ==== osd_ping(ping e11710 stamp 2014-09-18 23:37:21.787484) v2 ==== 47+0+0 (1954285583 0 0) 0xf0e3e00 con 0x14392c60 -3742> 2014-09-18 18:08:32.683042 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/36458 -- osd_ping(ping_reply e11710 stamp 2014-09-18 23:37:21.787484) v2 -- ?+0 0xf09c1c0 con 0x14392c60 -3741> 2014-09-18 18:08:32.683101 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.56 10.242.42.172:0/36458 653 ==== osd_ping(ping e11710 stamp 2014-09-18 23:37:21.787484) v2 ==== 47+0+0 (1954285583 0 0) 0x21d15a20 con 0x10a81700 -3740> 2014-09-18 18:08:32.683134 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/36458 -- osd_ping(ping_reply e11710 stamp 2014-09-18 23:37:21.787484) v2 -- ?+0 0xe76f660 con 0x10a81700 -3739> 2014-09-18 18:08:32.683966 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3738> 2014-09-18 18:08:32.683975 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11711 {2.4e8=[6,57,59]} v11711) v1 -- ?+0 0x12eeaa00 con 0x111771e0 -3737> 2014-09-18 18:08:32.683994 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3736> 2014-09-18 18:08:32.684004 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11711 {2.4e8=[6,57,59]} v11711) v1 -- ?+0 0x12ee8000 con 0x111771e0 -3735> 2014-09-18 18:08:32.684450 7f4ca0a54700 3 osd.42 11711 handle_osd_map epochs [11710,11711], i have 11711, src has [1,11711] -3734> 2014-09-18 18:08:32.684487 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 48 ==== pg_notify(2.1d8(9) epoch 11711) v5 ==== 1261+0+0 (470150545 0 0) 0xf0b6780 con 0x1395f340 -3733> 2014-09-18 18:08:32.684503 7f4c9f251700 5 -- op tracker -- , seq: 86068, time: 2014-09-18 18:08:32.681172, event: header_read, op: pg_notify(2.1d8(9) epoch 11711) -3732> 2014-09-18 18:08:32.684513 7f4c9f251700 5 -- op tracker -- , seq: 86068, time: 2014-09-18 18:08:32.681172, event: throttled, op: pg_notify(2.1d8(9) epoch 11711) -3731> 2014-09-18 18:08:32.684521 7f4c9f251700 5 -- op tracker -- , seq: 86068, time: 2014-09-18 18:08:32.681208, event: all_read, op: pg_notify(2.1d8(9) epoch 11711) -3730> 2014-09-18 18:08:32.684591 7f4c9f251700 5 -- op tracker -- , seq: 86068, time: 2014-09-18 18:08:32.684498, event: dispatched, op: pg_notify(2.1d8(9) epoch 11711) -3729> 2014-09-18 18:08:32.684599 7f4c9f251700 5 -- op tracker -- , seq: 86068, time: 2014-09-18 18:08:32.684597, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11711) -3728> 2014-09-18 18:08:32.684609 7f4c9f251700 5 -- op tracker -- , seq: 86068, time: 2014-09-18 18:08:32.684606, event: started, op: pg_notify(2.1d8(9) epoch 11711) -3727> 2014-09-18 18:08:32.684625 7f4c9f251700 5 -- op tracker -- , seq: 86068, time: 2014-09-18 18:08:32.684624, event: done, op: pg_notify(2.1d8(9) epoch 11711) -3726> 2014-09-18 18:08:32.684636 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2015 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x2d7fdcc0 con 0xe2d9a20 -3725> 2014-09-18 18:08:32.684645 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3724> 2014-09-18 18:08:32.684651 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2016 ==== pg_notify(2.26d(9) epoch 11711) v5 ==== 1257+0+0 (1319829862 0 0) 0xf09f860 con 0xe2d9a20 -3723> 2014-09-18 18:08:32.684668 7f4c9f251700 5 -- op tracker -- , seq: 86069, time: 2014-09-18 18:08:32.684588, event: header_read, op: pg_notify(2.26d(9) epoch 11711) -3722> 2014-09-18 18:08:32.684680 7f4c9f251700 5 -- op tracker -- , seq: 86069, time: 2014-09-18 18:08:32.684589, event: throttled, op: pg_notify(2.26d(9) epoch 11711) -3721> 2014-09-18 18:08:32.684690 7f4c9f251700 5 -- op tracker -- , seq: 86069, time: 2014-09-18 18:08:32.684616, event: all_read, op: pg_notify(2.26d(9) epoch 11711) -3720> 2014-09-18 18:08:32.684700 7f4c9f251700 5 -- op tracker -- , seq: 86069, time: 2014-09-18 18:08:32.684664, event: dispatched, op: pg_notify(2.26d(9) epoch 11711) -3719> 2014-09-18 18:08:32.684711 7f4c9f251700 5 -- op tracker -- , seq: 86069, time: 2014-09-18 18:08:32.684709, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11711) -3718> 2014-09-18 18:08:32.684723 7f4c9f251700 5 -- op tracker -- , seq: 86069, time: 2014-09-18 18:08:32.684720, event: started, op: pg_notify(2.26d(9) epoch 11711) -3717> 2014-09-18 18:08:32.684829 7f4c9f251700 5 -- op tracker -- , seq: 86069, time: 2014-09-18 18:08:32.684825, event: done, op: pg_notify(2.26d(9) epoch 11711) -3716> 2014-09-18 18:08:32.685082 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11711..11711 src has 1..11711) v3 -- ?+0 0x2fba1a80 con 0x140f95a0 -3715> 2014-09-18 18:08:32.685099 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11711) v5 -- ?+0 0xf577a40 con 0x140f95a0 -3714> 2014-09-18 18:08:32.685137 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3713> 2014-09-18 18:08:32.685136 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1044 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0xcc02d00 con 0x14358260 -3712> 2014-09-18 18:08:32.685145 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11711 {2.4e8=[6,57,59]} v11711) v1 -- ?+0 0x12eeba00 con 0x111771e0 -3711> 2014-09-18 18:08:32.685165 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3710> 2014-09-18 18:08:32.685255 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1045 ==== pg_notify(2.719(17) epoch 11711) v5 ==== 1601+0+0 (2055362414 0 0) 0xf72ef00 con 0x14358260 -3709> 2014-09-18 18:08:32.685291 7f4c9f251700 5 -- op tracker -- , seq: 86070, time: 2014-09-18 18:08:32.685207, event: header_read, op: pg_notify(2.719(17) epoch 11711) -3708> 2014-09-18 18:08:32.685301 7f4c9f251700 5 -- op tracker -- , seq: 86070, time: 2014-09-18 18:08:32.685208, event: throttled, op: pg_notify(2.719(17) epoch 11711) -3707> 2014-09-18 18:08:32.685308 7f4c9f251700 5 -- op tracker -- , seq: 86070, time: 2014-09-18 18:08:32.685241, event: all_read, op: pg_notify(2.719(17) epoch 11711) -3706> 2014-09-18 18:08:32.685319 7f4c9f251700 5 -- op tracker -- , seq: 86070, time: 2014-09-18 18:08:32.685277, event: dispatched, op: pg_notify(2.719(17) epoch 11711) -3705> 2014-09-18 18:08:32.685331 7f4c9f251700 5 -- op tracker -- , seq: 86070, time: 2014-09-18 18:08:32.685328, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11711) -3704> 2014-09-18 18:08:32.685342 7f4c9f251700 5 -- op tracker -- , seq: 86070, time: 2014-09-18 18:08:32.685340, event: started, op: pg_notify(2.719(17) epoch 11711) -3703> 2014-09-18 18:08:32.685365 7f4c9f251700 5 -- op tracker -- , seq: 86070, time: 2014-09-18 18:08:32.685362, event: done, op: pg_notify(2.719(17) epoch 11711) -3702> 2014-09-18 18:08:32.685389 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11711) v5 -- ?+0 0x11bbeb40 con 0x140f95a0 -3701> 2014-09-18 18:08:32.685411 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11711..11711 src has 1..11711) v3 -- ?+0 0x1182bd40 con 0x1395edc0 -3700> 2014-09-18 18:08:32.685422 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11711) v5 -- ?+0 0x11bc03a0 con 0x1395edc0 -3699> 2014-09-18 18:08:32.685504 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11711..11711 src has 1..11711) v3 -- ?+0 0x12bf2d00 con 0x1395f1e0 -3698> 2014-09-18 18:08:32.685521 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11711) v5 -- ?+0 0x11bbe1e0 con 0x1395f1e0 -3697> 2014-09-18 18:08:32.685556 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3696> 2014-09-18 18:08:32.685563 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11711 {2.4e8=[6,57,59]} v11711) v1 -- ?+0 0x1236c000 con 0x111771e0 -3695> 2014-09-18 18:08:32.685554 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1268 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x12bf3a80 con 0xe8db760 -3694> 2014-09-18 18:08:32.685585 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3693> 2014-09-18 18:08:32.685615 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1269 ==== pg_notify(2.17e(17) epoch 11711) v5 ==== 1593+0+0 (2166021876 0 0) 0xf0dc1c0 con 0xe8db760 -3692> 2014-09-18 18:08:32.685634 7f4c9f251700 5 -- op tracker -- , seq: 86071, time: 2014-09-18 18:08:32.685556, event: header_read, op: pg_notify(2.17e(17) epoch 11711) -3691> 2014-09-18 18:08:32.685648 7f4c9f251700 5 -- op tracker -- , seq: 86071, time: 2014-09-18 18:08:32.685556, event: throttled, op: pg_notify(2.17e(17) epoch 11711) -3690> 2014-09-18 18:08:32.685658 7f4c9f251700 5 -- op tracker -- , seq: 86071, time: 2014-09-18 18:08:32.685601, event: all_read, op: pg_notify(2.17e(17) epoch 11711) -3689> 2014-09-18 18:08:32.685669 7f4c9f251700 5 -- op tracker -- , seq: 86071, time: 2014-09-18 18:08:32.685629, event: dispatched, op: pg_notify(2.17e(17) epoch 11711) -3688> 2014-09-18 18:08:32.685680 7f4c9f251700 5 -- op tracker -- , seq: 86071, time: 2014-09-18 18:08:32.685677, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11711) -3687> 2014-09-18 18:08:32.685692 7f4c9f251700 5 -- op tracker -- , seq: 86071, time: 2014-09-18 18:08:32.685689, event: started, op: pg_notify(2.17e(17) epoch 11711) -3686> 2014-09-18 18:08:32.685715 7f4c9f251700 5 -- op tracker -- , seq: 86071, time: 2014-09-18 18:08:32.685712, event: done, op: pg_notify(2.17e(17) epoch 11711) -3685> 2014-09-18 18:08:32.685752 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3684> 2014-09-18 18:08:32.685760 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11711 {2.4e8=[6,57,59]} v11711) v1 -- ?+0 0x12ee9e00 con 0x111771e0 -3683> 2014-09-18 18:08:32.686079 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3682> 2014-09-18 18:08:32.686087 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11711 {2.4e8=[6,57,59]} v11711) v1 -- ?+0 0x1236f200 con 0x111771e0 -3681> 2014-09-18 18:08:32.686149 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 540 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0xe936240 con 0x1395ef20 -3680> 2014-09-18 18:08:32.686169 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3679> 2014-09-18 18:08:32.686194 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 541 ==== pg_notify(2.4d0(10) epoch 11711) v5 ==== 1292+0+0 (1649145814 0 0) 0xf09d660 con 0x1395ef20 -3678> 2014-09-18 18:08:32.686198 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11711..11711 src has 1..11711) v3 -- ?+0 0x2fb9ed80 con 0xcbecec0 -3677> 2014-09-18 18:08:32.686214 7f4c9f251700 5 -- op tracker -- , seq: 86072, time: 2014-09-18 18:08:32.686148, event: header_read, op: pg_notify(2.4d0(10) epoch 11711) -3676> 2014-09-18 18:08:32.686219 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(11) epoch 11711) v5 -- ?+0 0xf577860 con 0xcbecec0 -3675> 2014-09-18 18:08:32.686227 7f4c9f251700 5 -- op tracker -- , seq: 86072, time: 2014-09-18 18:08:32.686149, event: throttled, op: pg_notify(2.4d0(10) epoch 11711) -3674> 2014-09-18 18:08:32.686232 7f4c9f251700 5 -- op tracker -- , seq: 86072, time: 2014-09-18 18:08:32.686182, event: all_read, op: pg_notify(2.4d0(10) epoch 11711) -3673> 2014-09-18 18:08:32.686240 7f4c9f251700 5 -- op tracker -- , seq: 86072, time: 2014-09-18 18:08:32.686209, event: dispatched, op: pg_notify(2.4d0(10) epoch 11711) -3672> 2014-09-18 18:08:32.686250 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3671> 2014-09-18 18:08:32.686250 7f4c9f251700 5 -- op tracker -- , seq: 86072, time: 2014-09-18 18:08:32.686248, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11711) -3670> 2014-09-18 18:08:32.686257 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11711 {2.4e8=[6,57,59]} v11711) v1 -- ?+0 0x12eeb800 con 0x111771e0 -3669> 2014-09-18 18:08:32.686268 7f4c9f251700 5 -- op tracker -- , seq: 86072, time: 2014-09-18 18:08:32.686265, event: started, op: pg_notify(2.4d0(10) epoch 11711) -3668> 2014-09-18 18:08:32.686288 7f4c9f251700 5 -- op tracker -- , seq: 86072, time: 2014-09-18 18:08:32.686287, event: done, op: pg_notify(2.4d0(10) epoch 11711) -3667> 2014-09-18 18:08:32.686317 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -3666> 2014-09-18 18:08:32.686322 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11711 {2.4e8=[6,57,59]} v11711) v1 -- ?+0 0x1de9a200 con 0x111771e0 -3665> 2014-09-18 18:08:32.686706 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] exit Started/Primary/Peering/WaitActingChange 0.334918 1 0.000132 -3664> 2014-09-18 18:08:32.686741 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] exit Started/Primary 0.334961 0 0.000000 -3663> 2014-09-18 18:08:32.686750 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] exit Started 2.959585 0 0.000000 -3662> 2014-09-18 18:08:32.686803 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11708/11708) [42,3,57] r=0 lpr=11708 pi=11385-11707/7 crt=0'0 mlcod 0'0 inactive] enter Reset -3661> 2014-09-18 18:08:32.686848 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11711/11711) [42,3,57]/[6,57,59] r=-1 lpr=11711 pi=11385-11710/8 crt=0'0 remapped NOTIFY] exit Reset 0.000045 1 0.000136 -3660> 2014-09-18 18:08:32.686859 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11711/11711) [42,3,57]/[6,57,59] r=-1 lpr=11711 pi=11385-11710/8 crt=0'0 remapped NOTIFY] enter Started -3659> 2014-09-18 18:08:32.686867 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11711/11711) [42,3,57]/[6,57,59] r=-1 lpr=11711 pi=11385-11710/8 crt=0'0 remapped NOTIFY] enter Start -3658> 2014-09-18 18:08:32.686874 7f4c96a40700 1 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11711/11711) [42,3,57]/[6,57,59] r=-1 lpr=11711 pi=11385-11710/8 crt=0'0 remapped NOTIFY] state: transitioning to Stray -3657> 2014-09-18 18:08:32.686883 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11711/11711) [42,3,57]/[6,57,59] r=-1 lpr=11711 pi=11385-11710/8 crt=0'0 remapped NOTIFY] exit Start 0.000015 0 0.000000 -3656> 2014-09-18 18:08:32.686906 7f4c96a40700 5 osd.42 pg_epoch: 11711 pg[2.4e8( empty local-les=0 n=0 ec=188 les/c 11488/11488 11708/11711/11711) [42,3,57]/[6,57,59] r=-1 lpr=11711 pi=11385-11710/8 crt=0'0 remapped NOTIFY] enter Started/Stray -3655> 2014-09-18 18:08:32.687062 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11711..11711 src has 1..11711) v3 -- ?+0 0x22e16480 con 0x1088aec0 -3654> 2014-09-18 18:08:32.687077 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.4e8(8) epoch 11711) v5 -- ?+0 0xf2b8ee0 con 0x1088aec0 -3653> 2014-09-18 18:08:32.687107 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11711..11711 src has 1..11711) v3 -- ?+0 0x22e16240 con 0x23a74f20 -3652> 2014-09-18 18:08:32.687159 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11711) v5 -- ?+0 0xf2b7860 con 0x23a74f20 -3651> 2014-09-18 18:08:32.687579 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2600 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x14408000 con 0x4900b00 -3650> 2014-09-18 18:08:32.687596 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3649> 2014-09-18 18:08:32.687631 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2601 ==== pg_notify(2.71c(12) epoch 11711) v5 ==== 1424+0+0 (3184415381 0 0) 0xf7a7860 con 0x4900b00 -3648> 2014-09-18 18:08:32.687641 7f4c9f251700 5 -- op tracker -- , seq: 86073, time: 2014-09-18 18:08:32.687590, event: header_read, op: pg_notify(2.71c(12) epoch 11711) -3647> 2014-09-18 18:08:32.687651 7f4c9f251700 5 -- op tracker -- , seq: 86073, time: 2014-09-18 18:08:32.687590, event: throttled, op: pg_notify(2.71c(12) epoch 11711) -3646> 2014-09-18 18:08:32.687657 7f4c9f251700 5 -- op tracker -- , seq: 86073, time: 2014-09-18 18:08:32.687620, event: all_read, op: pg_notify(2.71c(12) epoch 11711) -3645> 2014-09-18 18:08:32.687663 7f4c9f251700 5 -- op tracker -- , seq: 86073, time: 2014-09-18 18:08:32.687638, event: dispatched, op: pg_notify(2.71c(12) epoch 11711) -3644> 2014-09-18 18:08:32.687669 7f4c9f251700 5 -- op tracker -- , seq: 86073, time: 2014-09-18 18:08:32.687668, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11711) -3643> 2014-09-18 18:08:32.687677 7f4c9f251700 5 -- op tracker -- , seq: 86073, time: 2014-09-18 18:08:32.687675, event: started, op: pg_notify(2.71c(12) epoch 11711) -3642> 2014-09-18 18:08:32.687694 7f4c9f251700 5 -- op tracker -- , seq: 86073, time: 2014-09-18 18:08:32.687692, event: done, op: pg_notify(2.71c(12) epoch 11711) -3641> 2014-09-18 18:08:32.687937 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3302 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0xf85ab40 con 0x4900580 -3640> 2014-09-18 18:08:32.687947 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3639> 2014-09-18 18:08:32.688336 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3303 ==== pg_notify(2.4f6(184) epoch 11711) v5 ==== 11296+0+0 (2822147685 0 0) 0xf91af00 con 0x4900580 -3638> 2014-09-18 18:08:32.688351 7f4c9f251700 5 -- op tracker -- , seq: 86074, time: 2014-09-18 18:08:32.688173, event: header_read, op: pg_notify(2.4f6(184) epoch 11711) -3637> 2014-09-18 18:08:32.688361 7f4c9f251700 5 -- op tracker -- , seq: 86074, time: 2014-09-18 18:08:32.688174, event: throttled, op: pg_notify(2.4f6(184) epoch 11711) -3636> 2014-09-18 18:08:32.688368 7f4c9f251700 5 -- op tracker -- , seq: 86074, time: 2014-09-18 18:08:32.688324, event: all_read, op: pg_notify(2.4f6(184) epoch 11711) -3635> 2014-09-18 18:08:32.688374 7f4c9f251700 5 -- op tracker -- , seq: 86074, time: 2014-09-18 18:08:32.688347, event: dispatched, op: pg_notify(2.4f6(184) epoch 11711) -3634> 2014-09-18 18:08:32.688381 7f4c9f251700 5 -- op tracker -- , seq: 86074, time: 2014-09-18 18:08:32.688380, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11711) -3633> 2014-09-18 18:08:32.688389 7f4c9f251700 5 -- op tracker -- , seq: 86074, time: 2014-09-18 18:08:32.688387, event: started, op: pg_notify(2.4f6(184) epoch 11711) -3632> 2014-09-18 18:08:32.688406 7f4c9f251700 5 -- op tracker -- , seq: 86074, time: 2014-09-18 18:08:32.688404, event: done, op: pg_notify(2.4f6(184) epoch 11711) -3631> 2014-09-18 18:08:32.688426 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3304 ==== pg_notify(2.17e(17) epoch 11711) v5 ==== 1593+0+0 (2166021876 0 0) 0xf91b2c0 con 0x4900580 -3630> 2014-09-18 18:08:32.688437 7f4c9f251700 5 -- op tracker -- , seq: 86075, time: 2014-09-18 18:08:32.688344, event: header_read, op: pg_notify(2.17e(17) epoch 11711) -3629> 2014-09-18 18:08:32.688443 7f4c9f251700 5 -- op tracker -- , seq: 86075, time: 2014-09-18 18:08:32.688345, event: throttled, op: pg_notify(2.17e(17) epoch 11711) -3628> 2014-09-18 18:08:32.688436 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 31 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x10e22240 con 0x111771e0 -3627> 2014-09-18 18:08:32.688452 7f4c9f251700 5 -- op tracker -- , seq: 86075, time: 2014-09-18 18:08:32.688382, event: all_read, op: pg_notify(2.17e(17) epoch 11711) -3626> 2014-09-18 18:08:32.688465 7f4c9f251700 5 -- op tracker -- , seq: 86075, time: 2014-09-18 18:08:32.688434, event: dispatched, op: pg_notify(2.17e(17) epoch 11711) -3625> 2014-09-18 18:08:32.688472 7f4c9f251700 5 -- op tracker -- , seq: 86075, time: 2014-09-18 18:08:32.688470, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11711) -3624> 2014-09-18 18:08:32.688479 7f4c9f251700 5 -- op tracker -- , seq: 86075, time: 2014-09-18 18:08:32.688478, event: started, op: pg_notify(2.17e(17) epoch 11711) -3623> 2014-09-18 18:08:32.688496 7f4c9f251700 5 -- op tracker -- , seq: 86075, time: 2014-09-18 18:08:32.688494, event: done, op: pg_notify(2.17e(17) epoch 11711) -3622> 2014-09-18 18:08:32.688512 7f4ca0a54700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3621> 2014-09-18 18:08:32.688973 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1200 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x1182bf80 con 0x1088b440 -3620> 2014-09-18 18:08:32.688987 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3619> 2014-09-18 18:08:32.689283 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1201 ==== pg_notify(2.2d8(173) epoch 11711) v5 ==== 10757+0+0 (3661695338 0 0) 0xe76d860 con 0x1088b440 -3618> 2014-09-18 18:08:32.689294 7f4c9f251700 5 -- op tracker -- , seq: 86076, time: 2014-09-18 18:08:32.689155, event: header_read, op: pg_notify(2.2d8(173) epoch 11711) -3617> 2014-09-18 18:08:32.689302 7f4c9f251700 5 -- op tracker -- , seq: 86076, time: 2014-09-18 18:08:32.689156, event: throttled, op: pg_notify(2.2d8(173) epoch 11711) -3616> 2014-09-18 18:08:32.689308 7f4c9f251700 5 -- op tracker -- , seq: 86076, time: 2014-09-18 18:08:32.689272, event: all_read, op: pg_notify(2.2d8(173) epoch 11711) -3615> 2014-09-18 18:08:32.689314 7f4c9f251700 5 -- op tracker -- , seq: 86076, time: 2014-09-18 18:08:32.689291, event: dispatched, op: pg_notify(2.2d8(173) epoch 11711) -3614> 2014-09-18 18:08:32.689320 7f4c9f251700 5 -- op tracker -- , seq: 86076, time: 2014-09-18 18:08:32.689318, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11711) -3613> 2014-09-18 18:08:32.689327 7f4c9f251700 5 -- op tracker -- , seq: 86076, time: 2014-09-18 18:08:32.689326, event: started, op: pg_notify(2.2d8(173) epoch 11711) -3612> 2014-09-18 18:08:32.689343 7f4c9f251700 5 -- op tracker -- , seq: 86076, time: 2014-09-18 18:08:32.689342, event: done, op: pg_notify(2.2d8(173) epoch 11711) -3611> 2014-09-18 18:08:32.690074 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3305 ==== pg_notify(2.1bb(9) epoch 11711) v5 ==== 1189+0+0 (1274510461 0 0) 0xf1ec760 con 0x4900580 -3610> 2014-09-18 18:08:32.690088 7f4c9f251700 5 -- op tracker -- , seq: 86077, time: 2014-09-18 18:08:32.690030, event: header_read, op: pg_notify(2.1bb(9) epoch 11711) -3609> 2014-09-18 18:08:32.690097 7f4c9f251700 5 -- op tracker -- , seq: 86077, time: 2014-09-18 18:08:32.690031, event: throttled, op: pg_notify(2.1bb(9) epoch 11711) -3608> 2014-09-18 18:08:32.690103 7f4c9f251700 5 -- op tracker -- , seq: 86077, time: 2014-09-18 18:08:32.690063, event: all_read, op: pg_notify(2.1bb(9) epoch 11711) -3607> 2014-09-18 18:08:32.690108 7f4c9f251700 5 -- op tracker -- , seq: 86077, time: 2014-09-18 18:08:32.690085, event: dispatched, op: pg_notify(2.1bb(9) epoch 11711) -3606> 2014-09-18 18:08:32.690115 7f4c9f251700 5 -- op tracker -- , seq: 86077, time: 2014-09-18 18:08:32.690113, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11711) -3605> 2014-09-18 18:08:32.690122 7f4c9f251700 5 -- op tracker -- , seq: 86077, time: 2014-09-18 18:08:32.690121, event: started, op: pg_notify(2.1bb(9) epoch 11711) -3604> 2014-09-18 18:08:32.690139 7f4c9f251700 5 -- op tracker -- , seq: 86077, time: 2014-09-18 18:08:32.690137, event: done, op: pg_notify(2.1bb(9) epoch 11711) -3603> 2014-09-18 18:08:32.690343 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 790 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x2d7fbd40 con 0xe9a27e0 -3602> 2014-09-18 18:08:32.690354 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3601> 2014-09-18 18:08:32.690615 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 791 ==== pg_notify(2.2d8(173) epoch 11711) v5 ==== 10757+0+0 (3661695338 0 0) 0x14220ee0 con 0xe9a27e0 -3600> 2014-09-18 18:08:32.690639 7f4c9f251700 5 -- op tracker -- , seq: 86078, time: 2014-09-18 18:08:32.690416, event: header_read, op: pg_notify(2.2d8(173) epoch 11711) -3599> 2014-09-18 18:08:32.690650 7f4c9f251700 5 -- op tracker -- , seq: 86078, time: 2014-09-18 18:08:32.690417, event: throttled, op: pg_notify(2.2d8(173) epoch 11711) -3598> 2014-09-18 18:08:32.690659 7f4c9f251700 5 -- op tracker -- , seq: 86078, time: 2014-09-18 18:08:32.690580, event: all_read, op: pg_notify(2.2d8(173) epoch 11711) -3597> 2014-09-18 18:08:32.690668 7f4c9f251700 5 -- op tracker -- , seq: 86078, time: 2014-09-18 18:08:32.690633, event: dispatched, op: pg_notify(2.2d8(173) epoch 11711) -3596> 2014-09-18 18:08:32.690675 7f4c9f251700 5 -- op tracker -- , seq: 86078, time: 2014-09-18 18:08:32.690673, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11711) -3595> 2014-09-18 18:08:32.690682 7f4c9f251700 5 -- op tracker -- , seq: 86078, time: 2014-09-18 18:08:32.690681, event: started, op: pg_notify(2.2d8(173) epoch 11711) -3594> 2014-09-18 18:08:32.690700 7f4c9f251700 5 -- op tracker -- , seq: 86078, time: 2014-09-18 18:08:32.690698, event: done, op: pg_notify(2.2d8(173) epoch 11711) -3593> 2014-09-18 18:08:32.697044 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3218 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0xe937440 con 0x1395ec60 -3592> 2014-09-18 18:08:32.697060 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3591> 2014-09-18 18:08:32.697099 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3219 ==== pg_notify(2.71(19) epoch 11711) v5 ==== 1795+0+0 (3283463911 0 0) 0xf2432c0 con 0x1395ec60 -3590> 2014-09-18 18:08:32.697111 7f4c9f251700 5 -- op tracker -- , seq: 86079, time: 2014-09-18 18:08:32.697053, event: header_read, op: pg_notify(2.71(19) epoch 11711) -3589> 2014-09-18 18:08:32.697119 7f4c9f251700 5 -- op tracker -- , seq: 86079, time: 2014-09-18 18:08:32.697054, event: throttled, op: pg_notify(2.71(19) epoch 11711) -3588> 2014-09-18 18:08:32.697125 7f4c9f251700 5 -- op tracker -- , seq: 86079, time: 2014-09-18 18:08:32.697089, event: all_read, op: pg_notify(2.71(19) epoch 11711) -3587> 2014-09-18 18:08:32.697131 7f4c9f251700 5 -- op tracker -- , seq: 86079, time: 2014-09-18 18:08:32.697108, event: dispatched, op: pg_notify(2.71(19) epoch 11711) -3586> 2014-09-18 18:08:32.697145 7f4c9f251700 5 -- op tracker -- , seq: 86079, time: 2014-09-18 18:08:32.697144, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11711) -3585> 2014-09-18 18:08:32.697153 7f4c9f251700 5 -- op tracker -- , seq: 86079, time: 2014-09-18 18:08:32.697151, event: started, op: pg_notify(2.71(19) epoch 11711) -3584> 2014-09-18 18:08:32.697170 7f4c9f251700 5 -- op tracker -- , seq: 86079, time: 2014-09-18 18:08:32.697168, event: done, op: pg_notify(2.71(19) epoch 11711) -3583> 2014-09-18 18:08:32.734862 7f4caae41700 5 osd.42 11711 tick -3582> 2014-09-18 18:08:32.738802 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=82 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).connect error 10.242.42.164:6854/39317, (111) Connection refused -3581> 2014-09-18 18:08:32.738820 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=82 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).fault (111) Connection refused -3580> 2014-09-18 18:08:32.743044 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.24 10.242.42.172:0/5612 113 ==== osd_ping(ping e11710 stamp 2014-09-18 23:37:21.846477) v2 ==== 47+0+0 (2893045231 0 0) 0x12c11fe0 con 0x110bb020 -3579> 2014-09-18 18:08:32.743077 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11711 stamp 2014-09-18 23:37:21.846477) v2 -- ?+0 0x21d15a20 con 0x110bb020 -3578> 2014-09-18 18:08:32.743098 7f4c9ca4c700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- osd_map(11711..11711 src has 1..11711) v3 -- ?+0 0x11bb7b00 con 0xdd8c940 -3577> 2014-09-18 18:08:32.744174 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.24 10.242.42.172:0/5612 113 ==== osd_ping(ping e11710 stamp 2014-09-18 23:37:21.846477) v2 ==== 47+0+0 (2893045231 0 0) 0x10415a20 con 0x10a5bb80 -3576> 2014-09-18 18:08:32.744209 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11711 stamp 2014-09-18 23:37:21.846477) v2 -- ?+0 0xf0e3e00 con 0x10a5bb80 -3575> 2014-09-18 18:08:32.758506 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.55 10.242.42.172:0/19404 1105 ==== osd_ping(ping e11711 stamp 2014-09-18 23:37:21.862803) v2 ==== 47+0+0 (969281153 0 0) 0xf2b4d00 con 0x4b7ab00 -3574> 2014-09-18 18:08:32.758505 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.55 10.242.42.172:0/19404 1105 ==== osd_ping(ping e11711 stamp 2014-09-18 23:37:21.862803) v2 ==== 47+0+0 (969281153 0 0) 0x1368cb40 con 0xeddec60 -3573> 2014-09-18 18:08:32.758536 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/19404 -- osd_ping(ping_reply e11711 stamp 2014-09-18 23:37:21.862803) v2 -- ?+0 0x12c11fe0 con 0x4b7ab00 -3572> 2014-09-18 18:08:32.758567 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/19404 -- osd_ping(ping_reply e11711 stamp 2014-09-18 23:37:21.862803) v2 -- ?+0 0x10415a20 con 0xeddec60 -3571> 2014-09-18 18:08:32.772984 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3220 ==== pg_notify(2.519(4) epoch 11711) v5 ==== 962+0+0 (3336034395 0 0) 0xf243680 con 0x1395ec60 -3570> 2014-09-18 18:08:32.773002 7f4c9f251700 5 -- op tracker -- , seq: 86080, time: 2014-09-18 18:08:32.772914, event: header_read, op: pg_notify(2.519(4) epoch 11711) -3569> 2014-09-18 18:08:32.773013 7f4c9f251700 5 -- op tracker -- , seq: 86080, time: 2014-09-18 18:08:32.772916, event: throttled, op: pg_notify(2.519(4) epoch 11711) -3568> 2014-09-18 18:08:32.773021 7f4c9f251700 5 -- op tracker -- , seq: 86080, time: 2014-09-18 18:08:32.772966, event: all_read, op: pg_notify(2.519(4) epoch 11711) -3567> 2014-09-18 18:08:32.773029 7f4c9f251700 5 -- op tracker -- , seq: 86080, time: 2014-09-18 18:08:32.772998, event: dispatched, op: pg_notify(2.519(4) epoch 11711) -3566> 2014-09-18 18:08:32.773036 7f4c9f251700 5 -- op tracker -- , seq: 86080, time: 2014-09-18 18:08:32.773034, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11711) -3565> 2014-09-18 18:08:32.773049 7f4c9f251700 5 -- op tracker -- , seq: 86080, time: 2014-09-18 18:08:32.773047, event: started, op: pg_notify(2.519(4) epoch 11711) -3564> 2014-09-18 18:08:32.773067 7f4c9f251700 5 -- op tracker -- , seq: 86080, time: 2014-09-18 18:08:32.773065, event: done, op: pg_notify(2.519(4) epoch 11711) -3563> 2014-09-18 18:08:32.776220 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3221 ==== pg_notify(2.410(12) epoch 11711) v5 ==== 1376+0+0 (3471971757 0 0) 0x853eb20 con 0x1395ec60 -3562> 2014-09-18 18:08:32.776235 7f4c9f251700 5 -- op tracker -- , seq: 86081, time: 2014-09-18 18:08:32.776174, event: header_read, op: pg_notify(2.410(12) epoch 11711) -3561> 2014-09-18 18:08:32.776243 7f4c9f251700 5 -- op tracker -- , seq: 86081, time: 2014-09-18 18:08:32.776175, event: throttled, op: pg_notify(2.410(12) epoch 11711) -3560> 2014-09-18 18:08:32.776249 7f4c9f251700 5 -- op tracker -- , seq: 86081, time: 2014-09-18 18:08:32.776208, event: all_read, op: pg_notify(2.410(12) epoch 11711) -3559> 2014-09-18 18:08:32.776255 7f4c9f251700 5 -- op tracker -- , seq: 86081, time: 2014-09-18 18:08:32.776231, event: dispatched, op: pg_notify(2.410(12) epoch 11711) -3558> 2014-09-18 18:08:32.776261 7f4c9f251700 5 -- op tracker -- , seq: 86081, time: 2014-09-18 18:08:32.776260, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11711) -3557> 2014-09-18 18:08:32.776269 7f4c9f251700 5 -- op tracker -- , seq: 86081, time: 2014-09-18 18:08:32.776267, event: started, op: pg_notify(2.410(12) epoch 11711) -3556> 2014-09-18 18:08:32.776370 7f4c9f251700 5 -- op tracker -- , seq: 86081, time: 2014-09-18 18:08:32.776367, event: done, op: pg_notify(2.410(12) epoch 11711) -3555> 2014-09-18 18:08:32.777573 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3222 ==== pg_notify(2.186(12) epoch 11711) v5 ==== 1386+0+0 (3235659098 0 0) 0x2b930d00 con 0x1395ec60 -3554> 2014-09-18 18:08:32.777586 7f4c9f251700 5 -- op tracker -- , seq: 86082, time: 2014-09-18 18:08:32.777519, event: header_read, op: pg_notify(2.186(12) epoch 11711) -3553> 2014-09-18 18:08:32.777593 7f4c9f251700 5 -- op tracker -- , seq: 86082, time: 2014-09-18 18:08:32.777521, event: throttled, op: pg_notify(2.186(12) epoch 11711) -3552> 2014-09-18 18:08:32.777599 7f4c9f251700 5 -- op tracker -- , seq: 86082, time: 2014-09-18 18:08:32.777559, event: all_read, op: pg_notify(2.186(12) epoch 11711) -3551> 2014-09-18 18:08:32.777605 7f4c9f251700 5 -- op tracker -- , seq: 86082, time: 2014-09-18 18:08:32.777583, event: dispatched, op: pg_notify(2.186(12) epoch 11711) -3550> 2014-09-18 18:08:32.777611 7f4c9f251700 5 -- op tracker -- , seq: 86082, time: 2014-09-18 18:08:32.777609, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11711) -3549> 2014-09-18 18:08:32.777618 7f4c9f251700 5 -- op tracker -- , seq: 86082, time: 2014-09-18 18:08:32.777617, event: started, op: pg_notify(2.186(12) epoch 11711) -3548> 2014-09-18 18:08:32.777638 7f4c9f251700 5 -- op tracker -- , seq: 86082, time: 2014-09-18 18:08:32.777636, event: done, op: pg_notify(2.186(12) epoch 11711) -3547> 2014-09-18 18:08:32.796544 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.46 10.242.42.178:0/4507 118 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:32.795910) v2 ==== 47+0+0 (1989212879 0 0) 0xf4ef4a0 con 0x10a5c3c0 -3546> 2014-09-18 18:08:32.796562 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.46 10.242.42.178:0/4507 118 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:32.795910) v2 ==== 47+0+0 (1989212879 0 0) 0xf2b3c20 con 0x110bb5a0 -3545> 2014-09-18 18:08:32.796586 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/4507 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:32.795910) v2 -- ?+0 0x1368cb40 con 0x10a5c3c0 -3544> 2014-09-18 18:08:32.796625 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/4507 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:32.795910) v2 -- ?+0 0xf2b4d00 con 0x110bb5a0 -3543> 2014-09-18 18:08:32.832678 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3223 ==== pg_notify(2.74e(11) epoch 11711) v5 ==== 1303+0+0 (1094054565 0 0) 0x1393c760 con 0x1395ec60 -3542> 2014-09-18 18:08:32.832696 7f4c9f251700 5 -- op tracker -- , seq: 86083, time: 2014-09-18 18:08:32.832607, event: header_read, op: pg_notify(2.74e(11) epoch 11711) -3541> 2014-09-18 18:08:32.832706 7f4c9f251700 5 -- op tracker -- , seq: 86083, time: 2014-09-18 18:08:32.832608, event: throttled, op: pg_notify(2.74e(11) epoch 11711) -3540> 2014-09-18 18:08:32.832715 7f4c9f251700 5 -- op tracker -- , seq: 86083, time: 2014-09-18 18:08:32.832664, event: all_read, op: pg_notify(2.74e(11) epoch 11711) -3539> 2014-09-18 18:08:32.832723 7f4c9f251700 5 -- op tracker -- , seq: 86083, time: 2014-09-18 18:08:32.832692, event: dispatched, op: pg_notify(2.74e(11) epoch 11711) -3538> 2014-09-18 18:08:32.832731 7f4c9f251700 5 -- op tracker -- , seq: 86083, time: 2014-09-18 18:08:32.832729, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11711) -3537> 2014-09-18 18:08:32.832740 7f4c9f251700 5 -- op tracker -- , seq: 86083, time: 2014-09-18 18:08:32.832738, event: started, op: pg_notify(2.74e(11) epoch 11711) -3536> 2014-09-18 18:08:32.832842 7f4c9f251700 5 -- op tracker -- , seq: 86083, time: 2014-09-18 18:08:32.832840, event: done, op: pg_notify(2.74e(11) epoch 11711) -3535> 2014-09-18 18:08:32.833519 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3224 ==== pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11711) v5 ==== 6481+0+0 (466462661 0 0) 0x1393be00 con 0x1395ec60 -3534> 2014-09-18 18:08:32.833533 7f4c9f251700 5 -- op tracker -- , seq: 86084, time: 2014-09-18 18:08:32.833398, event: header_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11711) -3533> 2014-09-18 18:08:32.833540 7f4c9f251700 5 -- op tracker -- , seq: 86084, time: 2014-09-18 18:08:32.833399, event: throttled, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11711) -3532> 2014-09-18 18:08:32.833547 7f4c9f251700 5 -- op tracker -- , seq: 86084, time: 2014-09-18 18:08:32.833495, event: all_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11711) -3531> 2014-09-18 18:08:32.833553 7f4c9f251700 5 -- op tracker -- , seq: 86084, time: 2014-09-18 18:08:32.833529, event: dispatched, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11711) -3530> 2014-09-18 18:08:32.833560 7f4c9f251700 5 -- op tracker -- , seq: 86084, time: 2014-09-18 18:08:32.833558, event: waiting_for_osdmap, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11711) -3529> 2014-09-18 18:08:32.833568 7f4c9f251700 5 -- op tracker -- , seq: 86084, time: 2014-09-18 18:08:32.833566, event: started, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11711) -3528> 2014-09-18 18:08:32.833598 7f4c9f251700 5 -- op tracker -- , seq: 86084, time: 2014-09-18 18:08:32.833596, event: done, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(53) epoch 11711) -3527> 2014-09-18 18:08:32.834179 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3225 ==== pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11711) v5 ==== 4008+0+0 (2999773299 0 0) 0x1393c580 con 0x1395ec60 -3526> 2014-09-18 18:08:32.834192 7f4c9f251700 5 -- op tracker -- , seq: 86085, time: 2014-09-18 18:08:32.834116, event: header_read, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11711) -3525> 2014-09-18 18:08:32.834200 7f4c9f251700 5 -- op tracker -- , seq: 86085, time: 2014-09-18 18:08:32.834117, event: throttled, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11711) -3524> 2014-09-18 18:08:32.834206 7f4c9f251700 5 -- op tracker -- , seq: 86085, time: 2014-09-18 18:08:32.834166, event: all_read, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11711) -3523> 2014-09-18 18:08:32.834213 7f4c9f251700 5 -- op tracker -- , seq: 86085, time: 2014-09-18 18:08:32.834189, event: dispatched, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11711) -3522> 2014-09-18 18:08:32.834219 7f4c9f251700 5 -- op tracker -- , seq: 86085, time: 2014-09-18 18:08:32.834217, event: waiting_for_osdmap, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11711) -3521> 2014-09-18 18:08:32.834227 7f4c9f251700 5 -- op tracker -- , seq: 86085, time: 2014-09-18 18:08:32.834225, event: started, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11711) -3520> 2014-09-18 18:08:32.834273 7f4c9f251700 5 -- op tracker -- , seq: 86085, time: 2014-09-18 18:08:32.834270, event: done, op: pg_notify(2.4c6(15),2.799(9),2.4d0(10) epoch 11711) -3519> 2014-09-18 18:08:32.896374 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3654 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x2fb9ed80 con 0xcbecec0 -3518> 2014-09-18 18:08:32.896393 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3517> 2014-09-18 18:08:32.896415 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3655 ==== pg_notify(2.4f6(12) epoch 11711) v5 ==== 1382+0+0 (771911102 0 0) 0xf974760 con 0xcbecec0 -3516> 2014-09-18 18:08:32.896423 7f4c9f251700 5 -- op tracker -- , seq: 86086, time: 2014-09-18 18:08:32.896372, event: header_read, op: pg_notify(2.4f6(12) epoch 11711) -3515> 2014-09-18 18:08:32.896437 7f4c9f251700 5 -- op tracker -- , seq: 86086, time: 2014-09-18 18:08:32.896373, event: throttled, op: pg_notify(2.4f6(12) epoch 11711) -3514> 2014-09-18 18:08:32.896445 7f4c9f251700 5 -- op tracker -- , seq: 86086, time: 2014-09-18 18:08:32.896398, event: all_read, op: pg_notify(2.4f6(12) epoch 11711) -3513> 2014-09-18 18:08:32.896453 7f4c9f251700 5 -- op tracker -- , seq: 86086, time: 2014-09-18 18:08:32.896420, event: dispatched, op: pg_notify(2.4f6(12) epoch 11711) -3512> 2014-09-18 18:08:32.896461 7f4c9f251700 5 -- op tracker -- , seq: 86086, time: 2014-09-18 18:08:32.896459, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11711) -3511> 2014-09-18 18:08:32.896479 7f4c9f251700 5 -- op tracker -- , seq: 86086, time: 2014-09-18 18:08:32.896477, event: started, op: pg_notify(2.4f6(12) epoch 11711) -3510> 2014-09-18 18:08:32.896497 7f4c9f251700 5 -- op tracker -- , seq: 86086, time: 2014-09-18 18:08:32.896496, event: done, op: pg_notify(2.4f6(12) epoch 11711) -3509> 2014-09-18 18:08:32.904312 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 542 ==== pg_info(1 pgs e11711:2.352) v4 ==== 738+0+0 (3644506356 0 0) 0xf1f1480 con 0x1395ef20 -3508> 2014-09-18 18:08:32.904327 7f4c9f251700 5 -- op tracker -- , seq: 86087, time: 2014-09-18 18:08:32.904268, event: header_read, op: pg_info(1 pgs e11711:2.352) -3507> 2014-09-18 18:08:32.904337 7f4c9f251700 5 -- op tracker -- , seq: 86087, time: 2014-09-18 18:08:32.904269, event: throttled, op: pg_info(1 pgs e11711:2.352) -3506> 2014-09-18 18:08:32.904343 7f4c9f251700 5 -- op tracker -- , seq: 86087, time: 2014-09-18 18:08:32.904297, event: all_read, op: pg_info(1 pgs e11711:2.352) -3505> 2014-09-18 18:08:32.904349 7f4c9f251700 5 -- op tracker -- , seq: 86087, time: 2014-09-18 18:08:32.904324, event: dispatched, op: pg_info(1 pgs e11711:2.352) -3504> 2014-09-18 18:08:32.904355 7f4c9f251700 5 -- op tracker -- , seq: 86087, time: 2014-09-18 18:08:32.904354, event: waiting_for_osdmap, op: pg_info(1 pgs e11711:2.352) -3503> 2014-09-18 18:08:32.904363 7f4c9f251700 5 -- op tracker -- , seq: 86087, time: 2014-09-18 18:08:32.904361, event: started, op: pg_info(1 pgs e11711:2.352) -3502> 2014-09-18 18:08:32.904380 7f4c9f251700 5 -- op tracker -- , seq: 86087, time: 2014-09-18 18:08:32.904378, event: done, op: pg_info(1 pgs e11711:2.352) -3501> 2014-09-18 18:08:32.904595 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3656 ==== pg_notify(2.3b6(10) epoch 11711) v5 ==== 1260+0+0 (1614144559 0 0) 0xf1e7fe0 con 0xcbecec0 -3500> 2014-09-18 18:08:32.904635 7f4c9f251700 5 -- op tracker -- , seq: 86088, time: 2014-09-18 18:08:32.904514, event: header_read, op: pg_notify(2.3b6(10) epoch 11711) -3499> 2014-09-18 18:08:32.904645 7f4c9f251700 5 -- op tracker -- , seq: 86088, time: 2014-09-18 18:08:32.904523, event: throttled, op: pg_notify(2.3b6(10) epoch 11711) -3498> 2014-09-18 18:08:32.904653 7f4c9f251700 5 -- op tracker -- , seq: 86088, time: 2014-09-18 18:08:32.904576, event: all_read, op: pg_notify(2.3b6(10) epoch 11711) -3497> 2014-09-18 18:08:32.904659 7f4c9f251700 5 -- op tracker -- , seq: 86088, time: 2014-09-18 18:08:32.904631, event: dispatched, op: pg_notify(2.3b6(10) epoch 11711) -3496> 2014-09-18 18:08:32.904666 7f4c9f251700 5 -- op tracker -- , seq: 86088, time: 2014-09-18 18:08:32.904664, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11711) -3495> 2014-09-18 18:08:32.904673 7f4c9f251700 5 -- op tracker -- , seq: 86088, time: 2014-09-18 18:08:32.904672, event: started, op: pg_notify(2.3b6(10) epoch 11711) -3494> 2014-09-18 18:08:32.904690 7f4c9f251700 5 -- op tracker -- , seq: 86088, time: 2014-09-18 18:08:32.904689, event: done, op: pg_notify(2.3b6(10) epoch 11711) -3493> 2014-09-18 18:08:32.907896 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3657 ==== pg_notify(2.517(11) epoch 11711) v5 ==== 1343+0+0 (2720920063 0 0) 0x1421ef00 con 0xcbecec0 -3492> 2014-09-18 18:08:32.907916 7f4c9f251700 5 -- op tracker -- , seq: 86089, time: 2014-09-18 18:08:32.907812, event: header_read, op: pg_notify(2.517(11) epoch 11711) -3491> 2014-09-18 18:08:32.907926 7f4c9f251700 5 -- op tracker -- , seq: 86089, time: 2014-09-18 18:08:32.907814, event: throttled, op: pg_notify(2.517(11) epoch 11711) -3490> 2014-09-18 18:08:32.907933 7f4c9f251700 5 -- op tracker -- , seq: 86089, time: 2014-09-18 18:08:32.907858, event: all_read, op: pg_notify(2.517(11) epoch 11711) -3489> 2014-09-18 18:08:32.907940 7f4c9f251700 5 -- op tracker -- , seq: 86089, time: 2014-09-18 18:08:32.907911, event: dispatched, op: pg_notify(2.517(11) epoch 11711) -3488> 2014-09-18 18:08:32.907947 7f4c9f251700 5 -- op tracker -- , seq: 86089, time: 2014-09-18 18:08:32.907945, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11711) -3487> 2014-09-18 18:08:32.907955 7f4c9f251700 5 -- op tracker -- , seq: 86089, time: 2014-09-18 18:08:32.907953, event: started, op: pg_notify(2.517(11) epoch 11711) -3486> 2014-09-18 18:08:32.908059 7f4c9f251700 5 -- op tracker -- , seq: 86089, time: 2014-09-18 18:08:32.908056, event: done, op: pg_notify(2.517(11) epoch 11711) -3485> 2014-09-18 18:08:32.910821 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3658 ==== pg_notify(2.2c7(139) epoch 11711) v5 ==== 8651+0+0 (2013665396 0 0) 0x1421fe00 con 0xcbecec0 -3484> 2014-09-18 18:08:32.910838 7f4c9f251700 5 -- op tracker -- , seq: 86090, time: 2014-09-18 18:08:32.910715, event: header_read, op: pg_notify(2.2c7(139) epoch 11711) -3483> 2014-09-18 18:08:32.910848 7f4c9f251700 5 -- op tracker -- , seq: 86090, time: 2014-09-18 18:08:32.910715, event: throttled, op: pg_notify(2.2c7(139) epoch 11711) -3482> 2014-09-18 18:08:32.910854 7f4c9f251700 5 -- op tracker -- , seq: 86090, time: 2014-09-18 18:08:32.910809, event: all_read, op: pg_notify(2.2c7(139) epoch 11711) -3481> 2014-09-18 18:08:32.910860 7f4c9f251700 5 -- op tracker -- , seq: 86090, time: 2014-09-18 18:08:32.910834, event: dispatched, op: pg_notify(2.2c7(139) epoch 11711) -3480> 2014-09-18 18:08:32.910866 7f4c9f251700 5 -- op tracker -- , seq: 86090, time: 2014-09-18 18:08:32.910865, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11711) -3479> 2014-09-18 18:08:32.910874 7f4c9f251700 5 -- op tracker -- , seq: 86090, time: 2014-09-18 18:08:32.910873, event: started, op: pg_notify(2.2c7(139) epoch 11711) -3478> 2014-09-18 18:08:32.910898 7f4c9f251700 5 -- op tracker -- , seq: 86090, time: 2014-09-18 18:08:32.910897, event: done, op: pg_notify(2.2c7(139) epoch 11711) -3477> 2014-09-18 18:08:32.912908 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3659 ==== pg_notify(2.40a(162) epoch 11711) v5 ==== 9736+0+0 (3050095015 0 0) 0x1421e3c0 con 0xcbecec0 -3476> 2014-09-18 18:08:32.912928 7f4c9f251700 5 -- op tracker -- , seq: 86091, time: 2014-09-18 18:08:32.912719, event: header_read, op: pg_notify(2.40a(162) epoch 11711) -3475> 2014-09-18 18:08:32.912939 7f4c9f251700 5 -- op tracker -- , seq: 86091, time: 2014-09-18 18:08:32.912720, event: throttled, op: pg_notify(2.40a(162) epoch 11711) -3474> 2014-09-18 18:08:32.912945 7f4c9f251700 5 -- op tracker -- , seq: 86091, time: 2014-09-18 18:08:32.912864, event: all_read, op: pg_notify(2.40a(162) epoch 11711) -3473> 2014-09-18 18:08:32.912950 7f4c9f251700 5 -- op tracker -- , seq: 86091, time: 2014-09-18 18:08:32.912924, event: dispatched, op: pg_notify(2.40a(162) epoch 11711) -3472> 2014-09-18 18:08:32.912954 7f4c9f251700 5 -- op tracker -- , seq: 86091, time: 2014-09-18 18:08:32.912953, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11711) -3471> 2014-09-18 18:08:32.912960 7f4c9f251700 5 -- op tracker -- , seq: 86091, time: 2014-09-18 18:08:32.912959, event: started, op: pg_notify(2.40a(162) epoch 11711) -3470> 2014-09-18 18:08:32.912978 7f4c9f251700 5 -- op tracker -- , seq: 86091, time: 2014-09-18 18:08:32.912976, event: done, op: pg_notify(2.40a(162) epoch 11711) -3469> 2014-09-18 18:08:32.914491 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3660 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11711) v5 ==== 6174+0+0 (3421183151 0 0) 0xf4eeb40 con 0xcbecec0 -3468> 2014-09-18 18:08:32.914509 7f4c9f251700 5 -- op tracker -- , seq: 86092, time: 2014-09-18 18:08:32.914408, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11711) -3467> 2014-09-18 18:08:32.914572 7f4c9f251700 5 -- op tracker -- , seq: 86092, time: 2014-09-18 18:08:32.914409, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11711) -3466> 2014-09-18 18:08:32.914579 7f4c9f251700 5 -- op tracker -- , seq: 86092, time: 2014-09-18 18:08:32.914479, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11711) -3465> 2014-09-18 18:08:32.914595 7f4c9f251700 5 -- op tracker -- , seq: 86092, time: 2014-09-18 18:08:32.914505, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11711) -3464> 2014-09-18 18:08:32.914603 7f4c9f251700 5 -- op tracker -- , seq: 86092, time: 2014-09-18 18:08:32.914601, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11711) -3463> 2014-09-18 18:08:32.914611 7f4c9f251700 5 -- op tracker -- , seq: 86092, time: 2014-09-18 18:08:32.914609, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11711) -3462> 2014-09-18 18:08:32.914651 7f4c9f251700 5 -- op tracker -- , seq: 86092, time: 2014-09-18 18:08:32.914649, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11711) -3461> 2014-09-18 18:08:32.915698 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.0 10.242.42.164:0/15878 161 ==== osd_ping(ping e11710 stamp 2014-09-18 18:08:32.915199) v2 ==== 47+0+0 (1845648414 0 0) 0xf576000 con 0x1d5bc7e0 -3460> 2014-09-18 18:08:32.915723 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:32.915199) v2 -- ?+0 0xf4ef4a0 con 0x1d5bc7e0 -3459> 2014-09-18 18:08:32.915732 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.0 10.242.42.164:0/15878 161 ==== osd_ping(ping e11710 stamp 2014-09-18 18:08:32.915199) v2 ==== 47+0+0 (1845648414 0 0) 0x2b9234a0 con 0x142e8260 -3458> 2014-09-18 18:08:32.915747 7f4c9b249700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- osd_map(11711..11711 src has 1..11711) v3 -- ?+0 0xe338ac0 con 0x10a878c0 -3457> 2014-09-18 18:08:32.915770 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:32.915199) v2 -- ?+0 0xf2b3c20 con 0x142e8260 -3456> 2014-09-18 18:08:32.927486 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 693 ==== pg_notify(2.4f6(15) epoch 11710) v5 ==== 1543+0+0 (1344465954 0 0) 0x21e123a0 con 0x1f103e40 -3455> 2014-09-18 18:08:32.927511 7f4c9f251700 5 -- op tracker -- , seq: 86093, time: 2014-09-18 18:08:32.927414, event: header_read, op: pg_notify(2.4f6(15) epoch 11710) -3454> 2014-09-18 18:08:32.927521 7f4c9f251700 5 -- op tracker -- , seq: 86093, time: 2014-09-18 18:08:32.927416, event: throttled, op: pg_notify(2.4f6(15) epoch 11710) -3453> 2014-09-18 18:08:32.927525 7f4c9f251700 5 -- op tracker -- , seq: 86093, time: 2014-09-18 18:08:32.927464, event: all_read, op: pg_notify(2.4f6(15) epoch 11710) -3452> 2014-09-18 18:08:32.927530 7f4c9f251700 5 -- op tracker -- , seq: 86093, time: 2014-09-18 18:08:32.927505, event: dispatched, op: pg_notify(2.4f6(15) epoch 11710) -3451> 2014-09-18 18:08:32.927534 7f4c9f251700 5 -- op tracker -- , seq: 86093, time: 2014-09-18 18:08:32.927533, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11710) -3450> 2014-09-18 18:08:32.927541 7f4c9f251700 5 -- op tracker -- , seq: 86093, time: 2014-09-18 18:08:32.927540, event: started, op: pg_notify(2.4f6(15) epoch 11710) -3449> 2014-09-18 18:08:32.927566 7f4c9f251700 5 -- op tracker -- , seq: 86093, time: 2014-09-18 18:08:32.927565, event: done, op: pg_notify(2.4f6(15) epoch 11710) -3448> 2014-09-18 18:08:32.929499 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 694 ==== pg_notify(2.799(10),2.4c6(15) epoch 11710) v5 ==== 2779+0+0 (887277735 0 0) 0x21e13480 con 0x1f103e40 -3447> 2014-09-18 18:08:32.929515 7f4c9f251700 5 -- op tracker -- , seq: 86094, time: 2014-09-18 18:08:32.929426, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11710) -3446> 2014-09-18 18:08:32.929522 7f4c9f251700 5 -- op tracker -- , seq: 86094, time: 2014-09-18 18:08:32.929427, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11710) -3445> 2014-09-18 18:08:32.929526 7f4c9f251700 5 -- op tracker -- , seq: 86094, time: 2014-09-18 18:08:32.929480, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11710) -3444> 2014-09-18 18:08:32.929530 7f4c9f251700 5 -- op tracker -- , seq: 86094, time: 2014-09-18 18:08:32.929511, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11710) -3443> 2014-09-18 18:08:32.929535 7f4c9f251700 5 -- op tracker -- , seq: 86094, time: 2014-09-18 18:08:32.929533, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11710) -3442> 2014-09-18 18:08:32.929541 7f4c9f251700 5 -- op tracker -- , seq: 86094, time: 2014-09-18 18:08:32.929539, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11710) -3441> 2014-09-18 18:08:32.929575 7f4c9f251700 5 -- op tracker -- , seq: 86094, time: 2014-09-18 18:08:32.929573, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11710) -3440> 2014-09-18 18:08:32.931450 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5089 ==== pg_notify(2.11b(8) epoch 11708) v5 ==== 1204+0+0 (2119951940 0 0) 0xf2b8ee0 con 0x1088aec0 -3439> 2014-09-18 18:08:32.931470 7f4c9f251700 5 -- op tracker -- , seq: 86095, time: 2014-09-18 18:08:32.931389, event: header_read, op: pg_notify(2.11b(8) epoch 11708) -3438> 2014-09-18 18:08:32.931478 7f4c9f251700 5 -- op tracker -- , seq: 86095, time: 2014-09-18 18:08:32.931390, event: throttled, op: pg_notify(2.11b(8) epoch 11708) -3437> 2014-09-18 18:08:32.931485 7f4c9f251700 5 -- op tracker -- , seq: 86095, time: 2014-09-18 18:08:32.931432, event: all_read, op: pg_notify(2.11b(8) epoch 11708) -3436> 2014-09-18 18:08:32.931492 7f4c9f251700 5 -- op tracker -- , seq: 86095, time: 2014-09-18 18:08:32.931465, event: dispatched, op: pg_notify(2.11b(8) epoch 11708) -3435> 2014-09-18 18:08:32.931499 7f4c9f251700 5 -- op tracker -- , seq: 86095, time: 2014-09-18 18:08:32.931497, event: waiting_for_osdmap, op: pg_notify(2.11b(8) epoch 11708) -3434> 2014-09-18 18:08:32.931507 7f4c9f251700 5 -- op tracker -- , seq: 86095, time: 2014-09-18 18:08:32.931505, event: started, op: pg_notify(2.11b(8) epoch 11708) -3433> 2014-09-18 18:08:32.931526 7f4c9f251700 5 -- op tracker -- , seq: 86095, time: 2014-09-18 18:08:32.931524, event: done, op: pg_notify(2.11b(8) epoch 11708) -3432> 2014-09-18 18:08:32.931880 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5090 ==== pg_notify(2.120(11),2.40a(162) epoch 11708) v5 ==== 11039+0+0 (257474890 0 0) 0x2b927e00 con 0x1088aec0 -3431> 2014-09-18 18:08:32.931896 7f4c9f251700 5 -- op tracker -- , seq: 86096, time: 2014-09-18 18:08:32.931676, event: header_read, op: pg_notify(2.120(11),2.40a(162) epoch 11708) -3430> 2014-09-18 18:08:32.931905 7f4c9f251700 5 -- op tracker -- , seq: 86096, time: 2014-09-18 18:08:32.931677, event: throttled, op: pg_notify(2.120(11),2.40a(162) epoch 11708) -3429> 2014-09-18 18:08:32.931912 7f4c9f251700 5 -- op tracker -- , seq: 86096, time: 2014-09-18 18:08:32.931866, event: all_read, op: pg_notify(2.120(11),2.40a(162) epoch 11708) -3428> 2014-09-18 18:08:32.931918 7f4c9f251700 5 -- op tracker -- , seq: 86096, time: 2014-09-18 18:08:32.931892, event: dispatched, op: pg_notify(2.120(11),2.40a(162) epoch 11708) -3427> 2014-09-18 18:08:32.931925 7f4c9f251700 5 -- op tracker -- , seq: 86096, time: 2014-09-18 18:08:32.931923, event: waiting_for_osdmap, op: pg_notify(2.120(11),2.40a(162) epoch 11708) -3426> 2014-09-18 18:08:32.931933 7f4c9f251700 5 -- op tracker -- , seq: 86096, time: 2014-09-18 18:08:32.931931, event: started, op: pg_notify(2.120(11),2.40a(162) epoch 11708) -3425> 2014-09-18 18:08:32.931957 7f4c9f251700 5 -- op tracker -- , seq: 86096, time: 2014-09-18 18:08:32.931956, event: done, op: pg_notify(2.120(11),2.40a(162) epoch 11708) -3424> 2014-09-18 18:08:32.934315 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5091 ==== pg_notify(2.410(12) epoch 11708) v5 ==== 1338+0+0 (403705036 0 0) 0xf866f00 con 0x1088aec0 -3423> 2014-09-18 18:08:32.934335 7f4c9f251700 5 -- op tracker -- , seq: 86097, time: 2014-09-18 18:08:32.934236, event: header_read, op: pg_notify(2.410(12) epoch 11708) -3422> 2014-09-18 18:08:32.934346 7f4c9f251700 5 -- op tracker -- , seq: 86097, time: 2014-09-18 18:08:32.934237, event: throttled, op: pg_notify(2.410(12) epoch 11708) -3421> 2014-09-18 18:08:32.934353 7f4c9f251700 5 -- op tracker -- , seq: 86097, time: 2014-09-18 18:08:32.934273, event: all_read, op: pg_notify(2.410(12) epoch 11708) -3420> 2014-09-18 18:08:32.934359 7f4c9f251700 5 -- op tracker -- , seq: 86097, time: 2014-09-18 18:08:32.934331, event: dispatched, op: pg_notify(2.410(12) epoch 11708) -3419> 2014-09-18 18:08:32.934365 7f4c9f251700 5 -- op tracker -- , seq: 86097, time: 2014-09-18 18:08:32.934364, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11708) -3418> 2014-09-18 18:08:32.934373 7f4c9f251700 5 -- op tracker -- , seq: 86097, time: 2014-09-18 18:08:32.934372, event: started, op: pg_notify(2.410(12) epoch 11708) -3417> 2014-09-18 18:08:32.934392 7f4c9f251700 5 -- op tracker -- , seq: 86097, time: 2014-09-18 18:08:32.934390, event: done, op: pg_notify(2.410(12) epoch 11708) -3416> 2014-09-18 18:08:32.935065 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5092 ==== pg_notify(2.179(10) epoch 11708) v5 ==== 1276+0+0 (2572760808 0 0) 0xf868d00 con 0x1088aec0 -3415> 2014-09-18 18:08:32.935085 7f4c9f251700 5 -- op tracker -- , seq: 86098, time: 2014-09-18 18:08:32.934968, event: header_read, op: pg_notify(2.179(10) epoch 11708) -3414> 2014-09-18 18:08:32.935095 7f4c9f251700 5 -- op tracker -- , seq: 86098, time: 2014-09-18 18:08:32.934969, event: throttled, op: pg_notify(2.179(10) epoch 11708) -3413> 2014-09-18 18:08:32.935101 7f4c9f251700 5 -- op tracker -- , seq: 86098, time: 2014-09-18 18:08:32.935029, event: all_read, op: pg_notify(2.179(10) epoch 11708) -3412> 2014-09-18 18:08:32.935107 7f4c9f251700 5 -- op tracker -- , seq: 86098, time: 2014-09-18 18:08:32.935080, event: dispatched, op: pg_notify(2.179(10) epoch 11708) -3411> 2014-09-18 18:08:32.935114 7f4c9f251700 5 -- op tracker -- , seq: 86098, time: 2014-09-18 18:08:32.935112, event: waiting_for_osdmap, op: pg_notify(2.179(10) epoch 11708) -3410> 2014-09-18 18:08:32.935122 7f4c9f251700 5 -- op tracker -- , seq: 86098, time: 2014-09-18 18:08:32.935120, event: started, op: pg_notify(2.179(10) epoch 11708) -3409> 2014-09-18 18:08:32.935140 7f4c9f251700 5 -- op tracker -- , seq: 86098, time: 2014-09-18 18:08:32.935139, event: done, op: pg_notify(2.179(10) epoch 11708) -3408> 2014-09-18 18:08:32.937868 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5093 ==== pg_notify(2.1a2(20) epoch 11708) v5 ==== 1798+0+0 (2804368626 0 0) 0xf729480 con 0x1088aec0 -3407> 2014-09-18 18:08:32.937886 7f4c9f251700 5 -- op tracker -- , seq: 86099, time: 2014-09-18 18:08:32.937812, event: header_read, op: pg_notify(2.1a2(20) epoch 11708) -3406> 2014-09-18 18:08:32.937896 7f4c9f251700 5 -- op tracker -- , seq: 86099, time: 2014-09-18 18:08:32.937813, event: throttled, op: pg_notify(2.1a2(20) epoch 11708) -3405> 2014-09-18 18:08:32.937902 7f4c9f251700 5 -- op tracker -- , seq: 86099, time: 2014-09-18 18:08:32.937856, event: all_read, op: pg_notify(2.1a2(20) epoch 11708) -3404> 2014-09-18 18:08:32.937908 7f4c9f251700 5 -- op tracker -- , seq: 86099, time: 2014-09-18 18:08:32.937882, event: dispatched, op: pg_notify(2.1a2(20) epoch 11708) -3403> 2014-09-18 18:08:32.937915 7f4c9f251700 5 -- op tracker -- , seq: 86099, time: 2014-09-18 18:08:32.937913, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11708) -3402> 2014-09-18 18:08:32.937923 7f4c9f251700 5 -- op tracker -- , seq: 86099, time: 2014-09-18 18:08:32.937921, event: started, op: pg_notify(2.1a2(20) epoch 11708) -3401> 2014-09-18 18:08:32.937941 7f4c9f251700 5 -- op tracker -- , seq: 86099, time: 2014-09-18 18:08:32.937940, event: done, op: pg_notify(2.1a2(20) epoch 11708) -3400> 2014-09-18 18:08:32.940297 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5094 ==== pg_notify(2.4c1(15) epoch 11708) v5 ==== 1507+0+0 (1261370541 0 0) 0xf7274a0 con 0x1088aec0 -3399> 2014-09-18 18:08:32.940316 7f4c9f251700 5 -- op tracker -- , seq: 86100, time: 2014-09-18 18:08:32.940252, event: header_read, op: pg_notify(2.4c1(15) epoch 11708) -3398> 2014-09-18 18:08:32.940328 7f4c9f251700 5 -- op tracker -- , seq: 86100, time: 2014-09-18 18:08:32.940253, event: throttled, op: pg_notify(2.4c1(15) epoch 11708) -3397> 2014-09-18 18:08:32.940337 7f4c9f251700 5 -- op tracker -- , seq: 86100, time: 2014-09-18 18:08:32.940284, event: all_read, op: pg_notify(2.4c1(15) epoch 11708) -3396> 2014-09-18 18:08:32.940346 7f4c9f251700 5 -- op tracker -- , seq: 86100, time: 2014-09-18 18:08:32.940311, event: dispatched, op: pg_notify(2.4c1(15) epoch 11708) -3395> 2014-09-18 18:08:32.940356 7f4c9f251700 5 -- op tracker -- , seq: 86100, time: 2014-09-18 18:08:32.940354, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11708) -3394> 2014-09-18 18:08:32.940369 7f4c9f251700 5 -- op tracker -- , seq: 86100, time: 2014-09-18 18:08:32.940366, event: started, op: pg_notify(2.4c1(15) epoch 11708) -3393> 2014-09-18 18:08:32.940394 7f4c9f251700 5 -- op tracker -- , seq: 86100, time: 2014-09-18 18:08:32.940391, event: done, op: pg_notify(2.4c1(15) epoch 11708) -3392> 2014-09-18 18:08:32.946755 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 343 ==== pg_notify(2.719(17) epoch 11710) v5 ==== 1601+0+0 (3687736434 0 0) 0x117392c0 con 0x12562260 -3391> 2014-09-18 18:08:32.946771 7f4c9f251700 5 -- op tracker -- , seq: 86101, time: 2014-09-18 18:08:32.946697, event: header_read, op: pg_notify(2.719(17) epoch 11710) -3390> 2014-09-18 18:08:32.946806 7f4c9f251700 5 -- op tracker -- , seq: 86101, time: 2014-09-18 18:08:32.946699, event: throttled, op: pg_notify(2.719(17) epoch 11710) -3389> 2014-09-18 18:08:32.946814 7f4c9f251700 5 -- op tracker -- , seq: 86101, time: 2014-09-18 18:08:32.946739, event: all_read, op: pg_notify(2.719(17) epoch 11710) -3388> 2014-09-18 18:08:32.946820 7f4c9f251700 5 -- op tracker -- , seq: 86101, time: 2014-09-18 18:08:32.946768, event: dispatched, op: pg_notify(2.719(17) epoch 11710) -3387> 2014-09-18 18:08:32.946828 7f4c9f251700 5 -- op tracker -- , seq: 86101, time: 2014-09-18 18:08:32.946826, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11710) -3386> 2014-09-18 18:08:32.946836 7f4c9f251700 5 -- op tracker -- , seq: 86101, time: 2014-09-18 18:08:32.946834, event: started, op: pg_notify(2.719(17) epoch 11710) -3385> 2014-09-18 18:08:32.946853 7f4c9f251700 5 -- op tracker -- , seq: 86101, time: 2014-09-18 18:08:32.946852, event: done, op: pg_notify(2.719(17) epoch 11710) -3384> 2014-09-18 18:08:32.946887 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 695 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0xcc03cc0 con 0x1f103e40 -3383> 2014-09-18 18:08:32.946901 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3382> 2014-09-18 18:08:32.946976 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 696 ==== pg_notify(2.4f6(15) epoch 11711) v5 ==== 1543+0+0 (139215828 0 0) 0x21e105a0 con 0x1f103e40 -3381> 2014-09-18 18:08:32.946996 7f4c9f251700 5 -- op tracker -- , seq: 86102, time: 2014-09-18 18:08:32.946899, event: header_read, op: pg_notify(2.4f6(15) epoch 11711) -3380> 2014-09-18 18:08:32.947007 7f4c9f251700 5 -- op tracker -- , seq: 86102, time: 2014-09-18 18:08:32.946900, event: throttled, op: pg_notify(2.4f6(15) epoch 11711) -3379> 2014-09-18 18:08:32.947013 7f4c9f251700 5 -- op tracker -- , seq: 86102, time: 2014-09-18 18:08:32.946942, event: all_read, op: pg_notify(2.4f6(15) epoch 11711) -3378> 2014-09-18 18:08:32.947019 7f4c9f251700 5 -- op tracker -- , seq: 86102, time: 2014-09-18 18:08:32.946992, event: dispatched, op: pg_notify(2.4f6(15) epoch 11711) -3377> 2014-09-18 18:08:32.947025 7f4c9f251700 5 -- op tracker -- , seq: 86102, time: 2014-09-18 18:08:32.947024, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11711) -3376> 2014-09-18 18:08:32.947033 7f4c9f251700 5 -- op tracker -- , seq: 86102, time: 2014-09-18 18:08:32.947032, event: started, op: pg_notify(2.4f6(15) epoch 11711) -3375> 2014-09-18 18:08:32.947051 7f4c9f251700 5 -- op tracker -- , seq: 86102, time: 2014-09-18 18:08:32.947050, event: done, op: pg_notify(2.4f6(15) epoch 11711) -3374> 2014-09-18 18:08:32.949017 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 697 ==== pg_notify(2.799(10),2.4c6(15) epoch 11711) v5 ==== 2779+0+0 (3280313943 0 0) 0x21e10d20 con 0x1f103e40 -3373> 2014-09-18 18:08:32.949033 7f4c9f251700 5 -- op tracker -- , seq: 86103, time: 2014-09-18 18:08:32.948965, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11711) -3372> 2014-09-18 18:08:32.949042 7f4c9f251700 5 -- op tracker -- , seq: 86103, time: 2014-09-18 18:08:32.948966, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11711) -3371> 2014-09-18 18:08:32.949048 7f4c9f251700 5 -- op tracker -- , seq: 86103, time: 2014-09-18 18:08:32.949006, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11711) -3370> 2014-09-18 18:08:32.949054 7f4c9f251700 5 -- op tracker -- , seq: 86103, time: 2014-09-18 18:08:32.949029, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11711) -3369> 2014-09-18 18:08:32.949061 7f4c9f251700 5 -- op tracker -- , seq: 86103, time: 2014-09-18 18:08:32.949059, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11711) -3368> 2014-09-18 18:08:32.949069 7f4c9f251700 5 -- op tracker -- , seq: 86103, time: 2014-09-18 18:08:32.949067, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11711) -3367> 2014-09-18 18:08:32.949093 7f4c9f251700 5 -- op tracker -- , seq: 86103, time: 2014-09-18 18:08:32.949092, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11711) -3366> 2014-09-18 18:08:32.962389 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 344 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0xef593c0 con 0x12562260 -3365> 2014-09-18 18:08:32.962406 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3364> 2014-09-18 18:08:32.962495 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 345 ==== pg_notify(2.719(17) epoch 11711) v5 ==== 1601+0+0 (904309781 0 0) 0x11739860 con 0x12562260 -3363> 2014-09-18 18:08:32.962514 7f4c9f251700 5 -- op tracker -- , seq: 86104, time: 2014-09-18 18:08:32.962387, event: header_read, op: pg_notify(2.719(17) epoch 11711) -3362> 2014-09-18 18:08:32.962525 7f4c9f251700 5 -- op tracker -- , seq: 86104, time: 2014-09-18 18:08:32.962387, event: throttled, op: pg_notify(2.719(17) epoch 11711) -3361> 2014-09-18 18:08:32.962532 7f4c9f251700 5 -- op tracker -- , seq: 86104, time: 2014-09-18 18:08:32.962453, event: all_read, op: pg_notify(2.719(17) epoch 11711) -3360> 2014-09-18 18:08:32.962554 7f4c9f251700 5 -- op tracker -- , seq: 86104, time: 2014-09-18 18:08:32.962510, event: dispatched, op: pg_notify(2.719(17) epoch 11711) -3359> 2014-09-18 18:08:32.962562 7f4c9f251700 5 -- op tracker -- , seq: 86104, time: 2014-09-18 18:08:32.962560, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11711) -3358> 2014-09-18 18:08:32.962570 7f4c9f251700 5 -- op tracker -- , seq: 86104, time: 2014-09-18 18:08:32.962568, event: started, op: pg_notify(2.719(17) epoch 11711) -3357> 2014-09-18 18:08:32.962694 7f4c9f251700 5 -- op tracker -- , seq: 86104, time: 2014-09-18 18:08:32.962692, event: done, op: pg_notify(2.719(17) epoch 11711) -3356> 2014-09-18 18:08:32.975273 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1857 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x1182c1c0 con 0x10f60520 -3355> 2014-09-18 18:08:32.975292 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3354> 2014-09-18 18:08:32.975302 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1858 ==== pg_notify(2.4f6(15) epoch 11711) v5 ==== 1581+0+0 (1370286919 0 0) 0xf12a960 con 0x10f60520 -3353> 2014-09-18 18:08:32.975312 7f4c9f251700 5 -- op tracker -- , seq: 86105, time: 2014-09-18 18:08:32.975198, event: header_read, op: pg_notify(2.4f6(15) epoch 11711) -3352> 2014-09-18 18:08:32.975322 7f4c9f251700 5 -- op tracker -- , seq: 86105, time: 2014-09-18 18:08:32.975198, event: throttled, op: pg_notify(2.4f6(15) epoch 11711) -3351> 2014-09-18 18:08:32.975344 7f4c9f251700 5 -- op tracker -- , seq: 86105, time: 2014-09-18 18:08:32.975258, event: all_read, op: pg_notify(2.4f6(15) epoch 11711) -3350> 2014-09-18 18:08:32.975351 7f4c9f251700 5 -- op tracker -- , seq: 86105, time: 2014-09-18 18:08:32.975309, event: dispatched, op: pg_notify(2.4f6(15) epoch 11711) -3349> 2014-09-18 18:08:32.975359 7f4c9f251700 5 -- op tracker -- , seq: 86105, time: 2014-09-18 18:08:32.975357, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11711) -3348> 2014-09-18 18:08:32.975367 7f4c9f251700 5 -- op tracker -- , seq: 86105, time: 2014-09-18 18:08:32.975365, event: started, op: pg_notify(2.4f6(15) epoch 11711) -3347> 2014-09-18 18:08:32.975386 7f4c9f251700 5 -- op tracker -- , seq: 86105, time: 2014-09-18 18:08:32.975384, event: done, op: pg_notify(2.4f6(15) epoch 11711) -3346> 2014-09-18 18:08:33.000300 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.11 10.242.42.168:0/17298 129 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:32.866945) v2 ==== 47+0+0 (3329344963 0 0) 0xf09da20 con 0x101cd180 -3345> 2014-09-18 18:08:33.000310 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.11 10.242.42.168:0/17298 129 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:32.866945) v2 ==== 47+0+0 (3329344963 0 0) 0xe76cd20 con 0xcc49e40 -3344> 2014-09-18 18:08:33.000326 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:32.866945) v2 -- ?+0 0xf576000 con 0x101cd180 -3343> 2014-09-18 18:08:33.000347 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:32.866945) v2 -- ?+0 0x2b9234a0 con 0xcc49e40 -3342> 2014-09-18 18:08:33.087603 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1730 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x14409200 con 0x4900000 -3341> 2014-09-18 18:08:33.087627 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11711,11711], i have 11711, src has [1,11711] -3340> 2014-09-18 18:08:33.087751 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1731 ==== pg_notify(2.e0(79) epoch 11711) v5 ==== 5119+0+0 (918716385 0 0) 0xf1ebfe0 con 0x4900000 -3339> 2014-09-18 18:08:33.087764 7f4c9f251700 5 -- op tracker -- , seq: 86106, time: 2014-09-18 18:08:33.087591, event: header_read, op: pg_notify(2.e0(79) epoch 11711) -3338> 2014-09-18 18:08:33.087775 7f4c9f251700 5 -- op tracker -- , seq: 86106, time: 2014-09-18 18:08:33.087592, event: throttled, op: pg_notify(2.e0(79) epoch 11711) -3337> 2014-09-18 18:08:33.087781 7f4c9f251700 5 -- op tracker -- , seq: 86106, time: 2014-09-18 18:08:33.087729, event: all_read, op: pg_notify(2.e0(79) epoch 11711) -3336> 2014-09-18 18:08:33.087787 7f4c9f251700 5 -- op tracker -- , seq: 86106, time: 2014-09-18 18:08:33.087760, event: dispatched, op: pg_notify(2.e0(79) epoch 11711) -3335> 2014-09-18 18:08:33.087794 7f4c9f251700 5 -- op tracker -- , seq: 86106, time: 2014-09-18 18:08:33.087792, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11711) -3334> 2014-09-18 18:08:33.087802 7f4c9f251700 5 -- op tracker -- , seq: 86106, time: 2014-09-18 18:08:33.087801, event: started, op: pg_notify(2.e0(79) epoch 11711) -3333> 2014-09-18 18:08:33.087824 7f4c9f251700 5 -- op tracker -- , seq: 86106, time: 2014-09-18 18:08:33.087822, event: done, op: pg_notify(2.e0(79) epoch 11711) -3332> 2014-09-18 18:08:33.088179 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1859 ==== pg_notify(2.517(11) epoch 11711) v5 ==== 1335+0+0 (936320983 0 0) 0xf12d0c0 con 0x10f60520 -3331> 2014-09-18 18:08:33.088196 7f4c9f251700 5 -- op tracker -- , seq: 86107, time: 2014-09-18 18:08:33.088126, event: header_read, op: pg_notify(2.517(11) epoch 11711) -3330> 2014-09-18 18:08:33.088206 7f4c9f251700 5 -- op tracker -- , seq: 86107, time: 2014-09-18 18:08:33.088127, event: throttled, op: pg_notify(2.517(11) epoch 11711) -3329> 2014-09-18 18:08:33.088212 7f4c9f251700 5 -- op tracker -- , seq: 86107, time: 2014-09-18 18:08:33.088165, event: all_read, op: pg_notify(2.517(11) epoch 11711) -3328> 2014-09-18 18:08:33.088218 7f4c9f251700 5 -- op tracker -- , seq: 86107, time: 2014-09-18 18:08:33.088192, event: dispatched, op: pg_notify(2.517(11) epoch 11711) -3327> 2014-09-18 18:08:33.088224 7f4c9f251700 5 -- op tracker -- , seq: 86107, time: 2014-09-18 18:08:33.088223, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11711) -3326> 2014-09-18 18:08:33.088232 7f4c9f251700 5 -- op tracker -- , seq: 86107, time: 2014-09-18 18:08:33.088230, event: started, op: pg_notify(2.517(11) epoch 11711) -3325> 2014-09-18 18:08:33.088255 7f4c9f251700 5 -- op tracker -- , seq: 86107, time: 2014-09-18 18:08:33.088252, event: done, op: pg_notify(2.517(11) epoch 11711) -3324> 2014-09-18 18:08:33.088543 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2360 ==== pg_notify(2.457(21) epoch 11711) v5 ==== 1875+0+0 (1245682839 0 0) 0xf2b8760 con 0xdd8c940 -3323> 2014-09-18 18:08:33.088556 7f4c9f251700 5 -- op tracker -- , seq: 86108, time: 2014-09-18 18:08:33.088487, event: header_read, op: pg_notify(2.457(21) epoch 11711) -3322> 2014-09-18 18:08:33.088563 7f4c9f251700 5 -- op tracker -- , seq: 86108, time: 2014-09-18 18:08:33.088488, event: throttled, op: pg_notify(2.457(21) epoch 11711) -3321> 2014-09-18 18:08:33.088570 7f4c9f251700 5 -- op tracker -- , seq: 86108, time: 2014-09-18 18:08:33.088531, event: all_read, op: pg_notify(2.457(21) epoch 11711) -3320> 2014-09-18 18:08:33.088575 7f4c9f251700 5 -- op tracker -- , seq: 86108, time: 2014-09-18 18:08:33.088553, event: dispatched, op: pg_notify(2.457(21) epoch 11711) -3319> 2014-09-18 18:08:33.088582 7f4c9f251700 5 -- op tracker -- , seq: 86108, time: 2014-09-18 18:08:33.088581, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11711) -3318> 2014-09-18 18:08:33.088589 7f4c9f251700 5 -- op tracker -- , seq: 86108, time: 2014-09-18 18:08:33.088588, event: started, op: pg_notify(2.457(21) epoch 11711) -3317> 2014-09-18 18:08:33.088607 7f4c9f251700 5 -- op tracker -- , seq: 86108, time: 2014-09-18 18:08:33.088606, event: done, op: pg_notify(2.457(21) epoch 11711) -3316> 2014-09-18 18:08:33.089051 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1860 ==== pg_notify(2.1d8(9) epoch 11711) v5 ==== 1261+0+0 (470150545 0 0) 0xf12ab40 con 0x10f60520 -3315> 2014-09-18 18:08:33.089072 7f4c9f251700 5 -- op tracker -- , seq: 86109, time: 2014-09-18 18:08:33.088971, event: header_read, op: pg_notify(2.1d8(9) epoch 11711) -3314> 2014-09-18 18:08:33.089083 7f4c9f251700 5 -- op tracker -- , seq: 86109, time: 2014-09-18 18:08:33.088972, event: throttled, op: pg_notify(2.1d8(9) epoch 11711) -3313> 2014-09-18 18:08:33.089090 7f4c9f251700 5 -- op tracker -- , seq: 86109, time: 2014-09-18 18:08:33.089009, event: all_read, op: pg_notify(2.1d8(9) epoch 11711) -3312> 2014-09-18 18:08:33.089102 7f4c9f251700 5 -- op tracker -- , seq: 86109, time: 2014-09-18 18:08:33.089067, event: dispatched, op: pg_notify(2.1d8(9) epoch 11711) -3311> 2014-09-18 18:08:33.089109 7f4c9f251700 5 -- op tracker -- , seq: 86109, time: 2014-09-18 18:08:33.089107, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11711) -3310> 2014-09-18 18:08:33.089117 7f4c9f251700 5 -- op tracker -- , seq: 86109, time: 2014-09-18 18:08:33.089115, event: started, op: pg_notify(2.1d8(9) epoch 11711) -3309> 2014-09-18 18:08:33.089136 7f4c9f251700 5 -- op tracker -- , seq: 86109, time: 2014-09-18 18:08:33.089134, event: done, op: pg_notify(2.1d8(9) epoch 11711) -3308> 2014-09-18 18:08:33.089461 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2361 ==== pg_notify(2.2c7(139) epoch 11711) v5 ==== 8651+0+0 (3115444127 0 0) 0xf098760 con 0xdd8c940 -3307> 2014-09-18 18:08:33.089475 7f4c9f251700 5 -- op tracker -- , seq: 86110, time: 2014-09-18 18:08:33.089219, event: header_read, op: pg_notify(2.2c7(139) epoch 11711) -3306> 2014-09-18 18:08:33.089483 7f4c9f251700 5 -- op tracker -- , seq: 86110, time: 2014-09-18 18:08:33.089220, event: throttled, op: pg_notify(2.2c7(139) epoch 11711) -3305> 2014-09-18 18:08:33.089489 7f4c9f251700 5 -- op tracker -- , seq: 86110, time: 2014-09-18 18:08:33.089438, event: all_read, op: pg_notify(2.2c7(139) epoch 11711) -3304> 2014-09-18 18:08:33.089495 7f4c9f251700 5 -- op tracker -- , seq: 86110, time: 2014-09-18 18:08:33.089471, event: dispatched, op: pg_notify(2.2c7(139) epoch 11711) -3303> 2014-09-18 18:08:33.089501 7f4c9f251700 5 -- op tracker -- , seq: 86110, time: 2014-09-18 18:08:33.089499, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11711) -3302> 2014-09-18 18:08:33.089508 7f4c9f251700 5 -- op tracker -- , seq: 86110, time: 2014-09-18 18:08:33.089507, event: started, op: pg_notify(2.2c7(139) epoch 11711) -3301> 2014-09-18 18:08:33.089526 7f4c9f251700 5 -- op tracker -- , seq: 86110, time: 2014-09-18 18:08:33.089524, event: done, op: pg_notify(2.2c7(139) epoch 11711) -3300> 2014-09-18 18:08:33.174112 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 970 ==== pg_notify(2.517(11) epoch 11711) v5 ==== 1335+0+0 (2106271476 0 0) 0x24fd1c20 con 0x10a878c0 -3299> 2014-09-18 18:08:33.174145 7f4c9f251700 5 -- op tracker -- , seq: 86111, time: 2014-09-18 18:08:33.174000, event: header_read, op: pg_notify(2.517(11) epoch 11711) -3298> 2014-09-18 18:08:33.174158 7f4c9f251700 5 -- op tracker -- , seq: 86111, time: 2014-09-18 18:08:33.174003, event: throttled, op: pg_notify(2.517(11) epoch 11711) -3297> 2014-09-18 18:08:33.174164 7f4c9f251700 5 -- op tracker -- , seq: 86111, time: 2014-09-18 18:08:33.174065, event: all_read, op: pg_notify(2.517(11) epoch 11711) -3296> 2014-09-18 18:08:33.174171 7f4c9f251700 5 -- op tracker -- , seq: 86111, time: 2014-09-18 18:08:33.174137, event: dispatched, op: pg_notify(2.517(11) epoch 11711) -3295> 2014-09-18 18:08:33.174177 7f4c9f251700 5 -- op tracker -- , seq: 86111, time: 2014-09-18 18:08:33.174176, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11711) -3294> 2014-09-18 18:08:33.174187 7f4c9f251700 5 -- op tracker -- , seq: 86111, time: 2014-09-18 18:08:33.174186, event: started, op: pg_notify(2.517(11) epoch 11711) -3293> 2014-09-18 18:08:33.174210 7f4c9f251700 5 -- op tracker -- , seq: 86111, time: 2014-09-18 18:08:33.174209, event: done, op: pg_notify(2.517(11) epoch 11711) -3292> 2014-09-18 18:08:33.174673 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 971 ==== pg_notify(2.4c6(15) epoch 11711) v5 ==== 1511+0+0 (2286882846 0 0) 0x16251660 con 0x10a878c0 -3291> 2014-09-18 18:08:33.174693 7f4c9f251700 5 -- op tracker -- , seq: 86112, time: 2014-09-18 18:08:33.174559, event: header_read, op: pg_notify(2.4c6(15) epoch 11711) -3290> 2014-09-18 18:08:33.174703 7f4c9f251700 5 -- op tracker -- , seq: 86112, time: 2014-09-18 18:08:33.174560, event: throttled, op: pg_notify(2.4c6(15) epoch 11711) -3289> 2014-09-18 18:08:33.174710 7f4c9f251700 5 -- op tracker -- , seq: 86112, time: 2014-09-18 18:08:33.174629, event: all_read, op: pg_notify(2.4c6(15) epoch 11711) -3288> 2014-09-18 18:08:33.174716 7f4c9f251700 5 -- op tracker -- , seq: 86112, time: 2014-09-18 18:08:33.174689, event: dispatched, op: pg_notify(2.4c6(15) epoch 11711) -3287> 2014-09-18 18:08:33.174722 7f4c9f251700 5 -- op tracker -- , seq: 86112, time: 2014-09-18 18:08:33.174721, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11711) -3286> 2014-09-18 18:08:33.174730 7f4c9f251700 5 -- op tracker -- , seq: 86112, time: 2014-09-18 18:08:33.174729, event: started, op: pg_notify(2.4c6(15) epoch 11711) -3285> 2014-09-18 18:08:33.174750 7f4c9f251700 5 -- op tracker -- , seq: 86112, time: 2014-09-18 18:08:33.174748, event: done, op: pg_notify(2.4c6(15) epoch 11711) -3284> 2014-09-18 18:08:33.210364 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1282 ==== pg_notify(2.4f6(15) epoch 11710) v5 ==== 1543+0+0 (2756257012 0 0) 0x11b0ffe0 con 0x1f102840 -3283> 2014-09-18 18:08:33.210383 7f4c9f251700 5 -- op tracker -- , seq: 86113, time: 2014-09-18 18:08:33.210310, event: header_read, op: pg_notify(2.4f6(15) epoch 11710) -3282> 2014-09-18 18:08:33.210394 7f4c9f251700 5 -- op tracker -- , seq: 86113, time: 2014-09-18 18:08:33.210311, event: throttled, op: pg_notify(2.4f6(15) epoch 11710) -3281> 2014-09-18 18:08:33.210400 7f4c9f251700 5 -- op tracker -- , seq: 86113, time: 2014-09-18 18:08:33.210351, event: all_read, op: pg_notify(2.4f6(15) epoch 11710) -3280> 2014-09-18 18:08:33.210406 7f4c9f251700 5 -- op tracker -- , seq: 86113, time: 2014-09-18 18:08:33.210379, event: dispatched, op: pg_notify(2.4f6(15) epoch 11710) -3279> 2014-09-18 18:08:33.210413 7f4c9f251700 5 -- op tracker -- , seq: 86113, time: 2014-09-18 18:08:33.210411, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11710) -3278> 2014-09-18 18:08:33.210421 7f4c9f251700 5 -- op tracker -- , seq: 86113, time: 2014-09-18 18:08:33.210419, event: started, op: pg_notify(2.4f6(15) epoch 11710) -3277> 2014-09-18 18:08:33.210439 7f4c9f251700 5 -- op tracker -- , seq: 86113, time: 2014-09-18 18:08:33.210437, event: done, op: pg_notify(2.4f6(15) epoch 11710) -3276> 2014-09-18 18:08:33.291853 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.13 10.242.42.168:0/10879 694 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:33.158289) v2 ==== 47+0+0 (685935898 0 0) 0x21ced2a0 con 0x140f7b80 -3275> 2014-09-18 18:08:33.291904 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:33.158289) v2 -- ?+0 0xf09da20 con 0x140f7b80 -3274> 2014-09-18 18:08:33.292052 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.13 10.242.42.168:0/10879 694 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:33.158289) v2 ==== 47+0+0 (685935898 0 0) 0xe288d00 con 0x109f9de0 -3273> 2014-09-18 18:08:33.292077 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:33.158289) v2 -- ?+0 0xe76cd20 con 0x109f9de0 -3272> 2014-09-18 18:08:33.449540 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.9 10.242.42.166:0/24959 111 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:33.451801) v2 ==== 47+0+0 (2384570730 0 0) 0x146df4a0 con 0x10a5c520 -3271> 2014-09-18 18:08:33.449565 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:33.451801) v2 -- ?+0 0x21ced2a0 con 0x10a5c520 -3270> 2014-09-18 18:08:33.449568 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.9 10.242.42.166:0/24959 111 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:33.451801) v2 ==== 47+0+0 (2384570730 0 0) 0xedfd2a0 con 0x110bb9c0 -3269> 2014-09-18 18:08:33.449592 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:33.451801) v2 -- ?+0 0xe288d00 con 0x110bb9c0 -3268> 2014-09-18 18:08:33.498067 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5095 ==== pg_notify(2.4e8(1) epoch 11709) v5 ==== 805+0+0 (2700409639 0 0) 0xf729840 con 0x1088aec0 -3267> 2014-09-18 18:08:33.498093 7f4c9f251700 5 -- op tracker -- , seq: 86114, time: 2014-09-18 18:08:33.497937, event: header_read, op: pg_notify(2.4e8(1) epoch 11709) -3266> 2014-09-18 18:08:33.498105 7f4c9f251700 5 -- op tracker -- , seq: 86114, time: 2014-09-18 18:08:33.497939, event: throttled, op: pg_notify(2.4e8(1) epoch 11709) -3265> 2014-09-18 18:08:33.498121 7f4c9f251700 5 -- op tracker -- , seq: 86114, time: 2014-09-18 18:08:33.498001, event: all_read, op: pg_notify(2.4e8(1) epoch 11709) -3264> 2014-09-18 18:08:33.498127 7f4c9f251700 5 -- op tracker -- , seq: 86114, time: 2014-09-18 18:08:33.498086, event: dispatched, op: pg_notify(2.4e8(1) epoch 11709) -3263> 2014-09-18 18:08:33.498134 7f4c9f251700 5 -- op tracker -- , seq: 86114, time: 2014-09-18 18:08:33.498133, event: waiting_for_osdmap, op: pg_notify(2.4e8(1) epoch 11709) -3262> 2014-09-18 18:08:33.498142 7f4c9f251700 5 -- op tracker -- , seq: 86114, time: 2014-09-18 18:08:33.498140, event: started, op: pg_notify(2.4e8(1) epoch 11709) -3261> 2014-09-18 18:08:33.498160 7f4c9f251700 5 -- op tracker -- , seq: 86114, time: 2014-09-18 18:08:33.498158, event: done, op: pg_notify(2.4e8(1) epoch 11709) -3260> 2014-09-18 18:08:33.499684 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5096 ==== pg_notify(2.62(44) epoch 11709) v5 ==== 3220+0+0 (3379373182 0 0) 0xf7265a0 con 0x1088aec0 -3259> 2014-09-18 18:08:33.499704 7f4c9f251700 5 -- op tracker -- , seq: 86115, time: 2014-09-18 18:08:33.499575, event: header_read, op: pg_notify(2.62(44) epoch 11709) -3258> 2014-09-18 18:08:33.499715 7f4c9f251700 5 -- op tracker -- , seq: 86115, time: 2014-09-18 18:08:33.499575, event: throttled, op: pg_notify(2.62(44) epoch 11709) -3257> 2014-09-18 18:08:33.499721 7f4c9f251700 5 -- op tracker -- , seq: 86115, time: 2014-09-18 18:08:33.499662, event: all_read, op: pg_notify(2.62(44) epoch 11709) -3256> 2014-09-18 18:08:33.499727 7f4c9f251700 5 -- op tracker -- , seq: 86115, time: 2014-09-18 18:08:33.499700, event: dispatched, op: pg_notify(2.62(44) epoch 11709) -3255> 2014-09-18 18:08:33.499734 7f4c9f251700 5 -- op tracker -- , seq: 86115, time: 2014-09-18 18:08:33.499733, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11709) -3254> 2014-09-18 18:08:33.499742 7f4c9f251700 5 -- op tracker -- , seq: 86115, time: 2014-09-18 18:08:33.499741, event: started, op: pg_notify(2.62(44) epoch 11709) -3253> 2014-09-18 18:08:33.499764 7f4c9f251700 5 -- op tracker -- , seq: 86115, time: 2014-09-18 18:08:33.499762, event: done, op: pg_notify(2.62(44) epoch 11709) -3252> 2014-09-18 18:08:33.501361 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.11 10.242.42.168:0/17298 130 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:33.368114) v2 ==== 47+0+0 (3218834889 0 0) 0xf09d480 con 0x101cd180 -3251> 2014-09-18 18:08:33.501375 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.11 10.242.42.168:0/17298 130 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:33.368114) v2 ==== 47+0+0 (3218834889 0 0) 0xe76dfe0 con 0xcc49e40 -3250> 2014-09-18 18:08:33.501386 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:33.368114) v2 -- ?+0 0x146df4a0 con 0x101cd180 -3249> 2014-09-18 18:08:33.501415 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/17298 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:33.368114) v2 -- ?+0 0xedfd2a0 con 0xcc49e40 -3248> 2014-09-18 18:08:33.501705 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5097 ==== pg_notify(2.3b6(10) epoch 11709) v5 ==== 1260+0+0 (1018607350 0 0) 0xf728940 con 0x1088aec0 -3247> 2014-09-18 18:08:33.501755 7f4c9f251700 5 -- op tracker -- , seq: 86116, time: 2014-09-18 18:08:33.501663, event: header_read, op: pg_notify(2.3b6(10) epoch 11709) -3246> 2014-09-18 18:08:33.501766 7f4c9f251700 5 -- op tracker -- , seq: 86116, time: 2014-09-18 18:08:33.501663, event: throttled, op: pg_notify(2.3b6(10) epoch 11709) -3245> 2014-09-18 18:08:33.501772 7f4c9f251700 5 -- op tracker -- , seq: 86116, time: 2014-09-18 18:08:33.501694, event: all_read, op: pg_notify(2.3b6(10) epoch 11709) -3244> 2014-09-18 18:08:33.501778 7f4c9f251700 5 -- op tracker -- , seq: 86116, time: 2014-09-18 18:08:33.501751, event: dispatched, op: pg_notify(2.3b6(10) epoch 11709) -3243> 2014-09-18 18:08:33.501785 7f4c9f251700 5 -- op tracker -- , seq: 86116, time: 2014-09-18 18:08:33.501783, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11709) -3242> 2014-09-18 18:08:33.501793 7f4c9f251700 5 -- op tracker -- , seq: 86116, time: 2014-09-18 18:08:33.501791, event: started, op: pg_notify(2.3b6(10) epoch 11709) -3241> 2014-09-18 18:08:33.501812 7f4c9f251700 5 -- op tracker -- , seq: 86116, time: 2014-09-18 18:08:33.501811, event: done, op: pg_notify(2.3b6(10) epoch 11709) -3240> 2014-09-18 18:08:33.503262 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5098 ==== pg_notify(2.6f8(10) epoch 11709) v5 ==== 1276+0+0 (1994933053 0 0) 0xf7261e0 con 0x1088aec0 -3239> 2014-09-18 18:08:33.503281 7f4c9f251700 5 -- op tracker -- , seq: 86117, time: 2014-09-18 18:08:33.503220, event: header_read, op: pg_notify(2.6f8(10) epoch 11709) -3238> 2014-09-18 18:08:33.503292 7f4c9f251700 5 -- op tracker -- , seq: 86117, time: 2014-09-18 18:08:33.503221, event: throttled, op: pg_notify(2.6f8(10) epoch 11709) -3237> 2014-09-18 18:08:33.503302 7f4c9f251700 5 -- op tracker -- , seq: 86117, time: 2014-09-18 18:08:33.503250, event: all_read, op: pg_notify(2.6f8(10) epoch 11709) -3236> 2014-09-18 18:08:33.503312 7f4c9f251700 5 -- op tracker -- , seq: 86117, time: 2014-09-18 18:08:33.503277, event: dispatched, op: pg_notify(2.6f8(10) epoch 11709) -3235> 2014-09-18 18:08:33.503321 7f4c9f251700 5 -- op tracker -- , seq: 86117, time: 2014-09-18 18:08:33.503320, event: waiting_for_osdmap, op: pg_notify(2.6f8(10) epoch 11709) -3234> 2014-09-18 18:08:33.503330 7f4c9f251700 5 -- op tracker -- , seq: 86117, time: 2014-09-18 18:08:33.503328, event: started, op: pg_notify(2.6f8(10) epoch 11709) -3233> 2014-09-18 18:08:33.503349 7f4c9f251700 5 -- op tracker -- , seq: 86117, time: 2014-09-18 18:08:33.503347, event: done, op: pg_notify(2.6f8(10) epoch 11709) -3232> 2014-09-18 18:08:33.503691 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5099 ==== pg_notify(2.517(11),2.519(23) epoch 11709) v5 ==== 3274+0+0 (1583564838 0 0) 0xf729c00 con 0x1088aec0 -3231> 2014-09-18 18:08:33.503708 7f4c9f251700 5 -- op tracker -- , seq: 86118, time: 2014-09-18 18:08:33.503627, event: header_read, op: pg_notify(2.517(11),2.519(23) epoch 11709) -3230> 2014-09-18 18:08:33.503719 7f4c9f251700 5 -- op tracker -- , seq: 86118, time: 2014-09-18 18:08:33.503628, event: throttled, op: pg_notify(2.517(11),2.519(23) epoch 11709) -3229> 2014-09-18 18:08:33.503725 7f4c9f251700 5 -- op tracker -- , seq: 86118, time: 2014-09-18 18:08:33.503678, event: all_read, op: pg_notify(2.517(11),2.519(23) epoch 11709) -3228> 2014-09-18 18:08:33.503731 7f4c9f251700 5 -- op tracker -- , seq: 86118, time: 2014-09-18 18:08:33.503704, event: dispatched, op: pg_notify(2.517(11),2.519(23) epoch 11709) -3227> 2014-09-18 18:08:33.503738 7f4c9f251700 5 -- op tracker -- , seq: 86118, time: 2014-09-18 18:08:33.503736, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.519(23) epoch 11709) -3226> 2014-09-18 18:08:33.503746 7f4c9f251700 5 -- op tracker -- , seq: 86118, time: 2014-09-18 18:08:33.503744, event: started, op: pg_notify(2.517(11),2.519(23) epoch 11709) -3225> 2014-09-18 18:08:33.503781 7f4c9f251700 5 -- op tracker -- , seq: 86118, time: 2014-09-18 18:08:33.503780, event: done, op: pg_notify(2.517(11),2.519(23) epoch 11709) -3224> 2014-09-18 18:08:33.540728 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.37 10.242.42.168:0/18319 105 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:33.407286) v2 ==== 47+0+0 (222396804 0 0) 0x1173a760 con 0x11ffdce0 -3223> 2014-09-18 18:08:33.540777 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:33.407286) v2 -- ?+0 0xf09d480 con 0x11ffdce0 -3222> 2014-09-18 18:08:33.540811 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.37 10.242.42.168:0/18319 105 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:33.407286) v2 ==== 47+0+0 (222396804 0 0) 0xf09eb40 con 0x110bb860 -3221> 2014-09-18 18:08:33.540841 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11711 stamp 2014-09-18 18:08:33.407286) v2 -- ?+0 0xe76dfe0 con 0x110bb860 -3220> 2014-09-18 18:08:33.560607 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3661 ==== pg_notify(2.1bb(9) epoch 11711) v5 ==== 1189+0+0 (3883847278 0 0) 0xf4f1a20 con 0xcbecec0 -3219> 2014-09-18 18:08:33.560628 7f4c9f251700 5 -- op tracker -- , seq: 86119, time: 2014-09-18 18:08:33.560467, event: header_read, op: pg_notify(2.1bb(9) epoch 11711) -3218> 2014-09-18 18:08:33.560637 7f4c9f251700 5 -- op tracker -- , seq: 86119, time: 2014-09-18 18:08:33.560468, event: throttled, op: pg_notify(2.1bb(9) epoch 11711) -3217> 2014-09-18 18:08:33.560642 7f4c9f251700 5 -- op tracker -- , seq: 86119, time: 2014-09-18 18:08:33.560507, event: all_read, op: pg_notify(2.1bb(9) epoch 11711) -3216> 2014-09-18 18:08:33.560646 7f4c9f251700 5 -- op tracker -- , seq: 86119, time: 2014-09-18 18:08:33.560623, event: dispatched, op: pg_notify(2.1bb(9) epoch 11711) -3215> 2014-09-18 18:08:33.560650 7f4c9f251700 5 -- op tracker -- , seq: 86119, time: 2014-09-18 18:08:33.560649, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11711) -3214> 2014-09-18 18:08:33.560665 7f4c9f251700 5 -- op tracker -- , seq: 86119, time: 2014-09-18 18:08:33.560663, event: started, op: pg_notify(2.1bb(9) epoch 11711) -3213> 2014-09-18 18:08:33.560688 7f4c9f251700 5 -- op tracker -- , seq: 86119, time: 2014-09-18 18:08:33.560686, event: done, op: pg_notify(2.1bb(9) epoch 11711) -3212> 2014-09-18 18:08:33.561761 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3662 ==== pg_notify(2.78f(56) epoch 11711) v5 ==== 3722+0+0 (3379149212 0 0) 0xf4ee960 con 0xcbecec0 -3211> 2014-09-18 18:08:33.561785 7f4c9f251700 5 -- op tracker -- , seq: 86120, time: 2014-09-18 18:08:33.561632, event: header_read, op: pg_notify(2.78f(56) epoch 11711) -3210> 2014-09-18 18:08:33.561800 7f4c9f251700 5 -- op tracker -- , seq: 86120, time: 2014-09-18 18:08:33.561634, event: throttled, op: pg_notify(2.78f(56) epoch 11711) -3209> 2014-09-18 18:08:33.561808 7f4c9f251700 5 -- op tracker -- , seq: 86120, time: 2014-09-18 18:08:33.561722, event: all_read, op: pg_notify(2.78f(56) epoch 11711) -3208> 2014-09-18 18:08:33.561817 7f4c9f251700 5 -- op tracker -- , seq: 86120, time: 2014-09-18 18:08:33.561780, event: dispatched, op: pg_notify(2.78f(56) epoch 11711) -3207> 2014-09-18 18:08:33.561827 7f4c9f251700 5 -- op tracker -- , seq: 86120, time: 2014-09-18 18:08:33.561825, event: waiting_for_osdmap, op: pg_notify(2.78f(56) epoch 11711) -3206> 2014-09-18 18:08:33.561835 7f4c9f251700 5 -- op tracker -- , seq: 86120, time: 2014-09-18 18:08:33.561833, event: started, op: pg_notify(2.78f(56) epoch 11711) -3205> 2014-09-18 18:08:33.561854 7f4c9f251700 5 -- op tracker -- , seq: 86120, time: 2014-09-18 18:08:33.561852, event: done, op: pg_notify(2.78f(56) epoch 11711) -3204> 2014-09-18 18:08:33.686373 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3306 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x10d16d80 con 0x4900580 -3203> 2014-09-18 18:08:33.686408 7f4c9f251700 3 osd.42 11711 handle_osd_map epochs [11712,11712], i have 11711, src has [1,11712] -3202> 2014-09-18 18:08:33.689566 7f4c9f251700 1 -- 10.242.42.178:0/2483 mark_down 0x1d7c6580 -- 0x14b4e580 -3201> 2014-09-18 18:08:33.689623 7f4c9f251700 1 -- 10.242.42.178:0/2483 mark_down 0x1d7c9440 -- 0x14b51c80 -3200> 2014-09-18 18:08:33.689629 7f4c885e8700 2 -- 10.242.42.178:0/2483 >> 10.242.42.178:6842/4508 pipe(0x14b4e580 sd=102 :59159 s=4 pgs=71 cs=1 l=1 c=0x1d7c6580).reader couldn't read tag, (0) Success -3199> 2014-09-18 18:08:33.689662 7f4c885e8700 2 -- 10.242.42.178:0/2483 >> 10.242.42.178:6842/4508 pipe(0x14b4e580 sd=102 :59159 s=4 pgs=71 cs=1 l=1 c=0x1d7c6580).fault (0) Success -3198> 2014-09-18 18:08:33.689669 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2362 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x11bb7b00 con 0xdd8c940 -3197> 2014-09-18 18:08:33.689683 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3196> 2014-09-18 18:08:33.689690 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3307 ==== pg_notify(2.4f6(184) epoch 11712) v5 ==== 11296+0+0 (322578732 0 0) 0x2b92f4a0 con 0x4900580 -3195> 2014-09-18 18:08:33.689704 7f4c9f251700 5 -- op tracker -- , seq: 86121, time: 2014-09-18 18:08:33.686568, event: header_read, op: pg_notify(2.4f6(184) epoch 11712) -3194> 2014-09-18 18:08:33.689717 7f4c9f251700 5 -- op tracker -- , seq: 86121, time: 2014-09-18 18:08:33.686569, event: throttled, op: pg_notify(2.4f6(184) epoch 11712) -3193> 2014-09-18 18:08:33.689724 7f4c9f251700 5 -- op tracker -- , seq: 86121, time: 2014-09-18 18:08:33.686810, event: all_read, op: pg_notify(2.4f6(184) epoch 11712) -3192> 2014-09-18 18:08:33.689732 7f4c9f251700 5 -- op tracker -- , seq: 86121, time: 2014-09-18 18:08:33.689699, event: dispatched, op: pg_notify(2.4f6(184) epoch 11712) -3191> 2014-09-18 18:08:33.689740 7f4c9f251700 5 -- op tracker -- , seq: 86121, time: 2014-09-18 18:08:33.689738, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11712) -3190> 2014-09-18 18:08:33.689749 7f4c9f251700 5 -- op tracker -- , seq: 86121, time: 2014-09-18 18:08:33.689747, event: started, op: pg_notify(2.4f6(184) epoch 11712) -3189> 2014-09-18 18:08:33.689784 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11712..11712 src has 1..11712) v3 -- ?+0 0x19df3180 con 0x1395f1e0 -3188> 2014-09-18 18:08:33.689820 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11712) v5 -- ?+0 0x141823c0 con 0x1395f1e0 -3187> 2014-09-18 18:08:33.689834 7f4c9f251700 5 -- op tracker -- , seq: 86121, time: 2014-09-18 18:08:33.689831, event: done, op: pg_notify(2.4f6(184) epoch 11712) -3186> 2014-09-18 18:08:33.689873 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1046 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0xcc02ac0 con 0x14358260 -3185> 2014-09-18 18:08:33.689889 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3184> 2014-09-18 18:08:33.689898 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 49 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x19df0fc0 con 0x1395f340 -3183> 2014-09-18 18:08:33.689909 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3182> 2014-09-18 18:08:33.689914 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 543 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0xeee9f80 con 0x1395ef20 -3181> 2014-09-18 18:08:33.689926 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3180> 2014-09-18 18:08:33.689934 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2363 ==== pg_notify(2.457(21) epoch 11712) v5 ==== 1875+0+0 (1227056799 0 0) 0xf097fe0 con 0xdd8c940 -3179> 2014-09-18 18:08:33.689945 7f4c9f251700 5 -- op tracker -- , seq: 86122, time: 2014-09-18 18:08:33.686477, event: header_read, op: pg_notify(2.457(21) epoch 11712) -3178> 2014-09-18 18:08:33.689952 7f4c9f251700 5 -- op tracker -- , seq: 86122, time: 2014-09-18 18:08:33.686478, event: throttled, op: pg_notify(2.457(21) epoch 11712) -3177> 2014-09-18 18:08:33.689962 7f4c9f251700 5 -- op tracker -- , seq: 86122, time: 2014-09-18 18:08:33.686546, event: all_read, op: pg_notify(2.457(21) epoch 11712) -3176> 2014-09-18 18:08:33.689969 7f4c9f251700 5 -- op tracker -- , seq: 86122, time: 2014-09-18 18:08:33.689942, event: dispatched, op: pg_notify(2.457(21) epoch 11712) -3175> 2014-09-18 18:08:33.689973 7f4c9f251700 5 -- op tracker -- , seq: 86122, time: 2014-09-18 18:08:33.689972, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11712) -3174> 2014-09-18 18:08:33.689980 7f4c9f251700 5 -- op tracker -- , seq: 86122, time: 2014-09-18 18:08:33.689977, event: started, op: pg_notify(2.457(21) epoch 11712) -3173> 2014-09-18 18:08:33.689999 7f4c9f251700 5 -- op tracker -- , seq: 86122, time: 2014-09-18 18:08:33.689996, event: done, op: pg_notify(2.457(21) epoch 11712) -3172> 2014-09-18 18:08:33.690012 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3308 ==== pg_notify(2.17e(17) epoch 11712) v5 ==== 1593+0+0 (2424305676 0 0) 0x2b9310c0 con 0x4900580 -3171> 2014-09-18 18:08:33.690029 7f4c9f251700 5 -- op tracker -- , seq: 86123, time: 2014-09-18 18:08:33.688507, event: header_read, op: pg_notify(2.17e(17) epoch 11712) -3170> 2014-09-18 18:08:33.690040 7f4c9f251700 5 -- op tracker -- , seq: 86123, time: 2014-09-18 18:08:33.688508, event: throttled, op: pg_notify(2.17e(17) epoch 11712) -3169> 2014-09-18 18:08:33.690051 7f4c9f251700 5 -- op tracker -- , seq: 86123, time: 2014-09-18 18:08:33.688546, event: all_read, op: pg_notify(2.17e(17) epoch 11712) -3168> 2014-09-18 18:08:33.690061 7f4c9f251700 5 -- op tracker -- , seq: 86123, time: 2014-09-18 18:08:33.690024, event: dispatched, op: pg_notify(2.17e(17) epoch 11712) -3167> 2014-09-18 18:08:33.690071 7f4c9f251700 5 -- op tracker -- , seq: 86123, time: 2014-09-18 18:08:33.690068, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11712) -3166> 2014-09-18 18:08:33.690083 7f4c9f251700 5 -- op tracker -- , seq: 86123, time: 2014-09-18 18:08:33.690080, event: started, op: pg_notify(2.17e(17) epoch 11712) -3165> 2014-09-18 18:08:33.690105 7f4c9f251700 5 -- op tracker -- , seq: 86123, time: 2014-09-18 18:08:33.690102, event: done, op: pg_notify(2.17e(17) epoch 11712) -3164> 2014-09-18 18:08:33.690120 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1047 ==== pg_notify(2.719(17) epoch 11712) v5 ==== 1601+0+0 (2886285998 0 0) 0xf72e000 con 0x14358260 -3163> 2014-09-18 18:08:33.690137 7f4c9f251700 5 -- op tracker -- , seq: 86124, time: 2014-09-18 18:08:33.688388, event: header_read, op: pg_notify(2.719(17) epoch 11712) -3162> 2014-09-18 18:08:33.690149 7f4c9f251700 5 -- op tracker -- , seq: 86124, time: 2014-09-18 18:08:33.688389, event: throttled, op: pg_notify(2.719(17) epoch 11712) -3161> 2014-09-18 18:08:33.690158 7f4c9f251700 5 -- op tracker -- , seq: 86124, time: 2014-09-18 18:08:33.688435, event: all_read, op: pg_notify(2.719(17) epoch 11712) -3160> 2014-09-18 18:08:33.690169 7f4c9f251700 5 -- op tracker -- , seq: 86124, time: 2014-09-18 18:08:33.690133, event: dispatched, op: pg_notify(2.719(17) epoch 11712) -3159> 2014-09-18 18:08:33.690182 7f4c9f251700 5 -- op tracker -- , seq: 86124, time: 2014-09-18 18:08:33.690179, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11712) -3158> 2014-09-18 18:08:33.690196 7f4c9f251700 5 -- op tracker -- , seq: 86124, time: 2014-09-18 18:08:33.690193, event: started, op: pg_notify(2.719(17) epoch 11712) -3157> 2014-09-18 18:08:33.690225 7f4c9f251700 5 -- op tracker -- , seq: 86124, time: 2014-09-18 18:08:33.690223, event: done, op: pg_notify(2.719(17) epoch 11712) -3156> 2014-09-18 18:08:33.690240 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 50 ==== pg_notify(2.1d8(9) epoch 11712) v5 ==== 1261+0+0 (4285252479 0 0) 0xf243a40 con 0x1395f340 -3155> 2014-09-18 18:08:33.690254 7f4c9f251700 5 -- op tracker -- , seq: 86125, time: 2014-09-18 18:08:33.688169, event: header_read, op: pg_notify(2.1d8(9) epoch 11712) -3154> 2014-09-18 18:08:33.690268 7f4c9f251700 5 -- op tracker -- , seq: 86125, time: 2014-09-18 18:08:33.688170, event: throttled, op: pg_notify(2.1d8(9) epoch 11712) -3153> 2014-09-18 18:08:33.690274 7f4c9f251700 5 -- op tracker -- , seq: 86125, time: 2014-09-18 18:08:33.688196, event: all_read, op: pg_notify(2.1d8(9) epoch 11712) -3152> 2014-09-18 18:08:33.690279 7f4c9f251700 5 -- op tracker -- , seq: 86125, time: 2014-09-18 18:08:33.690251, event: dispatched, op: pg_notify(2.1d8(9) epoch 11712) -3151> 2014-09-18 18:08:33.690285 7f4c9f251700 5 -- op tracker -- , seq: 86125, time: 2014-09-18 18:08:33.690283, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11712) -3150> 2014-09-18 18:08:33.690292 7f4c9f251700 5 -- op tracker -- , seq: 86125, time: 2014-09-18 18:08:33.690290, event: started, op: pg_notify(2.1d8(9) epoch 11712) -3149> 2014-09-18 18:08:33.690307 7f4c9f251700 5 -- op tracker -- , seq: 86125, time: 2014-09-18 18:08:33.690305, event: done, op: pg_notify(2.1d8(9) epoch 11712) -3148> 2014-09-18 18:08:33.690317 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 544 ==== pg_notify(2.4d0(10) epoch 11712) v5 ==== 1292+0+0 (1109490407 0 0) 0xe1adde0 con 0x1395ef20 -3147> 2014-09-18 18:08:33.690329 7f4c9f251700 5 -- op tracker -- , seq: 86126, time: 2014-09-18 18:08:33.688335, event: header_read, op: pg_notify(2.4d0(10) epoch 11712) -3146> 2014-09-18 18:08:33.690336 7f4c9f251700 5 -- op tracker -- , seq: 86126, time: 2014-09-18 18:08:33.688336, event: throttled, op: pg_notify(2.4d0(10) epoch 11712) -3145> 2014-09-18 18:08:33.690343 7f4c9f251700 5 -- op tracker -- , seq: 86126, time: 2014-09-18 18:08:33.688371, event: all_read, op: pg_notify(2.4d0(10) epoch 11712) -3144> 2014-09-18 18:08:33.690349 7f4c9f251700 5 -- op tracker -- , seq: 86126, time: 2014-09-18 18:08:33.690325, event: dispatched, op: pg_notify(2.4d0(10) epoch 11712) -3143> 2014-09-18 18:08:33.690357 7f4c9f251700 5 -- op tracker -- , seq: 86126, time: 2014-09-18 18:08:33.690354, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11712) -3142> 2014-09-18 18:08:33.690370 7f4c9f251700 5 -- op tracker -- , seq: 86126, time: 2014-09-18 18:08:33.690367, event: started, op: pg_notify(2.4d0(10) epoch 11712) -3141> 2014-09-18 18:08:33.690393 7f4c9f251700 5 -- op tracker -- , seq: 86126, time: 2014-09-18 18:08:33.690391, event: done, op: pg_notify(2.4d0(10) epoch 11712) -3140> 2014-09-18 18:08:33.690409 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2364 ==== pg_notify(2.2c7(139) epoch 11712) v5 ==== 8651+0+0 (495086425 0 0) 0x19f37e00 con 0xdd8c940 -3139> 2014-09-18 18:08:33.690425 7f4c9f251700 5 -- op tracker -- , seq: 86127, time: 2014-09-18 18:08:33.687848, event: header_read, op: pg_notify(2.2c7(139) epoch 11712) -3138> 2014-09-18 18:08:33.690436 7f4c9f251700 5 -- op tracker -- , seq: 86127, time: 2014-09-18 18:08:33.687849, event: throttled, op: pg_notify(2.2c7(139) epoch 11712) -3137> 2014-09-18 18:08:33.690447 7f4c9f251700 5 -- op tracker -- , seq: 86127, time: 2014-09-18 18:08:33.687990, event: all_read, op: pg_notify(2.2c7(139) epoch 11712) -3136> 2014-09-18 18:08:33.690456 7f4c9f251700 5 -- op tracker -- , seq: 86127, time: 2014-09-18 18:08:33.690421, event: dispatched, op: pg_notify(2.2c7(139) epoch 11712) -3135> 2014-09-18 18:08:33.690467 7f4c9f251700 5 -- op tracker -- , seq: 86127, time: 2014-09-18 18:08:33.690464, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11712) -3134> 2014-09-18 18:08:33.690479 7f4c9f251700 5 -- op tracker -- , seq: 86127, time: 2014-09-18 18:08:33.690476, event: started, op: pg_notify(2.2c7(139) epoch 11712) -3133> 2014-09-18 18:08:33.690500 7f4c9f251700 5 -- op tracker -- , seq: 86127, time: 2014-09-18 18:08:33.690498, event: done, op: pg_notify(2.2c7(139) epoch 11712) -3132> 2014-09-18 18:08:33.690529 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3309 ==== pg_notify(2.1bb(9) epoch 11712) v5 ==== 1189+0+0 (1858916325 0 0) 0xf2423c0 con 0x4900580 -3131> 2014-09-18 18:08:33.690554 7f4c9f251700 5 -- op tracker -- , seq: 86128, time: 2014-09-18 18:08:33.689376, event: header_read, op: pg_notify(2.1bb(9) epoch 11712) -3130> 2014-09-18 18:08:33.690566 7f4c9f251700 5 -- op tracker -- , seq: 86128, time: 2014-09-18 18:08:33.689377, event: throttled, op: pg_notify(2.1bb(9) epoch 11712) -3129> 2014-09-18 18:08:33.690577 7f4c9f251700 5 -- op tracker -- , seq: 86128, time: 2014-09-18 18:08:33.689413, event: all_read, op: pg_notify(2.1bb(9) epoch 11712) -3128> 2014-09-18 18:08:33.690584 7f4c9f251700 5 -- op tracker -- , seq: 86128, time: 2014-09-18 18:08:33.690549, event: dispatched, op: pg_notify(2.1bb(9) epoch 11712) -3127> 2014-09-18 18:08:33.690592 7f4c9f251700 5 -- op tracker -- , seq: 86128, time: 2014-09-18 18:08:33.690590, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11712) -3126> 2014-09-18 18:08:33.690601 7f4c9f251700 5 -- op tracker -- , seq: 86128, time: 2014-09-18 18:08:33.690599, event: started, op: pg_notify(2.1bb(9) epoch 11712) -3125> 2014-09-18 18:08:33.690619 7f4c9f251700 5 -- op tracker -- , seq: 86128, time: 2014-09-18 18:08:33.690617, event: done, op: pg_notify(2.1bb(9) epoch 11712) -3124> 2014-09-18 18:08:33.690631 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1270 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0xcc033c0 con 0xe8db760 -3123> 2014-09-18 18:08:33.690643 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3122> 2014-09-18 18:08:33.690642 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11712..11712 src has 1..11712) v3 -- ?+0 0x22e16000 con 0x140f95a0 -3121> 2014-09-18 18:08:33.690657 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11712) v5 -- ?+0 0x14182f00 con 0x140f95a0 -3120> 2014-09-18 18:08:33.690656 7f4c8e22f700 2 -- 10.242.42.178:0/2483 >> 10.242.42.178:6844/4508 pipe(0x14b51c80 sd=59 :58504 s=4 pgs=71 cs=1 l=1 c=0x1d7c9440).reader couldn't read tag, (0) Success -3119> 2014-09-18 18:08:33.690674 7f4c8e22f700 2 -- 10.242.42.178:0/2483 >> 10.242.42.178:6844/4508 pipe(0x14b51c80 sd=59 :58504 s=4 pgs=71 cs=1 l=1 c=0x1d7c9440).fault (0) Success -3118> 2014-09-18 18:08:33.690672 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1271 ==== pg_notify(2.17e(17) epoch 11712) v5 ==== 1593+0+0 (2424305676 0 0) 0xf0db0e0 con 0xe8db760 -3117> 2014-09-18 18:08:33.690691 7f4c9f251700 5 -- op tracker -- , seq: 86129, time: 2014-09-18 18:08:33.690602, event: header_read, op: pg_notify(2.17e(17) epoch 11712) -3116> 2014-09-18 18:08:33.690708 7f4c9f251700 5 -- op tracker -- , seq: 86129, time: 2014-09-18 18:08:33.690602, event: throttled, op: pg_notify(2.17e(17) epoch 11712) -3115> 2014-09-18 18:08:33.690718 7f4c9f251700 5 -- op tracker -- , seq: 86129, time: 2014-09-18 18:08:33.690656, event: all_read, op: pg_notify(2.17e(17) epoch 11712) -3114> 2014-09-18 18:08:33.690725 7f4c9f251700 5 -- op tracker -- , seq: 86129, time: 2014-09-18 18:08:33.690686, event: dispatched, op: pg_notify(2.17e(17) epoch 11712) -3113> 2014-09-18 18:08:33.690736 7f4c9f251700 5 -- op tracker -- , seq: 86129, time: 2014-09-18 18:08:33.690733, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11712) -3112> 2014-09-18 18:08:33.690747 7f4c9f251700 5 -- op tracker -- , seq: 86129, time: 2014-09-18 18:08:33.690745, event: started, op: pg_notify(2.17e(17) epoch 11712) -3111> 2014-09-18 18:08:33.690769 7f4c9f251700 5 -- op tracker -- , seq: 86129, time: 2014-09-18 18:08:33.690767, event: done, op: pg_notify(2.17e(17) epoch 11712) -3110> 2014-09-18 18:08:33.691262 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11712) v5 -- ?+0 0xf93a1e0 con 0x140f95a0 -3109> 2014-09-18 18:08:33.691369 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11712..11712 src has 1..11712) v3 -- ?+0 0x19df3840 con 0x1395edc0 -3108> 2014-09-18 18:08:33.691387 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11712) v5 -- ?+0 0xf245a20 con 0x1395edc0 -3107> 2014-09-18 18:08:33.691931 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11712..11712 src has 1..11712) v3 -- ?+0 0xf85ab40 con 0xcbecec0 -3106> 2014-09-18 18:08:33.691956 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(11) epoch 11712) v5 -- ?+0 0x14182000 con 0xcbecec0 -3105> 2014-09-18 18:08:33.692110 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2017 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x2d7fa6c0 con 0xe2d9a20 -3104> 2014-09-18 18:08:33.692139 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3103> 2014-09-18 18:08:33.692160 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2018 ==== pg_notify(2.26d(9) epoch 11712) v5 ==== 1257+0+0 (2407071540 0 0) 0xf0a1660 con 0xe2d9a20 -3102> 2014-09-18 18:08:33.692170 7f4c9f251700 5 -- op tracker -- , seq: 86130, time: 2014-09-18 18:08:33.692111, event: header_read, op: pg_notify(2.26d(9) epoch 11712) -3101> 2014-09-18 18:08:33.692179 7f4c9f251700 5 -- op tracker -- , seq: 86130, time: 2014-09-18 18:08:33.692112, event: throttled, op: pg_notify(2.26d(9) epoch 11712) -3100> 2014-09-18 18:08:33.692184 7f4c9f251700 5 -- op tracker -- , seq: 86130, time: 2014-09-18 18:08:33.692149, event: all_read, op: pg_notify(2.26d(9) epoch 11712) -3099> 2014-09-18 18:08:33.692190 7f4c9f251700 5 -- op tracker -- , seq: 86130, time: 2014-09-18 18:08:33.692167, event: dispatched, op: pg_notify(2.26d(9) epoch 11712) -3098> 2014-09-18 18:08:33.692197 7f4c9f251700 5 -- op tracker -- , seq: 86130, time: 2014-09-18 18:08:33.692196, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11712) -3097> 2014-09-18 18:08:33.692205 7f4c9f251700 5 -- op tracker -- , seq: 86130, time: 2014-09-18 18:08:33.692204, event: started, op: pg_notify(2.26d(9) epoch 11712) -3096> 2014-09-18 18:08:33.692223 7f4c9f251700 5 -- op tracker -- , seq: 86130, time: 2014-09-18 18:08:33.692221, event: done, op: pg_notify(2.26d(9) epoch 11712) -3095> 2014-09-18 18:08:33.692455 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11712..11712 src has 1..11712) v3 -- ?+0 0x22e19cc0 con 0x1088aec0 -3094> 2014-09-18 18:08:33.692469 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.4e8(8) epoch 11712) v5 -- ?+0 0xddf2d20 con 0x1088aec0 -3093> 2014-09-18 18:08:33.692499 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11712..11712 src has 1..11712) v3 -- ?+0 0x22e19a80 con 0x23a74f20 -3092> 2014-09-18 18:08:33.692509 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11712) v5 -- ?+0 0xf9730e0 con 0x23a74f20 -3091> 2014-09-18 18:08:33.693039 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1202 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x1182a6c0 con 0x1088b440 -3090> 2014-09-18 18:08:33.693053 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3089> 2014-09-18 18:08:33.693372 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1203 ==== pg_notify(2.2d8(173) epoch 11712) v5 ==== 10757+0+0 (1991284274 0 0) 0xe76eee0 con 0x1088b440 -3088> 2014-09-18 18:08:33.693383 7f4c9f251700 5 -- op tracker -- , seq: 86131, time: 2014-09-18 18:08:33.693219, event: header_read, op: pg_notify(2.2d8(173) epoch 11712) -3087> 2014-09-18 18:08:33.693392 7f4c9f251700 5 -- op tracker -- , seq: 86131, time: 2014-09-18 18:08:33.693220, event: throttled, op: pg_notify(2.2d8(173) epoch 11712) -3086> 2014-09-18 18:08:33.693398 7f4c9f251700 5 -- op tracker -- , seq: 86131, time: 2014-09-18 18:08:33.693362, event: all_read, op: pg_notify(2.2d8(173) epoch 11712) -3085> 2014-09-18 18:08:33.693404 7f4c9f251700 5 -- op tracker -- , seq: 86131, time: 2014-09-18 18:08:33.693380, event: dispatched, op: pg_notify(2.2d8(173) epoch 11712) -3084> 2014-09-18 18:08:33.693410 7f4c9f251700 5 -- op tracker -- , seq: 86131, time: 2014-09-18 18:08:33.693409, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11712) -3083> 2014-09-18 18:08:33.693418 7f4c9f251700 5 -- op tracker -- , seq: 86131, time: 2014-09-18 18:08:33.693417, event: started, op: pg_notify(2.2d8(173) epoch 11712) -3082> 2014-09-18 18:08:33.693509 7f4c9f251700 5 -- op tracker -- , seq: 86131, time: 2014-09-18 18:08:33.693507, event: done, op: pg_notify(2.2d8(173) epoch 11712) -3081> 2014-09-18 18:08:33.693659 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 792 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x2d7fa900 con 0xe9a27e0 -3080> 2014-09-18 18:08:33.693670 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3079> 2014-09-18 18:08:33.693962 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 793 ==== pg_notify(2.2d8(173) epoch 11712) v5 ==== 10757+0+0 (1991284274 0 0) 0x1421e000 con 0xe9a27e0 -3078> 2014-09-18 18:08:33.693972 7f4c9f251700 5 -- op tracker -- , seq: 86132, time: 2014-09-18 18:08:33.693806, event: header_read, op: pg_notify(2.2d8(173) epoch 11712) -3077> 2014-09-18 18:08:33.693980 7f4c9f251700 5 -- op tracker -- , seq: 86132, time: 2014-09-18 18:08:33.693807, event: throttled, op: pg_notify(2.2d8(173) epoch 11712) -3076> 2014-09-18 18:08:33.693986 7f4c9f251700 5 -- op tracker -- , seq: 86132, time: 2014-09-18 18:08:33.693951, event: all_read, op: pg_notify(2.2d8(173) epoch 11712) -3075> 2014-09-18 18:08:33.693992 7f4c9f251700 5 -- op tracker -- , seq: 86132, time: 2014-09-18 18:08:33.693970, event: dispatched, op: pg_notify(2.2d8(173) epoch 11712) -3074> 2014-09-18 18:08:33.693998 7f4c9f251700 5 -- op tracker -- , seq: 86132, time: 2014-09-18 18:08:33.693997, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11712) -3073> 2014-09-18 18:08:33.694006 7f4c9f251700 5 -- op tracker -- , seq: 86132, time: 2014-09-18 18:08:33.694004, event: started, op: pg_notify(2.2d8(173) epoch 11712) -3072> 2014-09-18 18:08:33.694022 7f4c9f251700 5 -- op tracker -- , seq: 86132, time: 2014-09-18 18:08:33.694020, event: done, op: pg_notify(2.2d8(173) epoch 11712) -3071> 2014-09-18 18:08:33.694505 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2602 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x1440b3c0 con 0x4900b00 -3070> 2014-09-18 18:08:33.694542 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3069> 2014-09-18 18:08:33.694612 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2603 ==== pg_notify(2.71c(12) epoch 11712) v5 ==== 1424+0+0 (485842767 0 0) 0xf05d840 con 0x4900b00 -3068> 2014-09-18 18:08:33.694628 7f4c9f251700 5 -- op tracker -- , seq: 86133, time: 2014-09-18 18:08:33.694527, event: header_read, op: pg_notify(2.71c(12) epoch 11712) -3067> 2014-09-18 18:08:33.694638 7f4c9f251700 5 -- op tracker -- , seq: 86133, time: 2014-09-18 18:08:33.694528, event: throttled, op: pg_notify(2.71c(12) epoch 11712) -3066> 2014-09-18 18:08:33.694644 7f4c9f251700 5 -- op tracker -- , seq: 86133, time: 2014-09-18 18:08:33.694579, event: all_read, op: pg_notify(2.71c(12) epoch 11712) -3065> 2014-09-18 18:08:33.694650 7f4c9f251700 5 -- op tracker -- , seq: 86133, time: 2014-09-18 18:08:33.694625, event: dispatched, op: pg_notify(2.71c(12) epoch 11712) -3064> 2014-09-18 18:08:33.694656 7f4c9f251700 5 -- op tracker -- , seq: 86133, time: 2014-09-18 18:08:33.694655, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11712) -3063> 2014-09-18 18:08:33.694664 7f4c9f251700 5 -- op tracker -- , seq: 86133, time: 2014-09-18 18:08:33.694662, event: started, op: pg_notify(2.71c(12) epoch 11712) -3062> 2014-09-18 18:08:33.694681 7f4c9f251700 5 -- op tracker -- , seq: 86133, time: 2014-09-18 18:08:33.694679, event: done, op: pg_notify(2.71c(12) epoch 11712) -3061> 2014-09-18 18:08:33.721100 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3226 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x15e32d00 con 0x1395ec60 -3060> 2014-09-18 18:08:33.721120 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -3059> 2014-09-18 18:08:33.721141 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3227 ==== pg_notify(2.71(19) epoch 11712) v5 ==== 1795+0+0 (3193281202 0 0) 0x1393cb20 con 0x1395ec60 -3058> 2014-09-18 18:08:33.721150 7f4c9f251700 5 -- op tracker -- , seq: 86134, time: 2014-09-18 18:08:33.721092, event: header_read, op: pg_notify(2.71(19) epoch 11712) -3057> 2014-09-18 18:08:33.721165 7f4c9f251700 5 -- op tracker -- , seq: 86134, time: 2014-09-18 18:08:33.721093, event: throttled, op: pg_notify(2.71(19) epoch 11712) -3056> 2014-09-18 18:08:33.721174 7f4c9f251700 5 -- op tracker -- , seq: 86134, time: 2014-09-18 18:08:33.721129, event: all_read, op: pg_notify(2.71(19) epoch 11712) -3055> 2014-09-18 18:08:33.721182 7f4c9f251700 5 -- op tracker -- , seq: 86134, time: 2014-09-18 18:08:33.721147, event: dispatched, op: pg_notify(2.71(19) epoch 11712) -3054> 2014-09-18 18:08:33.721191 7f4c9f251700 5 -- op tracker -- , seq: 86134, time: 2014-09-18 18:08:33.721189, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11712) -3053> 2014-09-18 18:08:33.721201 7f4c9f251700 5 -- op tracker -- , seq: 86134, time: 2014-09-18 18:08:33.721199, event: started, op: pg_notify(2.71(19) epoch 11712) -3052> 2014-09-18 18:08:33.721219 7f4c9f251700 5 -- op tracker -- , seq: 86134, time: 2014-09-18 18:08:33.721217, event: done, op: pg_notify(2.71(19) epoch 11712) -3051> 2014-09-18 18:08:33.734997 7f4caae41700 5 osd.42 11712 tick -3050> 2014-09-18 18:08:33.784945 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5100 ==== pg_notify(2.11b(8),2.120(11) epoch 11709) v5 ==== 2507+0+0 (1828116905 0 0) 0xddf2d20 con 0x1088aec0 -3049> 2014-09-18 18:08:33.784979 7f4c9f251700 5 -- op tracker -- , seq: 86135, time: 2014-09-18 18:08:33.784844, event: header_read, op: pg_notify(2.11b(8),2.120(11) epoch 11709) -3048> 2014-09-18 18:08:33.784988 7f4c9f251700 5 -- op tracker -- , seq: 86135, time: 2014-09-18 18:08:33.784847, event: throttled, op: pg_notify(2.11b(8),2.120(11) epoch 11709) -3047> 2014-09-18 18:08:33.784993 7f4c9f251700 5 -- op tracker -- , seq: 86135, time: 2014-09-18 18:08:33.784915, event: all_read, op: pg_notify(2.11b(8),2.120(11) epoch 11709) -3046> 2014-09-18 18:08:33.784997 7f4c9f251700 5 -- op tracker -- , seq: 86135, time: 2014-09-18 18:08:33.784974, event: dispatched, op: pg_notify(2.11b(8),2.120(11) epoch 11709) -3045> 2014-09-18 18:08:33.785001 7f4c9f251700 5 -- op tracker -- , seq: 86135, time: 2014-09-18 18:08:33.785000, event: waiting_for_osdmap, op: pg_notify(2.11b(8),2.120(11) epoch 11709) -3044> 2014-09-18 18:08:33.785007 7f4c9f251700 5 -- op tracker -- , seq: 86135, time: 2014-09-18 18:08:33.785006, event: started, op: pg_notify(2.11b(8),2.120(11) epoch 11709) -3043> 2014-09-18 18:08:33.785037 7f4c9f251700 5 -- op tracker -- , seq: 86135, time: 2014-09-18 18:08:33.785035, event: done, op: pg_notify(2.11b(8),2.120(11) epoch 11709) -3042> 2014-09-18 18:08:33.785652 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5101 ==== pg_notify(2.40a(162),2.410(12) epoch 11709) v5 ==== 11062+0+0 (341109580 0 0) 0x19f372c0 con 0x1088aec0 -3041> 2014-09-18 18:08:33.785671 7f4c9f251700 5 -- op tracker -- , seq: 86136, time: 2014-09-18 18:08:33.785509, event: header_read, op: pg_notify(2.40a(162),2.410(12) epoch 11709) -3040> 2014-09-18 18:08:33.785680 7f4c9f251700 5 -- op tracker -- , seq: 86136, time: 2014-09-18 18:08:33.785511, event: throttled, op: pg_notify(2.40a(162),2.410(12) epoch 11709) -3039> 2014-09-18 18:08:33.785684 7f4c9f251700 5 -- op tracker -- , seq: 86136, time: 2014-09-18 18:08:33.785637, event: all_read, op: pg_notify(2.40a(162),2.410(12) epoch 11709) -3038> 2014-09-18 18:08:33.785689 7f4c9f251700 5 -- op tracker -- , seq: 86136, time: 2014-09-18 18:08:33.785667, event: dispatched, op: pg_notify(2.40a(162),2.410(12) epoch 11709) -3037> 2014-09-18 18:08:33.785696 7f4c9f251700 5 -- op tracker -- , seq: 86136, time: 2014-09-18 18:08:33.785695, event: waiting_for_osdmap, op: pg_notify(2.40a(162),2.410(12) epoch 11709) -3036> 2014-09-18 18:08:33.785710 7f4c9f251700 5 -- op tracker -- , seq: 86136, time: 2014-09-18 18:08:33.785708, event: started, op: pg_notify(2.40a(162),2.410(12) epoch 11709) -3035> 2014-09-18 18:08:33.785747 7f4c9f251700 5 -- op tracker -- , seq: 86136, time: 2014-09-18 18:08:33.785745, event: done, op: pg_notify(2.40a(162),2.410(12) epoch 11709) -3034> 2014-09-18 18:08:33.786188 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5102 ==== pg_notify(2.179(10) epoch 11709) v5 ==== 1276+0+0 (2576087095 0 0) 0x19f36b40 con 0x1088aec0 -3033> 2014-09-18 18:08:33.786207 7f4c9f251700 5 -- op tracker -- , seq: 86137, time: 2014-09-18 18:08:33.786056, event: header_read, op: pg_notify(2.179(10) epoch 11709) -3032> 2014-09-18 18:08:33.786215 7f4c9f251700 5 -- op tracker -- , seq: 86137, time: 2014-09-18 18:08:33.786057, event: throttled, op: pg_notify(2.179(10) epoch 11709) -3031> 2014-09-18 18:08:33.786219 7f4c9f251700 5 -- op tracker -- , seq: 86137, time: 2014-09-18 18:08:33.786086, event: all_read, op: pg_notify(2.179(10) epoch 11709) -3030> 2014-09-18 18:08:33.786237 7f4c9f251700 5 -- op tracker -- , seq: 86137, time: 2014-09-18 18:08:33.786203, event: dispatched, op: pg_notify(2.179(10) epoch 11709) -3029> 2014-09-18 18:08:33.786241 7f4c9f251700 5 -- op tracker -- , seq: 86137, time: 2014-09-18 18:08:33.786240, event: waiting_for_osdmap, op: pg_notify(2.179(10) epoch 11709) -3028> 2014-09-18 18:08:33.786247 7f4c9f251700 5 -- op tracker -- , seq: 86137, time: 2014-09-18 18:08:33.786246, event: started, op: pg_notify(2.179(10) epoch 11709) -3027> 2014-09-18 18:08:33.786264 7f4c9f251700 5 -- op tracker -- , seq: 86137, time: 2014-09-18 18:08:33.786262, event: done, op: pg_notify(2.179(10) epoch 11709) -3026> 2014-09-18 18:08:33.786905 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5103 ==== pg_notify(2.1a2(20) epoch 11709) v5 ==== 1798+0+0 (891929268 0 0) 0xf186b40 con 0x1088aec0 -3025> 2014-09-18 18:08:33.786927 7f4c9f251700 5 -- op tracker -- , seq: 86138, time: 2014-09-18 18:08:33.786836, event: header_read, op: pg_notify(2.1a2(20) epoch 11709) -3024> 2014-09-18 18:08:33.786935 7f4c9f251700 5 -- op tracker -- , seq: 86138, time: 2014-09-18 18:08:33.786837, event: throttled, op: pg_notify(2.1a2(20) epoch 11709) -3023> 2014-09-18 18:08:33.786939 7f4c9f251700 5 -- op tracker -- , seq: 86138, time: 2014-09-18 18:08:33.786889, event: all_read, op: pg_notify(2.1a2(20) epoch 11709) -3022> 2014-09-18 18:08:33.786943 7f4c9f251700 5 -- op tracker -- , seq: 86138, time: 2014-09-18 18:08:33.786922, event: dispatched, op: pg_notify(2.1a2(20) epoch 11709) -3021> 2014-09-18 18:08:33.786948 7f4c9f251700 5 -- op tracker -- , seq: 86138, time: 2014-09-18 18:08:33.786946, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11709) -3020> 2014-09-18 18:08:33.786953 7f4c9f251700 5 -- op tracker -- , seq: 86138, time: 2014-09-18 18:08:33.786952, event: started, op: pg_notify(2.1a2(20) epoch 11709) -3019> 2014-09-18 18:08:33.786972 7f4c9f251700 5 -- op tracker -- , seq: 86138, time: 2014-09-18 18:08:33.786970, event: done, op: pg_notify(2.1a2(20) epoch 11709) -3018> 2014-09-18 18:08:33.788285 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5104 ==== pg_notify(2.4c1(15) epoch 11709) v5 ==== 1507+0+0 (3734046664 0 0) 0xf189480 con 0x1088aec0 -3017> 2014-09-18 18:08:33.788308 7f4c9f251700 5 -- op tracker -- , seq: 86139, time: 2014-09-18 18:08:33.788235, event: header_read, op: pg_notify(2.4c1(15) epoch 11709) -3016> 2014-09-18 18:08:33.788322 7f4c9f251700 5 -- op tracker -- , seq: 86139, time: 2014-09-18 18:08:33.788235, event: throttled, op: pg_notify(2.4c1(15) epoch 11709) -3015> 2014-09-18 18:08:33.788332 7f4c9f251700 5 -- op tracker -- , seq: 86139, time: 2014-09-18 18:08:33.788270, event: all_read, op: pg_notify(2.4c1(15) epoch 11709) -3014> 2014-09-18 18:08:33.788341 7f4c9f251700 5 -- op tracker -- , seq: 86139, time: 2014-09-18 18:08:33.788303, event: dispatched, op: pg_notify(2.4c1(15) epoch 11709) -3013> 2014-09-18 18:08:33.788350 7f4c9f251700 5 -- op tracker -- , seq: 86139, time: 2014-09-18 18:08:33.788348, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11709) -3012> 2014-09-18 18:08:33.788362 7f4c9f251700 5 -- op tracker -- , seq: 86139, time: 2014-09-18 18:08:33.788359, event: started, op: pg_notify(2.4c1(15) epoch 11709) -3011> 2014-09-18 18:08:33.788389 7f4c9f251700 5 -- op tracker -- , seq: 86139, time: 2014-09-18 18:08:33.788386, event: done, op: pg_notify(2.4c1(15) epoch 11709) -3010> 2014-09-18 18:08:33.793166 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.13 10.242.42.168:0/10879 695 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:33.659827) v2 ==== 47+0+0 (1117638429 0 0) 0x24f6de00 con 0x140f7b80 -3009> 2014-09-18 18:08:33.793196 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:33.659827) v2 -- ?+0 0x1173a760 con 0x140f7b80 -3008> 2014-09-18 18:08:33.793306 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.13 10.242.42.168:0/10879 695 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:33.659827) v2 ==== 47+0+0 (1117638429 0 0) 0x24f785a0 con 0x109f9de0 -3007> 2014-09-18 18:08:33.793326 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:33.659827) v2 -- ?+0 0xf09eb40 con 0x109f9de0 -3006> 2014-09-18 18:08:33.811386 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3228 ==== pg_notify(2.519(4) epoch 11712) v5 ==== 962+0+0 (3036863704 0 0) 0x1393a5a0 con 0x1395ec60 -3005> 2014-09-18 18:08:33.811406 7f4c9f251700 5 -- op tracker -- , seq: 86140, time: 2014-09-18 18:08:33.811339, event: header_read, op: pg_notify(2.519(4) epoch 11712) -3004> 2014-09-18 18:08:33.811418 7f4c9f251700 5 -- op tracker -- , seq: 86140, time: 2014-09-18 18:08:33.811340, event: throttled, op: pg_notify(2.519(4) epoch 11712) -3003> 2014-09-18 18:08:33.811429 7f4c9f251700 5 -- op tracker -- , seq: 86140, time: 2014-09-18 18:08:33.811370, event: all_read, op: pg_notify(2.519(4) epoch 11712) -3002> 2014-09-18 18:08:33.811436 7f4c9f251700 5 -- op tracker -- , seq: 86140, time: 2014-09-18 18:08:33.811401, event: dispatched, op: pg_notify(2.519(4) epoch 11712) -3001> 2014-09-18 18:08:33.811442 7f4c9f251700 5 -- op tracker -- , seq: 86140, time: 2014-09-18 18:08:33.811441, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11712) -3000> 2014-09-18 18:08:33.811450 7f4c9f251700 5 -- op tracker -- , seq: 86140, time: 2014-09-18 18:08:33.811449, event: started, op: pg_notify(2.519(4) epoch 11712) -2999> 2014-09-18 18:08:33.811469 7f4c9f251700 5 -- op tracker -- , seq: 86140, time: 2014-09-18 18:08:33.811468, event: done, op: pg_notify(2.519(4) epoch 11712) -2998> 2014-09-18 18:08:33.814468 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3229 ==== pg_notify(2.410(12) epoch 11712) v5 ==== 1376+0+0 (3308426883 0 0) 0x1393cd00 con 0x1395ec60 -2997> 2014-09-18 18:08:33.814485 7f4c9f251700 5 -- op tracker -- , seq: 86141, time: 2014-09-18 18:08:33.814413, event: header_read, op: pg_notify(2.410(12) epoch 11712) -2996> 2014-09-18 18:08:33.814495 7f4c9f251700 5 -- op tracker -- , seq: 86141, time: 2014-09-18 18:08:33.814414, event: throttled, op: pg_notify(2.410(12) epoch 11712) -2995> 2014-09-18 18:08:33.814501 7f4c9f251700 5 -- op tracker -- , seq: 86141, time: 2014-09-18 18:08:33.814453, event: all_read, op: pg_notify(2.410(12) epoch 11712) -2994> 2014-09-18 18:08:33.814507 7f4c9f251700 5 -- op tracker -- , seq: 86141, time: 2014-09-18 18:08:33.814482, event: dispatched, op: pg_notify(2.410(12) epoch 11712) -2993> 2014-09-18 18:08:33.814514 7f4c9f251700 5 -- op tracker -- , seq: 86141, time: 2014-09-18 18:08:33.814512, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11712) -2992> 2014-09-18 18:08:33.814521 7f4c9f251700 5 -- op tracker -- , seq: 86141, time: 2014-09-18 18:08:33.814520, event: started, op: pg_notify(2.410(12) epoch 11712) -2991> 2014-09-18 18:08:33.814546 7f4c9f251700 5 -- op tracker -- , seq: 86141, time: 2014-09-18 18:08:33.814544, event: done, op: pg_notify(2.410(12) epoch 11712) -2990> 2014-09-18 18:08:33.815486 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3230 ==== pg_notify(2.186(12) epoch 11712) v5 ==== 1386+0+0 (3487469368 0 0) 0x1393dc00 con 0x1395ec60 -2989> 2014-09-18 18:08:33.815506 7f4c9f251700 5 -- op tracker -- , seq: 86142, time: 2014-09-18 18:08:33.815420, event: header_read, op: pg_notify(2.186(12) epoch 11712) -2988> 2014-09-18 18:08:33.815517 7f4c9f251700 5 -- op tracker -- , seq: 86142, time: 2014-09-18 18:08:33.815421, event: throttled, op: pg_notify(2.186(12) epoch 11712) -2987> 2014-09-18 18:08:33.815523 7f4c9f251700 5 -- op tracker -- , seq: 86142, time: 2014-09-18 18:08:33.815467, event: all_read, op: pg_notify(2.186(12) epoch 11712) -2986> 2014-09-18 18:08:33.815529 7f4c9f251700 5 -- op tracker -- , seq: 86142, time: 2014-09-18 18:08:33.815502, event: dispatched, op: pg_notify(2.186(12) epoch 11712) -2985> 2014-09-18 18:08:33.815545 7f4c9f251700 5 -- op tracker -- , seq: 86142, time: 2014-09-18 18:08:33.815544, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11712) -2984> 2014-09-18 18:08:33.815553 7f4c9f251700 5 -- op tracker -- , seq: 86142, time: 2014-09-18 18:08:33.815551, event: started, op: pg_notify(2.186(12) epoch 11712) -2983> 2014-09-18 18:08:33.815572 7f4c9f251700 5 -- op tracker -- , seq: 86142, time: 2014-09-18 18:08:33.815570, event: done, op: pg_notify(2.186(12) epoch 11712) -2982> 2014-09-18 18:08:33.816497 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3231 ==== pg_notify(2.74e(11) epoch 11712) v5 ==== 1303+0+0 (600078832 0 0) 0x1393b680 con 0x1395ec60 -2981> 2014-09-18 18:08:33.816514 7f4c9f251700 5 -- op tracker -- , seq: 86143, time: 2014-09-18 18:08:33.816448, event: header_read, op: pg_notify(2.74e(11) epoch 11712) -2980> 2014-09-18 18:08:33.816524 7f4c9f251700 5 -- op tracker -- , seq: 86143, time: 2014-09-18 18:08:33.816450, event: throttled, op: pg_notify(2.74e(11) epoch 11712) -2979> 2014-09-18 18:08:33.816530 7f4c9f251700 5 -- op tracker -- , seq: 86143, time: 2014-09-18 18:08:33.816483, event: all_read, op: pg_notify(2.74e(11) epoch 11712) -2978> 2014-09-18 18:08:33.816536 7f4c9f251700 5 -- op tracker -- , seq: 86143, time: 2014-09-18 18:08:33.816510, event: dispatched, op: pg_notify(2.74e(11) epoch 11712) -2977> 2014-09-18 18:08:33.816543 7f4c9f251700 5 -- op tracker -- , seq: 86143, time: 2014-09-18 18:08:33.816542, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11712) -2976> 2014-09-18 18:08:33.816552 7f4c9f251700 5 -- op tracker -- , seq: 86143, time: 2014-09-18 18:08:33.816551, event: started, op: pg_notify(2.74e(11) epoch 11712) -2975> 2014-09-18 18:08:33.816571 7f4c9f251700 5 -- op tracker -- , seq: 86143, time: 2014-09-18 18:08:33.816569, event: done, op: pg_notify(2.74e(11) epoch 11712) -2974> 2014-09-18 18:08:33.886128 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3232 ==== pg_notify(2.4b1(13) epoch 11712) v5 ==== 1441+0+0 (3771763815 0 0) 0x1393cee0 con 0x1395ec60 -2973> 2014-09-18 18:08:33.886159 7f4c9f251700 5 -- op tracker -- , seq: 86144, time: 2014-09-18 18:08:33.885996, event: header_read, op: pg_notify(2.4b1(13) epoch 11712) -2972> 2014-09-18 18:08:33.886171 7f4c9f251700 5 -- op tracker -- , seq: 86144, time: 2014-09-18 18:08:33.885999, event: throttled, op: pg_notify(2.4b1(13) epoch 11712) -2971> 2014-09-18 18:08:33.886180 7f4c9f251700 5 -- op tracker -- , seq: 86144, time: 2014-09-18 18:08:33.886071, event: all_read, op: pg_notify(2.4b1(13) epoch 11712) -2970> 2014-09-18 18:08:33.886188 7f4c9f251700 5 -- op tracker -- , seq: 86144, time: 2014-09-18 18:08:33.886152, event: dispatched, op: pg_notify(2.4b1(13) epoch 11712) -2969> 2014-09-18 18:08:33.886196 7f4c9f251700 5 -- op tracker -- , seq: 86144, time: 2014-09-18 18:08:33.886194, event: waiting_for_osdmap, op: pg_notify(2.4b1(13) epoch 11712) -2968> 2014-09-18 18:08:33.886206 7f4c9f251700 5 -- op tracker -- , seq: 86144, time: 2014-09-18 18:08:33.886205, event: started, op: pg_notify(2.4b1(13) epoch 11712) -2967> 2014-09-18 18:08:33.886230 7f4c9f251700 5 -- op tracker -- , seq: 86144, time: 2014-09-18 18:08:33.886228, event: done, op: pg_notify(2.4b1(13) epoch 11712) -2966> 2014-09-18 18:08:33.886339 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3233 ==== pg_notify(2.799(9),2.4d0(10) epoch 11712) v5 ==== 2509+0+0 (267306571 0 0) 0x1393d2a0 con 0x1395ec60 -2965> 2014-09-18 18:08:33.886492 7f4c9f251700 5 -- op tracker -- , seq: 86145, time: 2014-09-18 18:08:33.886280, event: header_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11712) -2964> 2014-09-18 18:08:33.886501 7f4c9f251700 5 -- op tracker -- , seq: 86145, time: 2014-09-18 18:08:33.886281, event: throttled, op: pg_notify(2.799(9),2.4d0(10) epoch 11712) -2963> 2014-09-18 18:08:33.886507 7f4c9f251700 5 -- op tracker -- , seq: 86145, time: 2014-09-18 18:08:33.886326, event: all_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11712) -2962> 2014-09-18 18:08:33.886514 7f4c9f251700 5 -- op tracker -- , seq: 86145, time: 2014-09-18 18:08:33.886487, event: dispatched, op: pg_notify(2.799(9),2.4d0(10) epoch 11712) -2961> 2014-09-18 18:08:33.886520 7f4c9f251700 5 -- op tracker -- , seq: 86145, time: 2014-09-18 18:08:33.886519, event: waiting_for_osdmap, op: pg_notify(2.799(9),2.4d0(10) epoch 11712) -2960> 2014-09-18 18:08:33.886528 7f4c9f251700 5 -- op tracker -- , seq: 86145, time: 2014-09-18 18:08:33.886526, event: started, op: pg_notify(2.799(9),2.4d0(10) epoch 11712) -2959> 2014-09-18 18:08:33.886586 7f4c9f251700 5 -- op tracker -- , seq: 86145, time: 2014-09-18 18:08:33.886583, event: done, op: pg_notify(2.799(9),2.4d0(10) epoch 11712) -2958> 2014-09-18 18:08:33.886676 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3234 ==== pg_notify(2.4c1(15),2.78f(53),2.4c6(15) epoch 11712) v5 ==== 6551+0+0 (344208313 0 0) 0x1393c940 con 0x1395ec60 -2957> 2014-09-18 18:08:33.886689 7f4c9f251700 5 -- op tracker -- , seq: 86146, time: 2014-09-18 18:08:33.886569, event: header_read, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15) epoch 11712) -2956> 2014-09-18 18:08:33.886697 7f4c9f251700 5 -- op tracker -- , seq: 86146, time: 2014-09-18 18:08:33.886569, event: throttled, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15) epoch 11712) -2955> 2014-09-18 18:08:33.886703 7f4c9f251700 5 -- op tracker -- , seq: 86146, time: 2014-09-18 18:08:33.886664, event: all_read, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15) epoch 11712) -2954> 2014-09-18 18:08:33.886710 7f4c9f251700 5 -- op tracker -- , seq: 86146, time: 2014-09-18 18:08:33.886685, event: dispatched, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15) epoch 11712) -2953> 2014-09-18 18:08:33.886717 7f4c9f251700 5 -- op tracker -- , seq: 86146, time: 2014-09-18 18:08:33.886715, event: waiting_for_osdmap, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15) epoch 11712) -2952> 2014-09-18 18:08:33.886725 7f4c9f251700 5 -- op tracker -- , seq: 86146, time: 2014-09-18 18:08:33.886723, event: started, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15) epoch 11712) -2951> 2014-09-18 18:08:33.886757 7f4c9f251700 5 -- op tracker -- , seq: 86146, time: 2014-09-18 18:08:33.886756, event: done, op: pg_notify(2.4c1(15),2.78f(53),2.4c6(15) epoch 11712) -2950> 2014-09-18 18:08:33.931956 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1861 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x1182d3c0 con 0x10f60520 -2949> 2014-09-18 18:08:33.931977 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -2948> 2014-09-18 18:08:33.931983 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1862 ==== pg_notify(2.4f6(15) epoch 11712) v5 ==== 1581+0+0 (3327856075 0 0) 0x117392c0 con 0x10f60520 -2947> 2014-09-18 18:08:33.932004 7f4c9f251700 5 -- op tracker -- , seq: 86147, time: 2014-09-18 18:08:33.931906, event: header_read, op: pg_notify(2.4f6(15) epoch 11712) -2946> 2014-09-18 18:08:33.932013 7f4c9f251700 5 -- op tracker -- , seq: 86147, time: 2014-09-18 18:08:33.931906, event: throttled, op: pg_notify(2.4f6(15) epoch 11712) -2945> 2014-09-18 18:08:33.932018 7f4c9f251700 5 -- op tracker -- , seq: 86147, time: 2014-09-18 18:08:33.931955, event: all_read, op: pg_notify(2.4f6(15) epoch 11712) -2944> 2014-09-18 18:08:33.932025 7f4c9f251700 5 -- op tracker -- , seq: 86147, time: 2014-09-18 18:08:33.932001, event: dispatched, op: pg_notify(2.4f6(15) epoch 11712) -2943> 2014-09-18 18:08:33.932031 7f4c9f251700 5 -- op tracker -- , seq: 86147, time: 2014-09-18 18:08:33.932030, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11712) -2942> 2014-09-18 18:08:33.932039 7f4c9f251700 5 -- op tracker -- , seq: 86147, time: 2014-09-18 18:08:33.932038, event: started, op: pg_notify(2.4f6(15) epoch 11712) -2941> 2014-09-18 18:08:33.932060 7f4c9f251700 5 -- op tracker -- , seq: 86147, time: 2014-09-18 18:08:33.932058, event: done, op: pg_notify(2.4f6(15) epoch 11712) -2940> 2014-09-18 18:08:33.934407 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1863 ==== pg_notify(2.1d8(9) epoch 11712) v5 ==== 1261+0+0 (4285252479 0 0) 0x11739860 con 0x10f60520 -2939> 2014-09-18 18:08:33.934425 7f4c9f251700 5 -- op tracker -- , seq: 86148, time: 2014-09-18 18:08:33.934326, event: header_read, op: pg_notify(2.1d8(9) epoch 11712) -2938> 2014-09-18 18:08:33.934436 7f4c9f251700 5 -- op tracker -- , seq: 86148, time: 2014-09-18 18:08:33.934327, event: throttled, op: pg_notify(2.1d8(9) epoch 11712) -2937> 2014-09-18 18:08:33.934442 7f4c9f251700 5 -- op tracker -- , seq: 86148, time: 2014-09-18 18:08:33.934378, event: all_read, op: pg_notify(2.1d8(9) epoch 11712) -2936> 2014-09-18 18:08:33.934448 7f4c9f251700 5 -- op tracker -- , seq: 86148, time: 2014-09-18 18:08:33.934421, event: dispatched, op: pg_notify(2.1d8(9) epoch 11712) -2935> 2014-09-18 18:08:33.934455 7f4c9f251700 5 -- op tracker -- , seq: 86148, time: 2014-09-18 18:08:33.934453, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11712) -2934> 2014-09-18 18:08:33.934467 7f4c9f251700 5 -- op tracker -- , seq: 86148, time: 2014-09-18 18:08:33.934465, event: started, op: pg_notify(2.1d8(9) epoch 11712) -2933> 2014-09-18 18:08:33.934496 7f4c9f251700 5 -- op tracker -- , seq: 86148, time: 2014-09-18 18:08:33.934493, event: done, op: pg_notify(2.1d8(9) epoch 11712) -2932> 2014-09-18 18:08:33.934511 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1864 ==== pg_notify(2.517(11) epoch 11712) v5 ==== 1335+0+0 (1284265879 0 0) 0xec681e0 con 0x10f60520 -2931> 2014-09-18 18:08:33.934522 7f4c9f251700 5 -- op tracker -- , seq: 86149, time: 2014-09-18 18:08:33.934396, event: header_read, op: pg_notify(2.517(11) epoch 11712) -2930> 2014-09-18 18:08:33.934529 7f4c9f251700 5 -- op tracker -- , seq: 86149, time: 2014-09-18 18:08:33.934396, event: throttled, op: pg_notify(2.517(11) epoch 11712) -2929> 2014-09-18 18:08:33.934535 7f4c9f251700 5 -- op tracker -- , seq: 86149, time: 2014-09-18 18:08:33.934454, event: all_read, op: pg_notify(2.517(11) epoch 11712) -2928> 2014-09-18 18:08:33.934550 7f4c9f251700 5 -- op tracker -- , seq: 86149, time: 2014-09-18 18:08:33.934519, event: dispatched, op: pg_notify(2.517(11) epoch 11712) -2927> 2014-09-18 18:08:33.934557 7f4c9f251700 5 -- op tracker -- , seq: 86149, time: 2014-09-18 18:08:33.934556, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11712) -2926> 2014-09-18 18:08:33.934566 7f4c9f251700 5 -- op tracker -- , seq: 86149, time: 2014-09-18 18:08:33.934563, event: started, op: pg_notify(2.517(11) epoch 11712) -2925> 2014-09-18 18:08:33.934720 7f4c9f251700 5 -- op tracker -- , seq: 86149, time: 2014-09-18 18:08:33.934717, event: done, op: pg_notify(2.517(11) epoch 11712) -2924> 2014-09-18 18:08:33.958269 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.45 10.242.42.178:0/7086 107 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:33.957520) v2 ==== 47+0+0 (1296452783 0 0) 0xf1e9480 con 0x10a5d9c0 -2923> 2014-09-18 18:08:33.958296 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:33.957520) v2 -- ?+0 0x24f6de00 con 0x10a5d9c0 -2922> 2014-09-18 18:08:33.958313 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.45 10.242.42.178:0/7086 107 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:33.957520) v2 ==== 47+0+0 (1296452783 0 0) 0xf811a20 con 0x110bb700 -2921> 2014-09-18 18:08:33.958335 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:33.957520) v2 -- ?+0 0x24f785a0 con 0x110bb700 -2920> 2014-09-18 18:08:34.010626 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.23 10.242.42.172:0/11463 1549 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:23.114426) v2 ==== 47+0+0 (4177200593 0 0) 0xfcc8780 con 0xe567080 -2919> 2014-09-18 18:08:34.010664 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/11463 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:23.114426) v2 -- ?+0 0xf811a20 con 0xe567080 -2918> 2014-09-18 18:08:34.011814 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.23 10.242.42.172:0/11463 1549 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:23.114426) v2 ==== 47+0+0 (4177200593 0 0) 0xe1a32c0 con 0xec5c160 -2917> 2014-09-18 18:08:34.011839 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/11463 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:23.114426) v2 -- ?+0 0xf1e9480 con 0xec5c160 -2916> 2014-09-18 18:08:34.041997 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.37 10.242.42.168:0/18319 106 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:33.908665) v2 ==== 47+0+0 (4179210588 0 0) 0x1421e5a0 con 0x11ffdce0 -2915> 2014-09-18 18:08:34.042014 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:33.908665) v2 -- ?+0 0xe1a32c0 con 0x11ffdce0 -2914> 2014-09-18 18:08:34.042088 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.37 10.242.42.168:0/18319 106 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:33.908665) v2 ==== 47+0+0 (4179210588 0 0) 0x11fd21c0 con 0x110bb860 -2913> 2014-09-18 18:08:34.042112 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:33.908665) v2 -- ?+0 0xfcc8780 con 0x110bb860 -2912> 2014-09-18 18:08:34.090833 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.10 10.242.42.168:0/10695 689 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:33.957494) v2 ==== 47+0+0 (3321648737 0 0) 0xf4e9a20 con 0x140f9b20 -2911> 2014-09-18 18:08:34.090849 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10695 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:33.957494) v2 -- ?+0 0x1421e5a0 con 0x140f9b20 -2910> 2014-09-18 18:08:34.090901 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.10 10.242.42.168:0/10695 689 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:33.957494) v2 ==== 47+0+0 (3321648737 0 0) 0xf2b7fe0 con 0x109f9b20 -2909> 2014-09-18 18:08:34.090925 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10695 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:33.957494) v2 -- ?+0 0x11fd21c0 con 0x109f9b20 -2908> 2014-09-18 18:08:34.108557 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3663 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0xf85ab40 con 0xcbecec0 -2907> 2014-09-18 18:08:34.108584 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -2906> 2014-09-18 18:08:34.108594 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3664 ==== pg_info(1 pgs e11712:2.654(10)) v4 ==== 1272+0+0 (2668661153 0 0) 0x14182000 con 0xcbecec0 -2905> 2014-09-18 18:08:34.108610 7f4c9f251700 5 -- op tracker -- , seq: 86150, time: 2014-09-18 18:08:34.108549, event: header_read, op: pg_info(1 pgs e11712:2.654(10)) -2904> 2014-09-18 18:08:34.108625 7f4c9f251700 5 -- op tracker -- , seq: 86150, time: 2014-09-18 18:08:34.108550, event: throttled, op: pg_info(1 pgs e11712:2.654(10)) -2903> 2014-09-18 18:08:34.108635 7f4c9f251700 5 -- op tracker -- , seq: 86150, time: 2014-09-18 18:08:34.108579, event: all_read, op: pg_info(1 pgs e11712:2.654(10)) -2902> 2014-09-18 18:08:34.108645 7f4c9f251700 5 -- op tracker -- , seq: 86150, time: 2014-09-18 18:08:34.108604, event: dispatched, op: pg_info(1 pgs e11712:2.654(10)) -2901> 2014-09-18 18:08:34.108656 7f4c9f251700 5 -- op tracker -- , seq: 86150, time: 2014-09-18 18:08:34.108653, event: waiting_for_osdmap, op: pg_info(1 pgs e11712:2.654(10)) -2900> 2014-09-18 18:08:34.108670 7f4c9f251700 5 -- op tracker -- , seq: 86150, time: 2014-09-18 18:08:34.108667, event: started, op: pg_info(1 pgs e11712:2.654(10)) -2899> 2014-09-18 18:08:34.108695 7f4c9f251700 5 -- op tracker -- , seq: 86150, time: 2014-09-18 18:08:34.108694, event: done, op: pg_info(1 pgs e11712:2.654(10)) -2898> 2014-09-18 18:08:34.108711 7f4c96a40700 5 osd.42 pg_epoch: 11712 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] exit Started/Stray 4.385612 16 0.000478 -2897> 2014-09-18 18:08:34.108817 7f4c96a40700 5 osd.42 pg_epoch: 11712 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -2896> 2014-09-18 18:08:34.108830 7f4c96a40700 5 osd.42 pg_epoch: 11712 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11505 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2895> 2014-09-18 18:08:34.109623 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11712:2.654) v4 -- ?+0 0x24fd03c0 con 0xcbecec0 -2894> 2014-09-18 18:08:34.111239 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3665 ==== pg_notify(2.4f6(12) epoch 11712) v5 ==== 1382+0+0 (1824214491 0 0) 0x24fd03c0 con 0xcbecec0 -2893> 2014-09-18 18:08:34.111258 7f4c9f251700 5 -- op tracker -- , seq: 86151, time: 2014-09-18 18:08:34.111181, event: header_read, op: pg_notify(2.4f6(12) epoch 11712) -2892> 2014-09-18 18:08:34.111267 7f4c9f251700 5 -- op tracker -- , seq: 86151, time: 2014-09-18 18:08:34.111181, event: throttled, op: pg_notify(2.4f6(12) epoch 11712) -2891> 2014-09-18 18:08:34.111271 7f4c9f251700 5 -- op tracker -- , seq: 86151, time: 2014-09-18 18:08:34.111212, event: all_read, op: pg_notify(2.4f6(12) epoch 11712) -2890> 2014-09-18 18:08:34.111275 7f4c9f251700 5 -- op tracker -- , seq: 86151, time: 2014-09-18 18:08:34.111254, event: dispatched, op: pg_notify(2.4f6(12) epoch 11712) -2889> 2014-09-18 18:08:34.111279 7f4c9f251700 5 -- op tracker -- , seq: 86151, time: 2014-09-18 18:08:34.111278, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11712) -2888> 2014-09-18 18:08:34.111286 7f4c9f251700 5 -- op tracker -- , seq: 86151, time: 2014-09-18 18:08:34.111284, event: started, op: pg_notify(2.4f6(12) epoch 11712) -2887> 2014-09-18 18:08:34.111306 7f4c9f251700 5 -- op tracker -- , seq: 86151, time: 2014-09-18 18:08:34.111304, event: done, op: pg_notify(2.4f6(12) epoch 11712) -2886> 2014-09-18 18:08:34.114641 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3666 ==== pg_notify(2.3b6(10) epoch 11712) v5 ==== 1260+0+0 (3666749630 0 0) 0xf4f0580 con 0xcbecec0 -2885> 2014-09-18 18:08:34.114667 7f4c9f251700 5 -- op tracker -- , seq: 86152, time: 2014-09-18 18:08:34.114597, event: header_read, op: pg_notify(2.3b6(10) epoch 11712) -2884> 2014-09-18 18:08:34.114677 7f4c9f251700 5 -- op tracker -- , seq: 86152, time: 2014-09-18 18:08:34.114598, event: throttled, op: pg_notify(2.3b6(10) epoch 11712) -2883> 2014-09-18 18:08:34.114684 7f4c9f251700 5 -- op tracker -- , seq: 86152, time: 2014-09-18 18:08:34.114628, event: all_read, op: pg_notify(2.3b6(10) epoch 11712) -2882> 2014-09-18 18:08:34.114690 7f4c9f251700 5 -- op tracker -- , seq: 86152, time: 2014-09-18 18:08:34.114662, event: dispatched, op: pg_notify(2.3b6(10) epoch 11712) -2881> 2014-09-18 18:08:34.114696 7f4c9f251700 5 -- op tracker -- , seq: 86152, time: 2014-09-18 18:08:34.114695, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11712) -2880> 2014-09-18 18:08:34.114704 7f4c9f251700 5 -- op tracker -- , seq: 86152, time: 2014-09-18 18:08:34.114703, event: started, op: pg_notify(2.3b6(10) epoch 11712) -2879> 2014-09-18 18:08:34.114723 7f4c9f251700 5 -- op tracker -- , seq: 86152, time: 2014-09-18 18:08:34.114722, event: done, op: pg_notify(2.3b6(10) epoch 11712) -2878> 2014-09-18 18:08:34.115979 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3667 ==== pg_notify(2.517(11) epoch 11712) v5 ==== 1343+0+0 (1483949464 0 0) 0xf4ef680 con 0xcbecec0 -2877> 2014-09-18 18:08:34.115998 7f4c9f251700 5 -- op tracker -- , seq: 86153, time: 2014-09-18 18:08:34.115937, event: header_read, op: pg_notify(2.517(11) epoch 11712) -2876> 2014-09-18 18:08:34.116008 7f4c9f251700 5 -- op tracker -- , seq: 86153, time: 2014-09-18 18:08:34.115938, event: throttled, op: pg_notify(2.517(11) epoch 11712) -2875> 2014-09-18 18:08:34.116015 7f4c9f251700 5 -- op tracker -- , seq: 86153, time: 2014-09-18 18:08:34.115967, event: all_read, op: pg_notify(2.517(11) epoch 11712) -2874> 2014-09-18 18:08:34.116021 7f4c9f251700 5 -- op tracker -- , seq: 86153, time: 2014-09-18 18:08:34.115993, event: dispatched, op: pg_notify(2.517(11) epoch 11712) -2873> 2014-09-18 18:08:34.116027 7f4c9f251700 5 -- op tracker -- , seq: 86153, time: 2014-09-18 18:08:34.116026, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11712) -2872> 2014-09-18 18:08:34.116035 7f4c9f251700 5 -- op tracker -- , seq: 86153, time: 2014-09-18 18:08:34.116034, event: started, op: pg_notify(2.517(11) epoch 11712) -2871> 2014-09-18 18:08:34.116072 7f4c9f251700 5 -- op tracker -- , seq: 86153, time: 2014-09-18 18:08:34.116070, event: done, op: pg_notify(2.517(11) epoch 11712) -2870> 2014-09-18 18:08:34.117554 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3668 ==== pg_notify(2.2c7(139) epoch 11712) v5 ==== 8651+0+0 (3694498994 0 0) 0xf4efe00 con 0xcbecec0 -2869> 2014-09-18 18:08:34.117570 7f4c9f251700 5 -- op tracker -- , seq: 86154, time: 2014-09-18 18:08:34.117437, event: header_read, op: pg_notify(2.2c7(139) epoch 11712) -2868> 2014-09-18 18:08:34.117579 7f4c9f251700 5 -- op tracker -- , seq: 86154, time: 2014-09-18 18:08:34.117438, event: throttled, op: pg_notify(2.2c7(139) epoch 11712) -2867> 2014-09-18 18:08:34.117585 7f4c9f251700 5 -- op tracker -- , seq: 86154, time: 2014-09-18 18:08:34.117541, event: all_read, op: pg_notify(2.2c7(139) epoch 11712) -2866> 2014-09-18 18:08:34.117592 7f4c9f251700 5 -- op tracker -- , seq: 86154, time: 2014-09-18 18:08:34.117566, event: dispatched, op: pg_notify(2.2c7(139) epoch 11712) -2865> 2014-09-18 18:08:34.117598 7f4c9f251700 5 -- op tracker -- , seq: 86154, time: 2014-09-18 18:08:34.117597, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11712) -2864> 2014-09-18 18:08:34.117606 7f4c9f251700 5 -- op tracker -- , seq: 86154, time: 2014-09-18 18:08:34.117604, event: started, op: pg_notify(2.2c7(139) epoch 11712) -2863> 2014-09-18 18:08:34.117624 7f4c9f251700 5 -- op tracker -- , seq: 86154, time: 2014-09-18 18:08:34.117623, event: done, op: pg_notify(2.2c7(139) epoch 11712) -2862> 2014-09-18 18:08:34.120013 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3669 ==== pg_notify(2.40a(162) epoch 11712) v5 ==== 9736+0+0 (136304118 0 0) 0xf4ee000 con 0xcbecec0 -2861> 2014-09-18 18:08:34.120032 7f4c9f251700 5 -- op tracker -- , seq: 86155, time: 2014-09-18 18:08:34.119876, event: header_read, op: pg_notify(2.40a(162) epoch 11712) -2860> 2014-09-18 18:08:34.120044 7f4c9f251700 5 -- op tracker -- , seq: 86155, time: 2014-09-18 18:08:34.119877, event: throttled, op: pg_notify(2.40a(162) epoch 11712) -2859> 2014-09-18 18:08:34.120054 7f4c9f251700 5 -- op tracker -- , seq: 86155, time: 2014-09-18 18:08:34.119996, event: all_read, op: pg_notify(2.40a(162) epoch 11712) -2858> 2014-09-18 18:08:34.120063 7f4c9f251700 5 -- op tracker -- , seq: 86155, time: 2014-09-18 18:08:34.120027, event: dispatched, op: pg_notify(2.40a(162) epoch 11712) -2857> 2014-09-18 18:08:34.120073 7f4c9f251700 5 -- op tracker -- , seq: 86155, time: 2014-09-18 18:08:34.120071, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11712) -2856> 2014-09-18 18:08:34.120084 7f4c9f251700 5 -- op tracker -- , seq: 86155, time: 2014-09-18 18:08:34.120082, event: started, op: pg_notify(2.40a(162) epoch 11712) -2855> 2014-09-18 18:08:34.120214 7f4c9f251700 5 -- op tracker -- , seq: 86155, time: 2014-09-18 18:08:34.120211, event: done, op: pg_notify(2.40a(162) epoch 11712) -2854> 2014-09-18 18:08:34.120303 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3670 ==== pg_notify(2.186(12),2.719(17) epoch 11712) v5 ==== 2975+0+0 (2015566569 0 0) 0xe2870e0 con 0xcbecec0 -2853> 2014-09-18 18:08:34.120319 7f4c9f251700 5 -- op tracker -- , seq: 86156, time: 2014-09-18 18:08:34.120111, event: header_read, op: pg_notify(2.186(12),2.719(17) epoch 11712) -2852> 2014-09-18 18:08:34.120328 7f4c9f251700 5 -- op tracker -- , seq: 86156, time: 2014-09-18 18:08:34.120112, event: throttled, op: pg_notify(2.186(12),2.719(17) epoch 11712) -2851> 2014-09-18 18:08:34.120334 7f4c9f251700 5 -- op tracker -- , seq: 86156, time: 2014-09-18 18:08:34.120223, event: all_read, op: pg_notify(2.186(12),2.719(17) epoch 11712) -2850> 2014-09-18 18:08:34.120340 7f4c9f251700 5 -- op tracker -- , seq: 86156, time: 2014-09-18 18:08:34.120315, event: dispatched, op: pg_notify(2.186(12),2.719(17) epoch 11712) -2849> 2014-09-18 18:08:34.120347 7f4c9f251700 5 -- op tracker -- , seq: 86156, time: 2014-09-18 18:08:34.120345, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17) epoch 11712) -2848> 2014-09-18 18:08:34.120355 7f4c9f251700 5 -- op tracker -- , seq: 86156, time: 2014-09-18 18:08:34.120353, event: started, op: pg_notify(2.186(12),2.719(17) epoch 11712) -2847> 2014-09-18 18:08:34.120389 7f4c9f251700 5 -- op tracker -- , seq: 86156, time: 2014-09-18 18:08:34.120387, event: done, op: pg_notify(2.186(12),2.719(17) epoch 11712) -2846> 2014-09-18 18:08:34.120899 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3671 ==== pg_notify(2.1bb(9) epoch 11712) v5 ==== 1189+0+0 (3259334134 0 0) 0xe795680 con 0xcbecec0 -2845> 2014-09-18 18:08:34.120910 7f4c9f251700 5 -- op tracker -- , seq: 86157, time: 2014-09-18 18:08:34.120863, event: header_read, op: pg_notify(2.1bb(9) epoch 11712) -2844> 2014-09-18 18:08:34.120917 7f4c9f251700 5 -- op tracker -- , seq: 86157, time: 2014-09-18 18:08:34.120864, event: throttled, op: pg_notify(2.1bb(9) epoch 11712) -2843> 2014-09-18 18:08:34.120927 7f4c9f251700 5 -- op tracker -- , seq: 86157, time: 2014-09-18 18:08:34.120889, event: all_read, op: pg_notify(2.1bb(9) epoch 11712) -2842> 2014-09-18 18:08:34.120938 7f4c9f251700 5 -- op tracker -- , seq: 86157, time: 2014-09-18 18:08:34.120907, event: dispatched, op: pg_notify(2.1bb(9) epoch 11712) -2841> 2014-09-18 18:08:34.120949 7f4c9f251700 5 -- op tracker -- , seq: 86157, time: 2014-09-18 18:08:34.120946, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11712) -2840> 2014-09-18 18:08:34.120962 7f4c9f251700 5 -- op tracker -- , seq: 86157, time: 2014-09-18 18:08:34.120959, event: started, op: pg_notify(2.1bb(9) epoch 11712) -2839> 2014-09-18 18:08:34.120987 7f4c9f251700 5 -- op tracker -- , seq: 86157, time: 2014-09-18 18:08:34.120985, event: done, op: pg_notify(2.1bb(9) epoch 11712) -2838> 2014-09-18 18:08:34.121123 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3672 ==== pg_notify(2.71c(12),2.457(21) epoch 11712) v5 ==== 3211+0+0 (2336226857 0 0) 0xe796ee0 con 0xcbecec0 -2837> 2014-09-18 18:08:34.121137 7f4c9f251700 5 -- op tracker -- , seq: 86158, time: 2014-09-18 18:08:34.121071, event: header_read, op: pg_notify(2.71c(12),2.457(21) epoch 11712) -2836> 2014-09-18 18:08:34.121145 7f4c9f251700 5 -- op tracker -- , seq: 86158, time: 2014-09-18 18:08:34.121072, event: throttled, op: pg_notify(2.71c(12),2.457(21) epoch 11712) -2835> 2014-09-18 18:08:34.121152 7f4c9f251700 5 -- op tracker -- , seq: 86158, time: 2014-09-18 18:08:34.121111, event: all_read, op: pg_notify(2.71c(12),2.457(21) epoch 11712) -2834> 2014-09-18 18:08:34.121158 7f4c9f251700 5 -- op tracker -- , seq: 86158, time: 2014-09-18 18:08:34.121134, event: dispatched, op: pg_notify(2.71c(12),2.457(21) epoch 11712) -2833> 2014-09-18 18:08:34.121165 7f4c9f251700 5 -- op tracker -- , seq: 86158, time: 2014-09-18 18:08:34.121163, event: waiting_for_osdmap, op: pg_notify(2.71c(12),2.457(21) epoch 11712) -2832> 2014-09-18 18:08:34.121172 7f4c9f251700 5 -- op tracker -- , seq: 86158, time: 2014-09-18 18:08:34.121171, event: started, op: pg_notify(2.71c(12),2.457(21) epoch 11712) -2831> 2014-09-18 18:08:34.121198 7f4c9f251700 5 -- op tracker -- , seq: 86158, time: 2014-09-18 18:08:34.121196, event: done, op: pg_notify(2.71c(12),2.457(21) epoch 11712) -2830> 2014-09-18 18:08:34.175096 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1732 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x1440a880 con 0x4900000 -2829> 2014-09-18 18:08:34.175115 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -2828> 2014-09-18 18:08:34.175219 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1733 ==== pg_notify(2.e0(79) epoch 11712) v5 ==== 5119+0+0 (2697275998 0 0) 0xf1eaf00 con 0x4900000 -2827> 2014-09-18 18:08:34.175232 7f4c9f251700 5 -- op tracker -- , seq: 86159, time: 2014-09-18 18:08:34.175103, event: header_read, op: pg_notify(2.e0(79) epoch 11712) -2826> 2014-09-18 18:08:34.175247 7f4c9f251700 5 -- op tracker -- , seq: 86159, time: 2014-09-18 18:08:34.175104, event: throttled, op: pg_notify(2.e0(79) epoch 11712) -2825> 2014-09-18 18:08:34.175253 7f4c9f251700 5 -- op tracker -- , seq: 86159, time: 2014-09-18 18:08:34.175206, event: all_read, op: pg_notify(2.e0(79) epoch 11712) -2824> 2014-09-18 18:08:34.175259 7f4c9f251700 5 -- op tracker -- , seq: 86159, time: 2014-09-18 18:08:34.175229, event: dispatched, op: pg_notify(2.e0(79) epoch 11712) -2823> 2014-09-18 18:08:34.175266 7f4c9f251700 5 -- op tracker -- , seq: 86159, time: 2014-09-18 18:08:34.175265, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11712) -2822> 2014-09-18 18:08:34.175275 7f4c9f251700 5 -- op tracker -- , seq: 86159, time: 2014-09-18 18:08:34.175273, event: started, op: pg_notify(2.e0(79) epoch 11712) -2821> 2014-09-18 18:08:34.175294 7f4c9f251700 5 -- op tracker -- , seq: 86159, time: 2014-09-18 18:08:34.175293, event: done, op: pg_notify(2.e0(79) epoch 11712) -2820> 2014-09-18 18:08:34.187972 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.22 10.242.42.172:0/37668 442 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:23.291975) v2 ==== 47+0+0 (2020467458 0 0) 0xfae9660 con 0x109caf20 -2819> 2014-09-18 18:08:34.188002 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:23.291975) v2 -- ?+0 0xf2b7fe0 con 0x109caf20 -2818> 2014-09-18 18:08:34.188061 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.22 10.242.42.172:0/37668 442 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:23.291975) v2 ==== 47+0+0 (2020467458 0 0) 0x2b9881c0 con 0x14393340 -2817> 2014-09-18 18:08:34.188083 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:23.291975) v2 -- ?+0 0xf4e9a20 con 0x14393340 -2816> 2014-09-18 18:08:34.188384 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 346 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0xe938ac0 con 0x12562260 -2815> 2014-09-18 18:08:34.188402 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -2814> 2014-09-18 18:08:34.188476 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 347 ==== pg_notify(2.719(17) epoch 11712) v5 ==== 1601+0+0 (3815621077 0 0) 0x11739a40 con 0x12562260 -2813> 2014-09-18 18:08:34.188493 7f4c9f251700 5 -- op tracker -- , seq: 86160, time: 2014-09-18 18:08:34.188407, event: header_read, op: pg_notify(2.719(17) epoch 11712) -2812> 2014-09-18 18:08:34.188503 7f4c9f251700 5 -- op tracker -- , seq: 86160, time: 2014-09-18 18:08:34.188408, event: throttled, op: pg_notify(2.719(17) epoch 11712) -2811> 2014-09-18 18:08:34.188509 7f4c9f251700 5 -- op tracker -- , seq: 86160, time: 2014-09-18 18:08:34.188462, event: all_read, op: pg_notify(2.719(17) epoch 11712) -2810> 2014-09-18 18:08:34.188516 7f4c9f251700 5 -- op tracker -- , seq: 86160, time: 2014-09-18 18:08:34.188489, event: dispatched, op: pg_notify(2.719(17) epoch 11712) -2809> 2014-09-18 18:08:34.188522 7f4c9f251700 5 -- op tracker -- , seq: 86160, time: 2014-09-18 18:08:34.188521, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11712) -2808> 2014-09-18 18:08:34.188531 7f4c9f251700 5 -- op tracker -- , seq: 86160, time: 2014-09-18 18:08:34.188529, event: started, op: pg_notify(2.719(17) epoch 11712) -2807> 2014-09-18 18:08:34.188549 7f4c9f251700 5 -- op tracker -- , seq: 86160, time: 2014-09-18 18:08:34.188548, event: done, op: pg_notify(2.719(17) epoch 11712) -2806> 2014-09-18 18:08:34.250061 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 698 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0x11bb9600 con 0x1f103e40 -2805> 2014-09-18 18:08:34.250092 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -2804> 2014-09-18 18:08:34.250120 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 699 ==== pg_notify(2.4f6(15) epoch 11712) v5 ==== 1543+0+0 (2905918477 0 0) 0xe956960 con 0x1f103e40 -2803> 2014-09-18 18:08:34.250146 7f4c9f251700 5 -- op tracker -- , seq: 86161, time: 2014-09-18 18:08:34.250059, event: header_read, op: pg_notify(2.4f6(15) epoch 11712) -2802> 2014-09-18 18:08:34.250159 7f4c9f251700 5 -- op tracker -- , seq: 86161, time: 2014-09-18 18:08:34.250059, event: throttled, op: pg_notify(2.4f6(15) epoch 11712) -2801> 2014-09-18 18:08:34.250165 7f4c9f251700 5 -- op tracker -- , seq: 86161, time: 2014-09-18 18:08:34.250108, event: all_read, op: pg_notify(2.4f6(15) epoch 11712) -2800> 2014-09-18 18:08:34.250172 7f4c9f251700 5 -- op tracker -- , seq: 86161, time: 2014-09-18 18:08:34.250140, event: dispatched, op: pg_notify(2.4f6(15) epoch 11712) -2799> 2014-09-18 18:08:34.250178 7f4c9f251700 5 -- op tracker -- , seq: 86161, time: 2014-09-18 18:08:34.250177, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11712) -2798> 2014-09-18 18:08:34.250187 7f4c9f251700 5 -- op tracker -- , seq: 86161, time: 2014-09-18 18:08:34.250186, event: started, op: pg_notify(2.4f6(15) epoch 11712) -2797> 2014-09-18 18:08:34.250211 7f4c9f251700 5 -- op tracker -- , seq: 86161, time: 2014-09-18 18:08:34.250210, event: done, op: pg_notify(2.4f6(15) epoch 11712) -2796> 2014-09-18 18:08:34.252394 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 700 ==== pg_notify(2.4c6(15) epoch 11712) v5 ==== 1511+0+0 (448591970 0 0) 0xe956d20 con 0x1f103e40 -2795> 2014-09-18 18:08:34.252414 7f4c9f251700 5 -- op tracker -- , seq: 86162, time: 2014-09-18 18:08:34.252281, event: header_read, op: pg_notify(2.4c6(15) epoch 11712) -2794> 2014-09-18 18:08:34.252425 7f4c9f251700 5 -- op tracker -- , seq: 86162, time: 2014-09-18 18:08:34.252282, event: throttled, op: pg_notify(2.4c6(15) epoch 11712) -2793> 2014-09-18 18:08:34.252431 7f4c9f251700 5 -- op tracker -- , seq: 86162, time: 2014-09-18 18:08:34.252334, event: all_read, op: pg_notify(2.4c6(15) epoch 11712) -2792> 2014-09-18 18:08:34.252438 7f4c9f251700 5 -- op tracker -- , seq: 86162, time: 2014-09-18 18:08:34.252410, event: dispatched, op: pg_notify(2.4c6(15) epoch 11712) -2791> 2014-09-18 18:08:34.252444 7f4c9f251700 5 -- op tracker -- , seq: 86162, time: 2014-09-18 18:08:34.252443, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11712) -2790> 2014-09-18 18:08:34.252452 7f4c9f251700 5 -- op tracker -- , seq: 86162, time: 2014-09-18 18:08:34.252451, event: started, op: pg_notify(2.4c6(15) epoch 11712) -2789> 2014-09-18 18:08:34.252476 7f4c9f251700 5 -- op tracker -- , seq: 86162, time: 2014-09-18 18:08:34.252474, event: done, op: pg_notify(2.4c6(15) epoch 11712) -2788> 2014-09-18 18:08:34.252489 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 701 ==== pg_notify(2.799(10) epoch 11712) v5 ==== 1280+0+0 (865292180 0 0) 0xe959c00 con 0x1f103e40 -2787> 2014-09-18 18:08:34.252499 7f4c9f251700 5 -- op tracker -- , seq: 86163, time: 2014-09-18 18:08:34.252426, event: header_read, op: pg_notify(2.799(10) epoch 11712) -2786> 2014-09-18 18:08:34.252507 7f4c9f251700 5 -- op tracker -- , seq: 86163, time: 2014-09-18 18:08:34.252426, event: throttled, op: pg_notify(2.799(10) epoch 11712) -2785> 2014-09-18 18:08:34.252515 7f4c9f251700 5 -- op tracker -- , seq: 86163, time: 2014-09-18 18:08:34.252453, event: all_read, op: pg_notify(2.799(10) epoch 11712) -2784> 2014-09-18 18:08:34.252519 7f4c9f251700 5 -- op tracker -- , seq: 86163, time: 2014-09-18 18:08:34.252497, event: dispatched, op: pg_notify(2.799(10) epoch 11712) -2783> 2014-09-18 18:08:34.252524 7f4c9f251700 5 -- op tracker -- , seq: 86163, time: 2014-09-18 18:08:34.252522, event: waiting_for_osdmap, op: pg_notify(2.799(10) epoch 11712) -2782> 2014-09-18 18:08:34.252531 7f4c9f251700 5 -- op tracker -- , seq: 86163, time: 2014-09-18 18:08:34.252530, event: started, op: pg_notify(2.799(10) epoch 11712) -2781> 2014-09-18 18:08:34.252557 7f4c9f251700 5 -- op tracker -- , seq: 86163, time: 2014-09-18 18:08:34.252556, event: done, op: pg_notify(2.799(10) epoch 11712) -2780> 2014-09-18 18:08:34.256349 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.57 10.242.42.176:0/988 119 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:39.004125) v2 ==== 47+0+0 (1630426358 0 0) 0x1368dc20 con 0x12561e40 -2779> 2014-09-18 18:08:34.256352 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.57 10.242.42.176:0/988 119 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:39.004125) v2 ==== 47+0+0 (1630426358 0 0) 0xf7292a0 con 0x12abfc80 -2778> 2014-09-18 18:08:34.256375 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/988 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:39.004125) v2 -- ?+0 0xfae9660 con 0x12561e40 -2777> 2014-09-18 18:08:34.256396 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/988 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:39.004125) v2 -- ?+0 0x2b9881c0 con 0x12abfc80 -2776> 2014-09-18 18:08:34.304405 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.51 10.242.42.168:0/15056 150 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:34.171231) v2 ==== 47+0+0 (353599400 0 0) 0x105092a0 con 0xcc76f20 -2775> 2014-09-18 18:08:34.304439 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/15056 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:34.171231) v2 -- ?+0 0xf7292a0 con 0xcc76f20 -2774> 2014-09-18 18:08:34.304453 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.51 10.242.42.168:0/15056 150 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:34.171231) v2 ==== 47+0+0 (353599400 0 0) 0xf7870e0 con 0x17de92e0 -2773> 2014-09-18 18:08:34.304479 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/15056 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:34.171231) v2 -- ?+0 0x1368dc20 con 0x17de92e0 -2772> 2014-09-18 18:08:34.339099 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=36 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).connect error 10.242.42.164:6854/39317, (111) Connection refused -2771> 2014-09-18 18:08:34.339126 7f4c773de700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6854/39317 pipe(0x10b64940 sd=36 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3aec0).fault (111) Connection refused -2770> 2014-09-18 18:08:34.393051 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 972 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0xe338ac0 con 0x10a878c0 -2769> 2014-09-18 18:08:34.393072 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11712,11712], i have 11712, src has [1,11712] -2768> 2014-09-18 18:08:34.393101 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 973 ==== pg_notify(2.517(11),2.4c6(15) epoch 11712) v5 ==== 2834+0+0 (911316413 0 0) 0x16251840 con 0x10a878c0 -2767> 2014-09-18 18:08:34.393113 7f4c9f251700 5 -- op tracker -- , seq: 86164, time: 2014-09-18 18:08:34.393048, event: header_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11712) -2766> 2014-09-18 18:08:34.393125 7f4c9f251700 5 -- op tracker -- , seq: 86164, time: 2014-09-18 18:08:34.393049, event: throttled, op: pg_notify(2.517(11),2.4c6(15) epoch 11712) -2765> 2014-09-18 18:08:34.393132 7f4c9f251700 5 -- op tracker -- , seq: 86164, time: 2014-09-18 18:08:34.393091, event: all_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11712) -2764> 2014-09-18 18:08:34.393138 7f4c9f251700 5 -- op tracker -- , seq: 86164, time: 2014-09-18 18:08:34.393109, event: dispatched, op: pg_notify(2.517(11),2.4c6(15) epoch 11712) -2763> 2014-09-18 18:08:34.393145 7f4c9f251700 5 -- op tracker -- , seq: 86164, time: 2014-09-18 18:08:34.393143, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.4c6(15) epoch 11712) -2762> 2014-09-18 18:08:34.393153 7f4c9f251700 5 -- op tracker -- , seq: 86164, time: 2014-09-18 18:08:34.393152, event: started, op: pg_notify(2.517(11),2.4c6(15) epoch 11712) -2761> 2014-09-18 18:08:34.393195 7f4c9f251700 5 -- op tracker -- , seq: 86164, time: 2014-09-18 18:08:34.393193, event: done, op: pg_notify(2.517(11),2.4c6(15) epoch 11712) -2760> 2014-09-18 18:08:34.442351 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3673 ==== pg_info(1 pgs e11709:2.6f8) v4 ==== 718+0+0 (4058256355 0 0) 0xe794d20 con 0xcbecec0 -2759> 2014-09-18 18:08:34.442369 7f4c9f251700 5 -- op tracker -- , seq: 86165, time: 2014-09-18 18:08:34.442290, event: header_read, op: pg_info(1 pgs e11709:2.6f8) -2758> 2014-09-18 18:08:34.442380 7f4c9f251700 5 -- op tracker -- , seq: 86165, time: 2014-09-18 18:08:34.442291, event: throttled, op: pg_info(1 pgs e11709:2.6f8) -2757> 2014-09-18 18:08:34.442388 7f4c9f251700 5 -- op tracker -- , seq: 86165, time: 2014-09-18 18:08:34.442337, event: all_read, op: pg_info(1 pgs e11709:2.6f8) -2756> 2014-09-18 18:08:34.442395 7f4c9f251700 5 -- op tracker -- , seq: 86165, time: 2014-09-18 18:08:34.442365, event: dispatched, op: pg_info(1 pgs e11709:2.6f8) -2755> 2014-09-18 18:08:34.442402 7f4c9f251700 5 -- op tracker -- , seq: 86165, time: 2014-09-18 18:08:34.442401, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.6f8) -2754> 2014-09-18 18:08:34.442411 7f4c9f251700 5 -- op tracker -- , seq: 86165, time: 2014-09-18 18:08:34.442410, event: started, op: pg_info(1 pgs e11709:2.6f8) -2753> 2014-09-18 18:08:34.442429 7f4c9f251700 5 -- op tracker -- , seq: 86165, time: 2014-09-18 18:08:34.442428, event: done, op: pg_info(1 pgs e11709:2.6f8) -2752> 2014-09-18 18:08:34.442451 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_info(1 pgs e11712:2.6f8) v4 -- ?+0 0xf972d20 con 0x1395edc0 -2751> 2014-09-18 18:08:34.442472 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11712:2.6f8) v4 -- ?+0 0xf973fe0 con 0xcbecec0 -2750> 2014-09-18 18:08:34.442688 7f4c97241700 5 osd.42 pg_epoch: 11712 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11709 n=519 ec=188 les/c 11709/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=35 from 0//0//-1 -2749> 2014-09-18 18:08:34.442827 7f4c97241700 5 osd.42 pg_epoch: 11712 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11709 n=519 ec=188 les/c 11709/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 35 from 0//0//-1 -2748> 2014-09-18 18:08:34.442848 7f4c97241700 5 osd.42 pg_epoch: 11712 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11709 n=519 ec=188 les/c 11709/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 3.758867 18 0.004013 -2747> 2014-09-18 18:08:34.442861 7f4c97241700 5 osd.42 pg_epoch: 11712 pg[2.6f8( v 6788'24342 (4060'20573,6788'24342] local-les=11709 n=519 ec=188 les/c 11709/10730 11708/11708/9945) [42,35,30]/[42,30] r=0 lpr=11708 pi=10729-11707/10 bft=35 crt=6788'24342 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -2746> 2014-09-18 18:08:34.444561 7f4c8e22f700 1 -- 10.242.42.178:6807/2483 >> :/0 pipe(0x10b659c0 sd=59 :6807 s=0 pgs=0 cs=0 l=0 c=0x12b3b2e0).accept sd=59 10.242.42.164:40168/0 -2745> 2014-09-18 18:08:34.479467 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.30 10.242.42.176:0/1204 120 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:39.227217) v2 ==== 47+0+0 (2122528471 0 0) 0xf188b20 con 0x110bbc80 -2744> 2014-09-18 18:08:34.479492 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/1204 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:39.227217) v2 -- ?+0 0xf7870e0 con 0x110bbc80 -2743> 2014-09-18 18:08:34.479463 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.30 10.242.42.176:0/1204 120 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:39.227217) v2 ==== 47+0+0 (2122528471 0 0) 0x2b94c940 con 0x12abd1e0 -2742> 2014-09-18 18:08:34.479571 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/1204 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:39.227217) v2 -- ?+0 0x105092a0 con 0x12abd1e0 -2741> 2014-09-18 18:08:34.509966 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3674 ==== pg_info(1 pgs e11709:2.179) v4 ==== 718+0+0 (2457641373 0 0) 0xf973fe0 con 0xcbecec0 -2740> 2014-09-18 18:08:34.509985 7f4c9f251700 5 -- op tracker -- , seq: 86166, time: 2014-09-18 18:08:34.509839, event: header_read, op: pg_info(1 pgs e11709:2.179) -2739> 2014-09-18 18:08:34.509993 7f4c9f251700 5 -- op tracker -- , seq: 86166, time: 2014-09-18 18:08:34.509840, event: throttled, op: pg_info(1 pgs e11709:2.179) -2738> 2014-09-18 18:08:34.509997 7f4c9f251700 5 -- op tracker -- , seq: 86166, time: 2014-09-18 18:08:34.509865, event: all_read, op: pg_info(1 pgs e11709:2.179) -2737> 2014-09-18 18:08:34.510014 7f4c9f251700 5 -- op tracker -- , seq: 86166, time: 2014-09-18 18:08:34.509981, event: dispatched, op: pg_info(1 pgs e11709:2.179) -2736> 2014-09-18 18:08:34.510019 7f4c9f251700 5 -- op tracker -- , seq: 86166, time: 2014-09-18 18:08:34.510018, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.179) -2735> 2014-09-18 18:08:34.510025 7f4c9f251700 5 -- op tracker -- , seq: 86166, time: 2014-09-18 18:08:34.510023, event: started, op: pg_info(1 pgs e11709:2.179) -2734> 2014-09-18 18:08:34.510041 7f4c9f251700 5 -- op tracker -- , seq: 86166, time: 2014-09-18 18:08:34.510040, event: done, op: pg_info(1 pgs e11709:2.179) -2733> 2014-09-18 18:08:34.510154 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11712:2.179) v4 -- ?+0 0xf866f00 con 0xcbecec0 -2732> 2014-09-18 18:08:34.510176 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6801/988 -- pg_info(1 pgs e11712:2.179) v4 -- ?+0 0xf91a780 con 0x1395f080 -2731> 2014-09-18 18:08:34.510275 7f4c96a40700 5 osd.42 pg_epoch: 11712 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11709 n=519 ec=188 les/c 11709/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=35 from 0//0//-1 -2730> 2014-09-18 18:08:34.510401 7f4c96a40700 5 osd.42 pg_epoch: 11712 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11709 n=519 ec=188 les/c 11709/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 35 from 0//0//-1 -2729> 2014-09-18 18:08:34.510625 7f4c96a40700 5 osd.42 pg_epoch: 11712 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11709 n=519 ec=188 les/c 11709/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 3.823804 18 0.003175 -2728> 2014-09-18 18:08:34.510639 7f4c96a40700 5 osd.42 pg_epoch: 11712 pg[2.179( v 6752'29654 (4088'25910,6752'29654] local-les=11709 n=519 ec=188 les/c 11709/10657 11708/11708/9945) [42,35,57]/[42,57] r=0 lpr=11708 pi=10656-11707/10 bft=35 crt=6752'29654 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -2727> 2014-09-18 18:08:34.542418 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3675 ==== pg_info(1 pgs e11709:2.78f) v4 ==== 738+0+0 (4190966377 0 0) 0xf866f00 con 0xcbecec0 -2726> 2014-09-18 18:08:34.542435 7f4c9f251700 5 -- op tracker -- , seq: 86167, time: 2014-09-18 18:08:34.542295, event: header_read, op: pg_info(1 pgs e11709:2.78f) -2725> 2014-09-18 18:08:34.542443 7f4c9f251700 5 -- op tracker -- , seq: 86167, time: 2014-09-18 18:08:34.542296, event: throttled, op: pg_info(1 pgs e11709:2.78f) -2724> 2014-09-18 18:08:34.542447 7f4c9f251700 5 -- op tracker -- , seq: 86167, time: 2014-09-18 18:08:34.542317, event: all_read, op: pg_info(1 pgs e11709:2.78f) -2723> 2014-09-18 18:08:34.542464 7f4c9f251700 5 -- op tracker -- , seq: 86167, time: 2014-09-18 18:08:34.542432, event: dispatched, op: pg_info(1 pgs e11709:2.78f) -2722> 2014-09-18 18:08:34.542468 7f4c9f251700 5 -- op tracker -- , seq: 86167, time: 2014-09-18 18:08:34.542467, event: waiting_for_osdmap, op: pg_info(1 pgs e11709:2.78f) -2721> 2014-09-18 18:08:34.542474 7f4c9f251700 5 -- op tracker -- , seq: 86167, time: 2014-09-18 18:08:34.542473, event: started, op: pg_info(1 pgs e11709:2.78f) -2720> 2014-09-18 18:08:34.542490 7f4c9f251700 5 -- op tracker -- , seq: 86167, time: 2014-09-18 18:08:34.542489, event: done, op: pg_info(1 pgs e11709:2.78f) -2719> 2014-09-18 18:08:34.543335 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.37 10.242.42.168:0/18319 107 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:34.409848) v2 ==== 47+0+0 (9812693 0 0) 0x11fd12c0 con 0x110bb860 -2718> 2014-09-18 18:08:34.543372 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:34.409848) v2 -- ?+0 0xf188b20 con 0x110bb860 -2717> 2014-09-18 18:08:34.543383 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.37 10.242.42.168:0/18319 107 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:34.409848) v2 ==== 47+0+0 (9812693 0 0) 0x1421fa40 con 0x11ffdce0 -2716> 2014-09-18 18:08:34.543407 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:34.409848) v2 -- ?+0 0x2b94c940 con 0x11ffdce0 -2715> 2014-09-18 18:08:34.550845 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.9 10.242.42.166:0/24959 112 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:34.553131) v2 ==== 47+0+0 (1028314443 0 0) 0x146e1660 con 0x10a5c520 -2714> 2014-09-18 18:08:34.550863 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.9 10.242.42.166:0/24959 112 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:34.553131) v2 ==== 47+0+0 (1028314443 0 0) 0xedfb680 con 0x110bb9c0 -2713> 2014-09-18 18:08:34.550870 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:34.553131) v2 -- ?+0 0x1421fa40 con 0x10a5c520 -2712> 2014-09-18 18:08:34.550911 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/24959 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:34.553131) v2 -- ?+0 0x11fd12c0 con 0x110bb9c0 -2711> 2014-09-18 18:08:34.606727 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.43 10.242.42.178:0/4511 147 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:34.606016) v2 ==== 47+0+0 (1475455197 0 0) 0xf0a01c0 con 0xcc76dc0 -2710> 2014-09-18 18:08:34.606754 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/4511 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:34.606016) v2 -- ?+0 0x146e1660 con 0xcc76dc0 -2709> 2014-09-18 18:08:34.606764 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.43 10.242.42.178:0/4511 147 ==== osd_ping(ping e11712 stamp 2014-09-18 18:08:34.606016) v2 ==== 47+0+0 (1475455197 0 0) 0x21cfc760 con 0x17de6000 -2708> 2014-09-18 18:08:34.606793 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/4511 -- osd_ping(ping_reply e11712 stamp 2014-09-18 18:08:34.606016) v2 -- ?+0 0xedfb680 con 0x17de6000 -2707> 2014-09-18 18:08:34.689995 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.22 10.242.42.172:0/37668 443 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:23.793787) v2 ==== 47+0+0 (1120768311 0 0) 0x2b989840 con 0x14393340 -2706> 2014-09-18 18:08:34.690028 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:23.793787) v2 -- ?+0 0xf0a01c0 con 0x14393340 -2705> 2014-09-18 18:08:34.690037 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.22 10.242.42.172:0/37668 443 ==== osd_ping(ping e11712 stamp 2014-09-18 23:37:23.793787) v2 ==== 47+0+0 (1120768311 0 0) 0xfae7e00 con 0x109caf20 -2704> 2014-09-18 18:08:34.690059 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11712 stamp 2014-09-18 23:37:23.793787) v2 -- ?+0 0x21cfc760 con 0x109caf20 -2703> 2014-09-18 18:08:34.696234 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5105 ==== pg_notify(2.4e8(1) epoch 11710) v5 ==== 805+0+0 (3013660266 0 0) 0x1390c580 con 0x1088aec0 -2702> 2014-09-18 18:08:34.696255 7f4c9f251700 5 -- op tracker -- , seq: 86168, time: 2014-09-18 18:08:34.696165, event: header_read, op: pg_notify(2.4e8(1) epoch 11710) -2701> 2014-09-18 18:08:34.696264 7f4c9f251700 5 -- op tracker -- , seq: 86168, time: 2014-09-18 18:08:34.696167, event: throttled, op: pg_notify(2.4e8(1) epoch 11710) -2700> 2014-09-18 18:08:34.696273 7f4c9f251700 5 -- op tracker -- , seq: 86168, time: 2014-09-18 18:08:34.696214, event: all_read, op: pg_notify(2.4e8(1) epoch 11710) -2699> 2014-09-18 18:08:34.696280 7f4c9f251700 5 -- op tracker -- , seq: 86168, time: 2014-09-18 18:08:34.696248, event: dispatched, op: pg_notify(2.4e8(1) epoch 11710) -2698> 2014-09-18 18:08:34.696288 7f4c9f251700 5 -- op tracker -- , seq: 86168, time: 2014-09-18 18:08:34.696286, event: waiting_for_osdmap, op: pg_notify(2.4e8(1) epoch 11710) -2697> 2014-09-18 18:08:34.696296 7f4c9f251700 5 -- op tracker -- , seq: 86168, time: 2014-09-18 18:08:34.696295, event: started, op: pg_notify(2.4e8(1) epoch 11710) -2696> 2014-09-18 18:08:34.696315 7f4c9f251700 5 -- op tracker -- , seq: 86168, time: 2014-09-18 18:08:34.696314, event: done, op: pg_notify(2.4e8(1) epoch 11710) -2695> 2014-09-18 18:08:34.698062 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5106 ==== pg_notify(2.62(44) epoch 11710) v5 ==== 3220+0+0 (183455941 0 0) 0x130401e0 con 0x1088aec0 -2694> 2014-09-18 18:08:34.698083 7f4c9f251700 5 -- op tracker -- , seq: 86169, time: 2014-09-18 18:08:34.697929, event: header_read, op: pg_notify(2.62(44) epoch 11710) -2693> 2014-09-18 18:08:34.698093 7f4c9f251700 5 -- op tracker -- , seq: 86169, time: 2014-09-18 18:08:34.697930, event: throttled, op: pg_notify(2.62(44) epoch 11710) -2692> 2014-09-18 18:08:34.698100 7f4c9f251700 5 -- op tracker -- , seq: 86169, time: 2014-09-18 18:08:34.698025, event: all_read, op: pg_notify(2.62(44) epoch 11710) -2691> 2014-09-18 18:08:34.698106 7f4c9f251700 5 -- op tracker -- , seq: 86169, time: 2014-09-18 18:08:34.698078, event: dispatched, op: pg_notify(2.62(44) epoch 11710) -2690> 2014-09-18 18:08:34.698113 7f4c9f251700 5 -- op tracker -- , seq: 86169, time: 2014-09-18 18:08:34.698112, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11710) -2689> 2014-09-18 18:08:34.698122 7f4c9f251700 5 -- op tracker -- , seq: 86169, time: 2014-09-18 18:08:34.698120, event: started, op: pg_notify(2.62(44) epoch 11710) -2688> 2014-09-18 18:08:34.698149 7f4c9f251700 5 -- op tracker -- , seq: 86169, time: 2014-09-18 18:08:34.698146, event: done, op: pg_notify(2.62(44) epoch 11710) -2687> 2014-09-18 18:08:34.700483 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5107 ==== pg_notify(2.3b6(10) epoch 11710) v5 ==== 1260+0+0 (382496484 0 0) 0xf7863c0 con 0x1088aec0 -2686> 2014-09-18 18:08:34.700506 7f4c9f251700 5 -- op tracker -- , seq: 86170, time: 2014-09-18 18:08:34.700378, event: header_read, op: pg_notify(2.3b6(10) epoch 11710) -2685> 2014-09-18 18:08:34.700516 7f4c9f251700 5 -- op tracker -- , seq: 86170, time: 2014-09-18 18:08:34.700381, event: throttled, op: pg_notify(2.3b6(10) epoch 11710) -2684> 2014-09-18 18:08:34.700523 7f4c9f251700 5 -- op tracker -- , seq: 86170, time: 2014-09-18 18:08:34.700464, event: all_read, op: pg_notify(2.3b6(10) epoch 11710) -2683> 2014-09-18 18:08:34.700535 7f4c9f251700 5 -- op tracker -- , seq: 86170, time: 2014-09-18 18:08:34.700500, event: dispatched, op: pg_notify(2.3b6(10) epoch 11710) -2682> 2014-09-18 18:08:34.700542 7f4c9f251700 5 -- op tracker -- , seq: 86170, time: 2014-09-18 18:08:34.700540, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11710) -2681> 2014-09-18 18:08:34.700551 7f4c9f251700 5 -- op tracker -- , seq: 86170, time: 2014-09-18 18:08:34.700549, event: started, op: pg_notify(2.3b6(10) epoch 11710) -2680> 2014-09-18 18:08:34.700572 7f4c9f251700 5 -- op tracker -- , seq: 86170, time: 2014-09-18 18:08:34.700570, event: done, op: pg_notify(2.3b6(10) epoch 11710) -2679> 2014-09-18 18:08:34.701253 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5108 ==== pg_notify(2.6f8(10) epoch 11710) v5 ==== 1276+0+0 (2006913116 0 0) 0xf787fe0 con 0x1088aec0 -2678> 2014-09-18 18:08:34.701270 7f4c9f251700 5 -- op tracker -- , seq: 86171, time: 2014-09-18 18:08:34.701151, event: header_read, op: pg_notify(2.6f8(10) epoch 11710) -2677> 2014-09-18 18:08:34.701285 7f4c9f251700 5 -- op tracker -- , seq: 86171, time: 2014-09-18 18:08:34.701152, event: throttled, op: pg_notify(2.6f8(10) epoch 11710) -2676> 2014-09-18 18:08:34.701292 7f4c9f251700 5 -- op tracker -- , seq: 86171, time: 2014-09-18 18:08:34.701240, event: all_read, op: pg_notify(2.6f8(10) epoch 11710) -2675> 2014-09-18 18:08:34.701298 7f4c9f251700 5 -- op tracker -- , seq: 86171, time: 2014-09-18 18:08:34.701266, event: dispatched, op: pg_notify(2.6f8(10) epoch 11710) -2674> 2014-09-18 18:08:34.701304 7f4c9f251700 5 -- op tracker -- , seq: 86171, time: 2014-09-18 18:08:34.701303, event: waiting_for_osdmap, op: pg_notify(2.6f8(10) epoch 11710) -2673> 2014-09-18 18:08:34.701312 7f4c9f251700 5 -- op tracker -- , seq: 86171, time: 2014-09-18 18:08:34.701310, event: started, op: pg_notify(2.6f8(10) epoch 11710) -2672> 2014-09-18 18:08:34.701332 7f4c9f251700 5 -- op tracker -- , seq: 86171, time: 2014-09-18 18:08:34.701330, event: done, op: pg_notify(2.6f8(10) epoch 11710) -2671> 2014-09-18 18:08:34.702173 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5109 ==== pg_notify(2.517(11),2.519(23) epoch 11710) v5 ==== 3274+0+0 (65014839 0 0) 0xf7892a0 con 0x1088aec0 -2670> 2014-09-18 18:08:34.702186 7f4c9f251700 5 -- op tracker -- , seq: 86172, time: 2014-09-18 18:08:34.702108, event: header_read, op: pg_notify(2.517(11),2.519(23) epoch 11710) -2669> 2014-09-18 18:08:34.702194 7f4c9f251700 5 -- op tracker -- , seq: 86172, time: 2014-09-18 18:08:34.702109, event: throttled, op: pg_notify(2.517(11),2.519(23) epoch 11710) -2668> 2014-09-18 18:08:34.702200 7f4c9f251700 5 -- op tracker -- , seq: 86172, time: 2014-09-18 18:08:34.702162, event: all_read, op: pg_notify(2.517(11),2.519(23) epoch 11710) -2667> 2014-09-18 18:08:34.702206 7f4c9f251700 5 -- op tracker -- , seq: 86172, time: 2014-09-18 18:08:34.702183, event: dispatched, op: pg_notify(2.517(11),2.519(23) epoch 11710) -2666> 2014-09-18 18:08:34.702213 7f4c9f251700 5 -- op tracker -- , seq: 86172, time: 2014-09-18 18:08:34.702211, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.519(23) epoch 11710) -2665> 2014-09-18 18:08:34.702220 7f4c9f251700 5 -- op tracker -- , seq: 86172, time: 2014-09-18 18:08:34.702219, event: started, op: pg_notify(2.517(11),2.519(23) epoch 11710) -2664> 2014-09-18 18:08:34.702249 7f4c9f251700 5 -- op tracker -- , seq: 86172, time: 2014-09-18 18:08:34.702248, event: done, op: pg_notify(2.517(11),2.519(23) epoch 11710) -2663> 2014-09-18 18:08:34.719124 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3676 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x1533bd40 con 0xcbecec0 -2662> 2014-09-18 18:08:34.719162 7f4c9f251700 3 osd.42 11712 handle_osd_map epochs [11713,11713], i have 11712, src has [1,11713] -2661> 2014-09-18 18:08:34.722579 7f4c9f251700 1 -- 10.242.42.178:6807/2483 mark_down 10.242.42.164:6854/39317 -- 0x10b64940 -2660> 2014-09-18 18:08:34.722602 7f4c9f251700 1 -- 10.242.42.178:0/2483 mark_down 0x23a75760 -- 0x10b64680 -2659> 2014-09-18 18:08:34.722613 7f4c9f251700 1 -- 10.242.42.178:0/2483 mark_down 0x23a746e0 -- 0x13d50680 -2658> 2014-09-18 18:08:34.723252 7f4c9f251700 1 osd.42 11713 ms_handle_reset con 0x12b3aec0 session 0xf146000 -2657> 2014-09-18 18:08:34.723262 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3677 ==== pg_notify(2.4f6(12) epoch 11713) v5 ==== 1382+0+0 (3289036305 0 0) 0xe797480 con 0xcbecec0 -2656> 2014-09-18 18:08:34.723320 7f4c9f251700 5 -- op tracker -- , seq: 86173, time: 2014-09-18 18:08:34.719049, event: header_read, op: pg_notify(2.4f6(12) epoch 11713) -2655> 2014-09-18 18:08:34.723389 7f4c9f251700 5 -- op tracker -- , seq: 86173, time: 2014-09-18 18:08:34.719050, event: throttled, op: pg_notify(2.4f6(12) epoch 11713) -2654> 2014-09-18 18:08:34.723400 7f4c9f251700 5 -- op tracker -- , seq: 86173, time: 2014-09-18 18:08:34.719102, event: all_read, op: pg_notify(2.4f6(12) epoch 11713) -2653> 2014-09-18 18:08:34.723410 7f4c9f251700 5 -- op tracker -- , seq: 86173, time: 2014-09-18 18:08:34.723315, event: dispatched, op: pg_notify(2.4f6(12) epoch 11713) -2652> 2014-09-18 18:08:34.723421 7f4c9f251700 5 -- op tracker -- , seq: 86173, time: 2014-09-18 18:08:34.723418, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11713) -2651> 2014-09-18 18:08:34.723433 7f4c9f251700 5 -- op tracker -- , seq: 86173, time: 2014-09-18 18:08:34.723430, event: started, op: pg_notify(2.4f6(12) epoch 11713) -2650> 2014-09-18 18:08:34.723551 7f4c9f251700 5 -- op tracker -- , seq: 86173, time: 2014-09-18 18:08:34.723549, event: done, op: pg_notify(2.4f6(12) epoch 11713) -2649> 2014-09-18 18:08:34.723615 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 942 ==== osd_map(11711..11713 src has 1..11713) v3 ==== 967+0+0 (1213856413 0 0) 0x22e16900 con 0x1395f8c0 -2648> 2014-09-18 18:08:34.723630 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11711,11713], i have 11713, src has [1,11713] -2647> 2014-09-18 18:08:34.723643 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 51 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x22e17d40 con 0x1395f340 -2646> 2014-09-18 18:08:34.723653 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2645> 2014-09-18 18:08:34.723662 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2604 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x1440ad00 con 0x4900b00 -2644> 2014-09-18 18:08:34.723672 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2643> 2014-09-18 18:08:34.723680 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1272 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x11174ac0 con 0xe8db760 -2642> 2014-09-18 18:08:34.723690 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2641> 2014-09-18 18:08:34.723698 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 794 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x2d7fc1c0 con 0xe9a27e0 -2640> 2014-09-18 18:08:34.723709 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2639> 2014-09-18 18:08:34.723716 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1204 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x1182d840 con 0x1088b440 -2638> 2014-09-18 18:08:34.723728 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2637> 2014-09-18 18:08:34.723736 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 943 ==== pg_notify(2.78f(156) epoch 11713) v5 ==== 9386+0+0 (2652298001 0 0) 0xf93d840 con 0x1395f8c0 -2636> 2014-09-18 18:08:34.723749 7f4c9f251700 5 -- op tracker -- , seq: 86174, time: 2014-09-18 18:08:34.721037, event: header_read, op: pg_notify(2.78f(156) epoch 11713) -2635> 2014-09-18 18:08:34.723759 7f4c9f251700 5 -- op tracker -- , seq: 86174, time: 2014-09-18 18:08:34.721038, event: throttled, op: pg_notify(2.78f(156) epoch 11713) -2634> 2014-09-18 18:08:34.723769 7f4c9f251700 5 -- op tracker -- , seq: 86174, time: 2014-09-18 18:08:34.721220, event: all_read, op: pg_notify(2.78f(156) epoch 11713) -2633> 2014-09-18 18:08:34.723779 7f4c9f251700 5 -- op tracker -- , seq: 86174, time: 2014-09-18 18:08:34.723744, event: dispatched, op: pg_notify(2.78f(156) epoch 11713) -2632> 2014-09-18 18:08:34.723790 7f4c9f251700 5 -- op tracker -- , seq: 86174, time: 2014-09-18 18:08:34.723787, event: waiting_for_osdmap, op: pg_notify(2.78f(156) epoch 11713) -2631> 2014-09-18 18:08:34.723801 7f4c9f251700 5 -- op tracker -- , seq: 86174, time: 2014-09-18 18:08:34.723799, event: started, op: pg_notify(2.78f(156) epoch 11713) -2630> 2014-09-18 18:08:34.723829 7f4c9f251700 5 -- op tracker -- , seq: 86174, time: 2014-09-18 18:08:34.723827, event: done, op: pg_notify(2.78f(156) epoch 11713) -2629> 2014-09-18 18:08:34.723869 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 52 ==== pg_notify(2.1d8(9) epoch 11713) v5 ==== 1261+0+0 (561800178 0 0) 0xf975a20 con 0x1395f340 -2628> 2014-09-18 18:08:34.723882 7f4c9f251700 5 -- op tracker -- , seq: 86175, time: 2014-09-18 18:08:34.720842, event: header_read, op: pg_notify(2.1d8(9) epoch 11713) -2627> 2014-09-18 18:08:34.723893 7f4c9f251700 5 -- op tracker -- , seq: 86175, time: 2014-09-18 18:08:34.720843, event: throttled, op: pg_notify(2.1d8(9) epoch 11713) -2626> 2014-09-18 18:08:34.723903 7f4c9f251700 5 -- op tracker -- , seq: 86175, time: 2014-09-18 18:08:34.720898, event: all_read, op: pg_notify(2.1d8(9) epoch 11713) -2625> 2014-09-18 18:08:34.723912 7f4c9f251700 5 -- op tracker -- , seq: 86175, time: 2014-09-18 18:08:34.723878, event: dispatched, op: pg_notify(2.1d8(9) epoch 11713) -2624> 2014-09-18 18:08:34.723923 7f4c9f251700 5 -- op tracker -- , seq: 86175, time: 2014-09-18 18:08:34.723921, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11713) -2623> 2014-09-18 18:08:34.723935 7f4c9f251700 5 -- op tracker -- , seq: 86175, time: 2014-09-18 18:08:34.723933, event: started, op: pg_notify(2.1d8(9) epoch 11713) -2622> 2014-09-18 18:08:34.723954 7f4c9f251700 5 -- op tracker -- , seq: 86175, time: 2014-09-18 18:08:34.723953, event: done, op: pg_notify(2.1d8(9) epoch 11713) -2621> 2014-09-18 18:08:34.723968 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 545 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0xeee9680 con 0x1395ef20 -2620> 2014-09-18 18:08:34.723980 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2619> 2014-09-18 18:08:34.723989 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2605 ==== pg_notify(2.71c(12) epoch 11713) v5 ==== 1424+0+0 (2759380797 0 0) 0xf05c1c0 con 0x4900b00 -2618> 2014-09-18 18:08:34.724001 7f4c9f251700 5 -- op tracker -- , seq: 86176, time: 2014-09-18 18:08:34.721554, event: header_read, op: pg_notify(2.71c(12) epoch 11713) -2617> 2014-09-18 18:08:34.724011 7f4c9f251700 5 -- op tracker -- , seq: 86176, time: 2014-09-18 18:08:34.721555, event: throttled, op: pg_notify(2.71c(12) epoch 11713) -2616> 2014-09-18 18:08:34.724021 7f4c9f251700 5 -- op tracker -- , seq: 86176, time: 2014-09-18 18:08:34.721598, event: all_read, op: pg_notify(2.71c(12) epoch 11713) -2615> 2014-09-18 18:08:34.724031 7f4c9f251700 5 -- op tracker -- , seq: 86176, time: 2014-09-18 18:08:34.723997, event: dispatched, op: pg_notify(2.71c(12) epoch 11713) -2614> 2014-09-18 18:08:34.724041 7f4c9f251700 5 -- op tracker -- , seq: 86176, time: 2014-09-18 18:08:34.724039, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11713) -2613> 2014-09-18 18:08:34.724052 7f4c9f251700 5 -- op tracker -- , seq: 86176, time: 2014-09-18 18:08:34.724050, event: started, op: pg_notify(2.71c(12) epoch 11713) -2612> 2014-09-18 18:08:34.724104 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11713..11713 src has 1..11713) v3 -- ?+0 0x22e16fc0 con 0x1395edc0 -2611> 2014-09-18 18:08:34.724136 7f4c9f251700 5 -- op tracker -- , seq: 86176, time: 2014-09-18 18:08:34.724133, event: done, op: pg_notify(2.71c(12) epoch 11713) -2610> 2014-09-18 18:08:34.724150 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11713) v5 -- ?+0 0xf487fe0 con 0x1395edc0 -2609> 2014-09-18 18:08:34.724154 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1273 ==== pg_notify(2.17e(17) epoch 11713) v5 ==== 1593+0+0 (1893919749 0 0) 0xf0dba40 con 0xe8db760 -2608> 2014-09-18 18:08:34.724167 7f4c9f251700 5 -- op tracker -- , seq: 86177, time: 2014-09-18 18:08:34.722725, event: header_read, op: pg_notify(2.17e(17) epoch 11713) -2607> 2014-09-18 18:08:34.724177 7f4c9f251700 5 -- op tracker -- , seq: 86177, time: 2014-09-18 18:08:34.722726, event: throttled, op: pg_notify(2.17e(17) epoch 11713) -2606> 2014-09-18 18:08:34.724187 7f4c9f251700 5 -- op tracker -- , seq: 86177, time: 2014-09-18 18:08:34.722781, event: all_read, op: pg_notify(2.17e(17) epoch 11713) -2605> 2014-09-18 18:08:34.724197 7f4c9f251700 5 -- op tracker -- , seq: 86177, time: 2014-09-18 18:08:34.724163, event: dispatched, op: pg_notify(2.17e(17) epoch 11713) -2604> 2014-09-18 18:08:34.724167 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11713..11713 src has 1..11713) v3 -- ?+0 0x22e16b40 con 0x1395f1e0 -2603> 2014-09-18 18:08:34.724207 7f4c9f251700 5 -- op tracker -- , seq: 86177, time: 2014-09-18 18:08:34.724204, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11713) -2602> 2014-09-18 18:08:34.724221 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11713) v5 -- ?+0 0xf4890c0 con 0x1395f1e0 -2601> 2014-09-18 18:08:34.724234 7f4c9f251700 5 -- op tracker -- , seq: 86177, time: 2014-09-18 18:08:34.724230, event: started, op: pg_notify(2.17e(17) epoch 11713) -2600> 2014-09-18 18:08:34.724308 7f4c9f251700 5 -- op tracker -- , seq: 86177, time: 2014-09-18 18:08:34.724305, event: done, op: pg_notify(2.17e(17) epoch 11713) -2599> 2014-09-18 18:08:34.724329 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 795 ==== pg_notify(2.2d8(173) epoch 11713) v5 ==== 10757+0+0 (917128075 0 0) 0x142203a0 con 0xe9a27e0 -2598> 2014-09-18 18:08:34.724348 7f4c9f251700 5 -- op tracker -- , seq: 86178, time: 2014-09-18 18:08:34.721805, event: header_read, op: pg_notify(2.2d8(173) epoch 11713) -2597> 2014-09-18 18:08:34.724381 7f4c9f251700 5 -- op tracker -- , seq: 86178, time: 2014-09-18 18:08:34.721806, event: throttled, op: pg_notify(2.2d8(173) epoch 11713) -2596> 2014-09-18 18:08:34.724397 7f4c9f251700 5 -- op tracker -- , seq: 86178, time: 2014-09-18 18:08:34.721995, event: all_read, op: pg_notify(2.2d8(173) epoch 11713) -2595> 2014-09-18 18:08:34.724409 7f4c9f251700 5 -- op tracker -- , seq: 86178, time: 2014-09-18 18:08:34.724343, event: dispatched, op: pg_notify(2.2d8(173) epoch 11713) -2594> 2014-09-18 18:08:34.724420 7f4c9f251700 5 -- op tracker -- , seq: 86178, time: 2014-09-18 18:08:34.724418, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11713) -2593> 2014-09-18 18:08:34.724432 7f4c9f251700 5 -- op tracker -- , seq: 86178, time: 2014-09-18 18:08:34.724430, event: started, op: pg_notify(2.2d8(173) epoch 11713) -2592> 2014-09-18 18:08:34.724432 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11713..11713 src has 1..11713) v3 -- ?+0 0xe339cc0 con 0x140f95a0 -2591> 2014-09-18 18:08:34.724453 7f4c9f251700 5 -- op tracker -- , seq: 86178, time: 2014-09-18 18:08:34.724451, event: done, op: pg_notify(2.2d8(173) epoch 11713) -2590> 2014-09-18 18:08:34.724463 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11713) v5 -- ?+0 0xf0aaf00 con 0x140f95a0 -2589> 2014-09-18 18:08:34.724495 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1205 ==== pg_notify(2.2d8(173) epoch 11713) v5 ==== 10757+0+0 (917128075 0 0) 0xe76e580 con 0x1088b440 -2588> 2014-09-18 18:08:34.724539 7f4c9f251700 5 -- op tracker -- , seq: 86179, time: 2014-09-18 18:08:34.721892, event: header_read, op: pg_notify(2.2d8(173) epoch 11713) -2587> 2014-09-18 18:08:34.724548 7f4c9f251700 5 -- op tracker -- , seq: 86179, time: 2014-09-18 18:08:34.721893, event: throttled, op: pg_notify(2.2d8(173) epoch 11713) -2586> 2014-09-18 18:08:34.724554 7f4c9f251700 5 -- op tracker -- , seq: 86179, time: 2014-09-18 18:08:34.722097, event: all_read, op: pg_notify(2.2d8(173) epoch 11713) -2585> 2014-09-18 18:08:34.724565 7f4c9f251700 5 -- op tracker -- , seq: 86179, time: 2014-09-18 18:08:34.724533, event: dispatched, op: pg_notify(2.2d8(173) epoch 11713) -2584> 2014-09-18 18:08:34.724572 7f4c9f251700 5 -- op tracker -- , seq: 86179, time: 2014-09-18 18:08:34.724570, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11713) -2583> 2014-09-18 18:08:34.724580 7f4c9f251700 5 -- op tracker -- , seq: 86179, time: 2014-09-18 18:08:34.724578, event: started, op: pg_notify(2.2d8(173) epoch 11713) -2582> 2014-09-18 18:08:34.724600 7f4c9f251700 5 -- op tracker -- , seq: 86179, time: 2014-09-18 18:08:34.724597, event: done, op: pg_notify(2.2d8(173) epoch 11713) -2581> 2014-09-18 18:08:34.724639 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3678 ==== pg_notify(2.3b6(10) epoch 11713) v5 ==== 1260+0+0 (1073499423 0 0) 0xe797de0 con 0xcbecec0 -2580> 2014-09-18 18:08:34.724660 7f4c9f251700 5 -- op tracker -- , seq: 86180, time: 2014-09-18 18:08:34.724242, event: header_read, op: pg_notify(2.3b6(10) epoch 11713) -2579> 2014-09-18 18:08:34.724669 7f4c9f251700 5 -- op tracker -- , seq: 86180, time: 2014-09-18 18:08:34.724243, event: throttled, op: pg_notify(2.3b6(10) epoch 11713) -2578> 2014-09-18 18:08:34.724676 7f4c9f251700 5 -- op tracker -- , seq: 86180, time: 2014-09-18 18:08:34.724285, event: all_read, op: pg_notify(2.3b6(10) epoch 11713) -2577> 2014-09-18 18:08:34.724681 7f4c9f251700 5 -- op tracker -- , seq: 86180, time: 2014-09-18 18:08:34.724655, event: dispatched, op: pg_notify(2.3b6(10) epoch 11713) -2576> 2014-09-18 18:08:34.724685 7f4c9f251700 5 -- op tracker -- , seq: 86180, time: 2014-09-18 18:08:34.724684, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11713) -2575> 2014-09-18 18:08:34.724691 7f4c9f251700 5 -- op tracker -- , seq: 86180, time: 2014-09-18 18:08:34.724690, event: started, op: pg_notify(2.3b6(10) epoch 11713) -2574> 2014-09-18 18:08:34.724703 7f4c9f251700 5 -- op tracker -- , seq: 86180, time: 2014-09-18 18:08:34.724702, event: done, op: pg_notify(2.3b6(10) epoch 11713) -2573> 2014-09-18 18:08:34.724714 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 546 ==== pg_notify(2.4d0(10) epoch 11713) v5 ==== 1292+0+0 (1122075700 0 0) 0xf0dcb20 con 0x1395ef20 -2572> 2014-09-18 18:08:34.724727 7f4c9f251700 5 -- op tracker -- , seq: 86181, time: 2014-09-18 18:08:34.723661, event: header_read, op: pg_notify(2.4d0(10) epoch 11713) -2571> 2014-09-18 18:08:34.724732 7f4c9f251700 5 -- op tracker -- , seq: 86181, time: 2014-09-18 18:08:34.723662, event: throttled, op: pg_notify(2.4d0(10) epoch 11713) -2570> 2014-09-18 18:08:34.724736 7f4c9f251700 5 -- op tracker -- , seq: 86181, time: 2014-09-18 18:08:34.723806, event: all_read, op: pg_notify(2.4d0(10) epoch 11713) -2569> 2014-09-18 18:08:34.724743 7f4c9f251700 5 -- op tracker -- , seq: 86181, time: 2014-09-18 18:08:34.724724, event: dispatched, op: pg_notify(2.4d0(10) epoch 11713) -2568> 2014-09-18 18:08:34.724749 7f4c9f251700 5 -- op tracker -- , seq: 86181, time: 2014-09-18 18:08:34.724747, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11713) -2567> 2014-09-18 18:08:34.724754 7f4c9f251700 5 -- op tracker -- , seq: 86181, time: 2014-09-18 18:08:34.724752, event: started, op: pg_notify(2.4d0(10) epoch 11713) -2566> 2014-09-18 18:08:34.724768 7f4c9f251700 5 -- op tracker -- , seq: 86181, time: 2014-09-18 18:08:34.724767, event: done, op: pg_notify(2.4d0(10) epoch 11713) -2565> 2014-09-18 18:08:34.724778 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2019 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x12bf33c0 con 0xe2d9a20 -2564> 2014-09-18 18:08:34.724786 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2563> 2014-09-18 18:08:34.724790 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1048 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0xcc02880 con 0x14358260 -2562> 2014-09-18 18:08:34.724796 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2561> 2014-09-18 18:08:34.724799 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2020 ==== pg_notify(2.26d(9) epoch 11713) v5 ==== 1257+0+0 (1967306639 0 0) 0xf1ebe00 con 0xe2d9a20 -2560> 2014-09-18 18:08:34.724806 7f4c9f251700 5 -- op tracker -- , seq: 86182, time: 2014-09-18 18:08:34.724073, event: header_read, op: pg_notify(2.26d(9) epoch 11713) -2559> 2014-09-18 18:08:34.724810 7f4c9f251700 5 -- op tracker -- , seq: 86182, time: 2014-09-18 18:08:34.724073, event: throttled, op: pg_notify(2.26d(9) epoch 11713) -2558> 2014-09-18 18:08:34.724814 7f4c9f251700 5 -- op tracker -- , seq: 86182, time: 2014-09-18 18:08:34.724143, event: all_read, op: pg_notify(2.26d(9) epoch 11713) -2557> 2014-09-18 18:08:34.724818 7f4c9f251700 5 -- op tracker -- , seq: 86182, time: 2014-09-18 18:08:34.724803, event: dispatched, op: pg_notify(2.26d(9) epoch 11713) -2556> 2014-09-18 18:08:34.724822 7f4c9f251700 5 -- op tracker -- , seq: 86182, time: 2014-09-18 18:08:34.724821, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11713) -2555> 2014-09-18 18:08:34.724828 7f4c9f251700 5 -- op tracker -- , seq: 86182, time: 2014-09-18 18:08:34.724826, event: started, op: pg_notify(2.26d(9) epoch 11713) -2554> 2014-09-18 18:08:34.724839 7f4c9f251700 5 -- op tracker -- , seq: 86182, time: 2014-09-18 18:08:34.724837, event: done, op: pg_notify(2.26d(9) epoch 11713) -2553> 2014-09-18 18:08:34.724846 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1049 ==== pg_notify(2.719(17) epoch 11713) v5 ==== 1601+0+0 (1109521097 0 0) 0xf730580 con 0x14358260 -2552> 2014-09-18 18:08:34.724853 7f4c9f251700 5 -- op tracker -- , seq: 86183, time: 2014-09-18 18:08:34.724655, event: header_read, op: pg_notify(2.719(17) epoch 11713) -2551> 2014-09-18 18:08:34.724857 7f4c9f251700 5 -- op tracker -- , seq: 86183, time: 2014-09-18 18:08:34.724656, event: throttled, op: pg_notify(2.719(17) epoch 11713) -2550> 2014-09-18 18:08:34.724861 7f4c9f251700 5 -- op tracker -- , seq: 86183, time: 2014-09-18 18:08:34.724689, event: all_read, op: pg_notify(2.719(17) epoch 11713) -2549> 2014-09-18 18:08:34.724865 7f4c9f251700 5 -- op tracker -- , seq: 86183, time: 2014-09-18 18:08:34.724851, event: dispatched, op: pg_notify(2.719(17) epoch 11713) -2548> 2014-09-18 18:08:34.724869 7f4c9f251700 5 -- op tracker -- , seq: 86183, time: 2014-09-18 18:08:34.724868, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11713) -2547> 2014-09-18 18:08:34.724880 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11713) v5 -- ?+0 0xf4883a0 con 0x140f95a0 -2546> 2014-09-18 18:08:34.724876 7f4c9f251700 5 -- op tracker -- , seq: 86183, time: 2014-09-18 18:08:34.724874, event: started, op: pg_notify(2.719(17) epoch 11713) -2545> 2014-09-18 18:08:34.724959 7f4c9f251700 5 -- op tracker -- , seq: 86183, time: 2014-09-18 18:08:34.724958, event: done, op: pg_notify(2.719(17) epoch 11713) -2544> 2014-09-18 18:08:34.725134 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2365 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0xe269600 con 0xdd8c940 -2543> 2014-09-18 18:08:34.725146 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2542> 2014-09-18 18:08:34.725226 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2366 ==== pg_notify(2.457(21) epoch 11713) v5 ==== 1875+0+0 (344947721 0 0) 0x19f36780 con 0xdd8c940 -2541> 2014-09-18 18:08:34.725238 7f4c9f251700 5 -- op tracker -- , seq: 86184, time: 2014-09-18 18:08:34.725151, event: header_read, op: pg_notify(2.457(21) epoch 11713) -2540> 2014-09-18 18:08:34.725248 7f4c9f251700 5 -- op tracker -- , seq: 86184, time: 2014-09-18 18:08:34.725152, event: throttled, op: pg_notify(2.457(21) epoch 11713) -2539> 2014-09-18 18:08:34.725255 7f4c9f251700 5 -- op tracker -- , seq: 86184, time: 2014-09-18 18:08:34.725203, event: all_read, op: pg_notify(2.457(21) epoch 11713) -2538> 2014-09-18 18:08:34.725261 7f4c9f251700 5 -- op tracker -- , seq: 86184, time: 2014-09-18 18:08:34.725234, event: dispatched, op: pg_notify(2.457(21) epoch 11713) -2537> 2014-09-18 18:08:34.725269 7f4c9f251700 5 -- op tracker -- , seq: 86184, time: 2014-09-18 18:08:34.725267, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11713) -2536> 2014-09-18 18:08:34.725278 7f4c9f251700 5 -- op tracker -- , seq: 86184, time: 2014-09-18 18:08:34.725276, event: started, op: pg_notify(2.457(21) epoch 11713) -2535> 2014-09-18 18:08:34.725295 7f4c9f251700 5 -- op tracker -- , seq: 86184, time: 2014-09-18 18:08:34.725293, event: done, op: pg_notify(2.457(21) epoch 11713) -2534> 2014-09-18 18:08:34.725409 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] exit Started/Primary/Active/Activating 3.387865 20 0.000984 -2533> 2014-09-18 18:08:34.725529 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] exit Started/Primary/Active 3.388460 0 0.000000 -2532> 2014-09-18 18:08:34.725546 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Started/Primary 5.003673 0 0.000000 -2531> 2014-09-18 18:08:34.725557 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Started 5.003730 0 0.000000 -2530> 2014-09-18 18:08:34.725566 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11708/11708/11258) [27,42,1]/[42,35] r=0 lpr=11708 pi=5697-11707/155 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Reset -2529> 2014-09-18 18:08:34.725623 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000057 1 0.000211 -2528> 2014-09-18 18:08:34.725636 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started -2527> 2014-09-18 18:08:34.725642 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Start -2526> 2014-09-18 18:08:34.725651 7f4c97241700 1 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -2525> 2014-09-18 18:08:34.725661 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000018 0 0.000000 -2524> 2014-09-18 18:08:34.725671 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -2523> 2014-09-18 18:08:34.725680 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -2522> 2014-09-18 18:08:34.725689 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -2521> 2014-09-18 18:08:34.725777 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2367 ==== pg_notify(2.2c7(139) epoch 11713) v5 ==== 8651+0+0 (3458577546 0 0) 0x19f390c0 con 0xdd8c940 -2520> 2014-09-18 18:08:34.725795 7f4c9f251700 5 -- op tracker -- , seq: 86185, time: 2014-09-18 18:08:34.725644, event: header_read, op: pg_notify(2.2c7(139) epoch 11713) -2519> 2014-09-18 18:08:34.725808 7f4c9f251700 5 -- op tracker -- , seq: 86185, time: 2014-09-18 18:08:34.725645, event: throttled, op: pg_notify(2.2c7(139) epoch 11713) -2518> 2014-09-18 18:08:34.725815 7f4c9f251700 5 -- op tracker -- , seq: 86185, time: 2014-09-18 18:08:34.725761, event: all_read, op: pg_notify(2.2c7(139) epoch 11713) -2517> 2014-09-18 18:08:34.725822 7f4c9f251700 5 -- op tracker -- , seq: 86185, time: 2014-09-18 18:08:34.725789, event: dispatched, op: pg_notify(2.2c7(139) epoch 11713) -2516> 2014-09-18 18:08:34.725832 7f4c9f251700 5 -- op tracker -- , seq: 86185, time: 2014-09-18 18:08:34.725830, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11713) -2515> 2014-09-18 18:08:34.725855 7f4c9f251700 5 -- op tracker -- , seq: 86185, time: 2014-09-18 18:08:34.725843, event: started, op: pg_notify(2.2c7(139) epoch 11713) -2514> 2014-09-18 18:08:34.725879 7f4c9f251700 5 -- op tracker -- , seq: 86185, time: 2014-09-18 18:08:34.725876, event: done, op: pg_notify(2.2c7(139) epoch 11713) -2513> 2014-09-18 18:08:34.726107 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -2512> 2014-09-18 18:08:34.726115 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_alive(want up_thru 11713 have 11713) v1 -- ?+0 0x146e01c0 con 0x111771e0 -2511> 2014-09-18 18:08:34.726132 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11711..11713 src has 1..11713) v3 -- ?+0 0x2d7fc1c0 con 0x1395f8c0 -2510> 2014-09-18 18:08:34.726143 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.78f epoch 11713) v3 -- ?+0 0x146de3c0 con 0x1395f8c0 -2509> 2014-09-18 18:08:34.726209 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11713..11713 src has 1..11713) v3 -- ?+0 0xcc00240 con 0xcbecec0 -2508> 2014-09-18 18:08:34.726236 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_query(2.78f epoch 11713) v3 -- ?+0 0x24f78d20 con 0xcbecec0 -2507> 2014-09-18 18:08:34.726286 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11713..11713 src has 1..11713) v3 -- ?+0 0xe3381c0 con 0x1088aec0 -2506> 2014-09-18 18:08:34.726302 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.4e8(8) epoch 11713) v5 -- ?+0 0xf0acd00 con 0x1088aec0 -2505> 2014-09-18 18:08:34.726421 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11713..11713 src has 1..11713) v3 -- ?+0 0x11175a80 con 0x23a74f20 -2504> 2014-09-18 18:08:34.726434 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11713) v5 -- ?+0 0x14184580 con 0x23a74f20 -2503> 2014-09-18 18:08:34.727903 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3679 ==== pg_notify(2.517(11) epoch 11713) v5 ==== 1343+0+0 (2348953787 0 0) 0x24f78d20 con 0xcbecec0 -2502> 2014-09-18 18:08:34.727919 7f4c9f251700 5 -- op tracker -- , seq: 86186, time: 2014-09-18 18:08:34.727849, event: header_read, op: pg_notify(2.517(11) epoch 11713) -2501> 2014-09-18 18:08:34.727939 7f4c9f251700 5 -- op tracker -- , seq: 86186, time: 2014-09-18 18:08:34.727850, event: throttled, op: pg_notify(2.517(11) epoch 11713) -2500> 2014-09-18 18:08:34.727949 7f4c9f251700 5 -- op tracker -- , seq: 86186, time: 2014-09-18 18:08:34.727890, event: all_read, op: pg_notify(2.517(11) epoch 11713) -2499> 2014-09-18 18:08:34.727959 7f4c9f251700 5 -- op tracker -- , seq: 86186, time: 2014-09-18 18:08:34.727915, event: dispatched, op: pg_notify(2.517(11) epoch 11713) -2498> 2014-09-18 18:08:34.727969 7f4c9f251700 5 -- op tracker -- , seq: 86186, time: 2014-09-18 18:08:34.727967, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11713) -2497> 2014-09-18 18:08:34.727977 7f4c9f251700 5 -- op tracker -- , seq: 86186, time: 2014-09-18 18:08:34.727975, event: started, op: pg_notify(2.517(11) epoch 11713) -2496> 2014-09-18 18:08:34.728018 7f4c9f251700 5 -- op tracker -- , seq: 86186, time: 2014-09-18 18:08:34.728015, event: done, op: pg_notify(2.517(11) epoch 11713) -2495> 2014-09-18 18:08:34.728588 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3310 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x10d18400 con 0x4900580 -2494> 2014-09-18 18:08:34.728604 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2493> 2014-09-18 18:08:34.728967 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3311 ==== pg_notify(2.4f6(184) epoch 11713) v5 ==== 11296+0+0 (3014301223 0 0) 0xf244b20 con 0x4900580 -2492> 2014-09-18 18:08:34.728979 7f4c9f251700 5 -- op tracker -- , seq: 86187, time: 2014-09-18 18:08:34.728786, event: header_read, op: pg_notify(2.4f6(184) epoch 11713) -2491> 2014-09-18 18:08:34.728988 7f4c9f251700 5 -- op tracker -- , seq: 86187, time: 2014-09-18 18:08:34.728787, event: throttled, op: pg_notify(2.4f6(184) epoch 11713) -2490> 2014-09-18 18:08:34.728993 7f4c9f251700 5 -- op tracker -- , seq: 86187, time: 2014-09-18 18:08:34.728953, event: all_read, op: pg_notify(2.4f6(184) epoch 11713) -2489> 2014-09-18 18:08:34.728999 7f4c9f251700 5 -- op tracker -- , seq: 86187, time: 2014-09-18 18:08:34.728976, event: dispatched, op: pg_notify(2.4f6(184) epoch 11713) -2488> 2014-09-18 18:08:34.729005 7f4c9f251700 5 -- op tracker -- , seq: 86187, time: 2014-09-18 18:08:34.729004, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11713) -2487> 2014-09-18 18:08:34.729013 7f4c9f251700 5 -- op tracker -- , seq: 86187, time: 2014-09-18 18:08:34.729011, event: started, op: pg_notify(2.4f6(184) epoch 11713) -2486> 2014-09-18 18:08:34.729030 7f4c9f251700 5 -- op tracker -- , seq: 86187, time: 2014-09-18 18:08:34.729028, event: done, op: pg_notify(2.4f6(184) epoch 11713) -2485> 2014-09-18 18:08:34.729528 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3680 ==== pg_notify(2.2c7(139) epoch 11713) v5 ==== 8651+0+0 (261245793 0 0) 0xe7941e0 con 0xcbecec0 -2484> 2014-09-18 18:08:34.729544 7f4c9f251700 5 -- op tracker -- , seq: 86188, time: 2014-09-18 18:08:34.729412, event: header_read, op: pg_notify(2.2c7(139) epoch 11713) -2483> 2014-09-18 18:08:34.729554 7f4c9f251700 5 -- op tracker -- , seq: 86188, time: 2014-09-18 18:08:34.729413, event: throttled, op: pg_notify(2.2c7(139) epoch 11713) -2482> 2014-09-18 18:08:34.729560 7f4c9f251700 5 -- op tracker -- , seq: 86188, time: 2014-09-18 18:08:34.729514, event: all_read, op: pg_notify(2.2c7(139) epoch 11713) -2481> 2014-09-18 18:08:34.729566 7f4c9f251700 5 -- op tracker -- , seq: 86188, time: 2014-09-18 18:08:34.729540, event: dispatched, op: pg_notify(2.2c7(139) epoch 11713) -2480> 2014-09-18 18:08:34.729572 7f4c9f251700 5 -- op tracker -- , seq: 86188, time: 2014-09-18 18:08:34.729571, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11713) -2479> 2014-09-18 18:08:34.729580 7f4c9f251700 5 -- op tracker -- , seq: 86188, time: 2014-09-18 18:08:34.729578, event: started, op: pg_notify(2.2c7(139) epoch 11713) -2478> 2014-09-18 18:08:34.729597 7f4c9f251700 5 -- op tracker -- , seq: 86188, time: 2014-09-18 18:08:34.729595, event: done, op: pg_notify(2.2c7(139) epoch 11713) -2477> 2014-09-18 18:08:34.731547 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3312 ==== pg_notify(2.17e(17) epoch 11713) v5 ==== 1593+0+0 (1893919749 0 0) 0xf243860 con 0x4900580 -2476> 2014-09-18 18:08:34.731563 7f4c9f251700 5 -- op tracker -- , seq: 86189, time: 2014-09-18 18:08:34.731490, event: header_read, op: pg_notify(2.17e(17) epoch 11713) -2475> 2014-09-18 18:08:34.731571 7f4c9f251700 5 -- op tracker -- , seq: 86189, time: 2014-09-18 18:08:34.731491, event: throttled, op: pg_notify(2.17e(17) epoch 11713) -2474> 2014-09-18 18:08:34.731577 7f4c9f251700 5 -- op tracker -- , seq: 86189, time: 2014-09-18 18:08:34.731536, event: all_read, op: pg_notify(2.17e(17) epoch 11713) -2473> 2014-09-18 18:08:34.731588 7f4c9f251700 5 -- op tracker -- , seq: 86189, time: 2014-09-18 18:08:34.731559, event: dispatched, op: pg_notify(2.17e(17) epoch 11713) -2472> 2014-09-18 18:08:34.731596 7f4c9f251700 5 -- op tracker -- , seq: 86189, time: 2014-09-18 18:08:34.731594, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11713) -2471> 2014-09-18 18:08:34.731603 7f4c9f251700 5 -- op tracker -- , seq: 86189, time: 2014-09-18 18:08:34.731602, event: started, op: pg_notify(2.17e(17) epoch 11713) -2470> 2014-09-18 18:08:34.731625 7f4c9f251700 5 -- op tracker -- , seq: 86189, time: 2014-09-18 18:08:34.731622, event: done, op: pg_notify(2.17e(17) epoch 11713) -2469> 2014-09-18 18:08:34.732193 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3313 ==== pg_notify(2.1bb(9) epoch 11713) v5 ==== 1189+0+0 (3194691796 0 0) 0xec6aee0 con 0x4900580 -2468> 2014-09-18 18:08:34.732206 7f4c9f251700 5 -- op tracker -- , seq: 86190, time: 2014-09-18 18:08:34.732143, event: header_read, op: pg_notify(2.1bb(9) epoch 11713) -2467> 2014-09-18 18:08:34.732215 7f4c9f251700 5 -- op tracker -- , seq: 86190, time: 2014-09-18 18:08:34.732144, event: throttled, op: pg_notify(2.1bb(9) epoch 11713) -2466> 2014-09-18 18:08:34.732221 7f4c9f251700 5 -- op tracker -- , seq: 86190, time: 2014-09-18 18:08:34.732181, event: all_read, op: pg_notify(2.1bb(9) epoch 11713) -2465> 2014-09-18 18:08:34.732226 7f4c9f251700 5 -- op tracker -- , seq: 86190, time: 2014-09-18 18:08:34.732203, event: dispatched, op: pg_notify(2.1bb(9) epoch 11713) -2464> 2014-09-18 18:08:34.732233 7f4c9f251700 5 -- op tracker -- , seq: 86190, time: 2014-09-18 18:08:34.732231, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11713) -2463> 2014-09-18 18:08:34.732249 7f4c9f251700 5 -- op tracker -- , seq: 86190, time: 2014-09-18 18:08:34.732248, event: started, op: pg_notify(2.1bb(9) epoch 11713) -2462> 2014-09-18 18:08:34.732266 7f4c9f251700 5 -- op tracker -- , seq: 86190, time: 2014-09-18 18:08:34.732264, event: done, op: pg_notify(2.1bb(9) epoch 11713) -2461> 2014-09-18 18:08:34.735076 7f4caae41700 5 osd.42 11713 tick -2460> 2014-09-18 18:08:34.740138 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 944 ==== pg_notify(2.78f(156) epoch 11713) v5 ==== 9386+0+0 (2652298001 0 0) 0x146de3c0 con 0x1395f8c0 -2459> 2014-09-18 18:08:34.740157 7f4c9f251700 5 -- op tracker -- , seq: 86191, time: 2014-09-18 18:08:34.739985, event: header_read, op: pg_notify(2.78f(156) epoch 11713) -2458> 2014-09-18 18:08:34.740168 7f4c9f251700 5 -- op tracker -- , seq: 86191, time: 2014-09-18 18:08:34.739986, event: throttled, op: pg_notify(2.78f(156) epoch 11713) -2457> 2014-09-18 18:08:34.740174 7f4c9f251700 5 -- op tracker -- , seq: 86191, time: 2014-09-18 18:08:34.740124, event: all_read, op: pg_notify(2.78f(156) epoch 11713) -2456> 2014-09-18 18:08:34.740180 7f4c9f251700 5 -- op tracker -- , seq: 86191, time: 2014-09-18 18:08:34.740153, event: dispatched, op: pg_notify(2.78f(156) epoch 11713) -2455> 2014-09-18 18:08:34.740187 7f4c9f251700 5 -- op tracker -- , seq: 86191, time: 2014-09-18 18:08:34.740186, event: waiting_for_osdmap, op: pg_notify(2.78f(156) epoch 11713) -2454> 2014-09-18 18:08:34.740195 7f4c9f251700 5 -- op tracker -- , seq: 86191, time: 2014-09-18 18:08:34.740194, event: started, op: pg_notify(2.78f(156) epoch 11713) -2453> 2014-09-18 18:08:34.740217 7f4c9f251700 5 -- op tracker -- , seq: 86191, time: 2014-09-18 18:08:34.740214, event: done, op: pg_notify(2.78f(156) epoch 11713) -2452> 2014-09-18 18:08:34.779208 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3235 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x15e306c0 con 0x1395ec60 -2451> 2014-09-18 18:08:34.779227 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2450> 2014-09-18 18:08:34.779234 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3236 ==== pg_notify(2.71(19) epoch 11713) v5 ==== 1795+0+0 (1305526069 0 0) 0xe7954a0 con 0x1395ec60 -2449> 2014-09-18 18:08:34.779242 7f4c9f251700 5 -- op tracker -- , seq: 86192, time: 2014-09-18 18:08:34.779174, event: header_read, op: pg_notify(2.71(19) epoch 11713) -2448> 2014-09-18 18:08:34.779251 7f4c9f251700 5 -- op tracker -- , seq: 86192, time: 2014-09-18 18:08:34.779174, event: throttled, op: pg_notify(2.71(19) epoch 11713) -2447> 2014-09-18 18:08:34.779266 7f4c9f251700 5 -- op tracker -- , seq: 86192, time: 2014-09-18 18:08:34.779222, event: all_read, op: pg_notify(2.71(19) epoch 11713) -2446> 2014-09-18 18:08:34.779275 7f4c9f251700 5 -- op tracker -- , seq: 86192, time: 2014-09-18 18:08:34.779239, event: dispatched, op: pg_notify(2.71(19) epoch 11713) -2445> 2014-09-18 18:08:34.779283 7f4c9f251700 5 -- op tracker -- , seq: 86192, time: 2014-09-18 18:08:34.779282, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11713) -2444> 2014-09-18 18:08:34.779292 7f4c9f251700 5 -- op tracker -- , seq: 86192, time: 2014-09-18 18:08:34.779291, event: started, op: pg_notify(2.71(19) epoch 11713) -2443> 2014-09-18 18:08:34.779311 7f4c9f251700 5 -- op tracker -- , seq: 86192, time: 2014-09-18 18:08:34.779310, event: done, op: pg_notify(2.71(19) epoch 11713) -2442> 2014-09-18 18:08:34.852876 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3681 ==== pg_notify(2.40a(162) epoch 11713) v5 ==== 9736+0+0 (2729008224 0 0) 0xe796760 con 0xcbecec0 -2441> 2014-09-18 18:08:34.852904 7f4c9f251700 5 -- op tracker -- , seq: 86193, time: 2014-09-18 18:08:34.852730, event: header_read, op: pg_notify(2.40a(162) epoch 11713) -2440> 2014-09-18 18:08:34.852912 7f4c9f251700 5 -- op tracker -- , seq: 86193, time: 2014-09-18 18:08:34.852731, event: throttled, op: pg_notify(2.40a(162) epoch 11713) -2439> 2014-09-18 18:08:34.852917 7f4c9f251700 5 -- op tracker -- , seq: 86193, time: 2014-09-18 18:08:34.852861, event: all_read, op: pg_notify(2.40a(162) epoch 11713) -2438> 2014-09-18 18:08:34.852921 7f4c9f251700 5 -- op tracker -- , seq: 86193, time: 2014-09-18 18:08:34.852899, event: dispatched, op: pg_notify(2.40a(162) epoch 11713) -2437> 2014-09-18 18:08:34.852925 7f4c9f251700 5 -- op tracker -- , seq: 86193, time: 2014-09-18 18:08:34.852924, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11713) -2436> 2014-09-18 18:08:34.852937 7f4c9f251700 5 -- op tracker -- , seq: 86193, time: 2014-09-18 18:08:34.852935, event: started, op: pg_notify(2.40a(162) epoch 11713) -2435> 2014-09-18 18:08:34.852967 7f4c9f251700 5 -- op tracker -- , seq: 86193, time: 2014-09-18 18:08:34.852965, event: done, op: pg_notify(2.40a(162) epoch 11713) -2434> 2014-09-18 18:08:34.853904 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3682 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11713) v5 ==== 6174+0+0 (1883018900 0 0) 0xe796b20 con 0xcbecec0 -2433> 2014-09-18 18:08:34.853921 7f4c9f251700 5 -- op tracker -- , seq: 86194, time: 2014-09-18 18:08:34.853818, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11713) -2432> 2014-09-18 18:08:34.853929 7f4c9f251700 5 -- op tracker -- , seq: 86194, time: 2014-09-18 18:08:34.853819, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11713) -2431> 2014-09-18 18:08:34.853934 7f4c9f251700 5 -- op tracker -- , seq: 86194, time: 2014-09-18 18:08:34.853891, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11713) -2430> 2014-09-18 18:08:34.853939 7f4c9f251700 5 -- op tracker -- , seq: 86194, time: 2014-09-18 18:08:34.853917, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11713) -2429> 2014-09-18 18:08:34.853944 7f4c9f251700 5 -- op tracker -- , seq: 86194, time: 2014-09-18 18:08:34.853942, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11713) -2428> 2014-09-18 18:08:34.853950 7f4c9f251700 5 -- op tracker -- , seq: 86194, time: 2014-09-18 18:08:34.853948, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11713) -2427> 2014-09-18 18:08:34.854006 7f4c9f251700 5 -- op tracker -- , seq: 86194, time: 2014-09-18 18:08:34.854004, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11713) -2426> 2014-09-18 18:08:34.858692 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3237 ==== pg_notify(2.519(4) epoch 11713) v5 ==== 962+0+0 (1085491126 0 0) 0xe796940 con 0x1395ec60 -2425> 2014-09-18 18:08:34.858712 7f4c9f251700 5 -- op tracker -- , seq: 86195, time: 2014-09-18 18:08:34.858611, event: header_read, op: pg_notify(2.519(4) epoch 11713) -2424> 2014-09-18 18:08:34.858723 7f4c9f251700 5 -- op tracker -- , seq: 86195, time: 2014-09-18 18:08:34.858613, event: throttled, op: pg_notify(2.519(4) epoch 11713) -2423> 2014-09-18 18:08:34.858732 7f4c9f251700 5 -- op tracker -- , seq: 86195, time: 2014-09-18 18:08:34.858651, event: all_read, op: pg_notify(2.519(4) epoch 11713) -2422> 2014-09-18 18:08:34.858742 7f4c9f251700 5 -- op tracker -- , seq: 86195, time: 2014-09-18 18:08:34.858707, event: dispatched, op: pg_notify(2.519(4) epoch 11713) -2421> 2014-09-18 18:08:34.858753 7f4c9f251700 5 -- op tracker -- , seq: 86195, time: 2014-09-18 18:08:34.858750, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11713) -2420> 2014-09-18 18:08:34.858767 7f4c9f251700 5 -- op tracker -- , seq: 86195, time: 2014-09-18 18:08:34.858764, event: started, op: pg_notify(2.519(4) epoch 11713) -2419> 2014-09-18 18:08:34.858802 7f4c9f251700 5 -- op tracker -- , seq: 86195, time: 2014-09-18 18:08:34.858800, event: done, op: pg_notify(2.519(4) epoch 11713) -2418> 2014-09-18 18:08:34.862332 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3238 ==== pg_notify(2.410(12) epoch 11713) v5 ==== 1376+0+0 (1612153650 0 0) 0xe7952c0 con 0x1395ec60 -2417> 2014-09-18 18:08:34.862352 7f4c9f251700 5 -- op tracker -- , seq: 86196, time: 2014-09-18 18:08:34.862223, event: header_read, op: pg_notify(2.410(12) epoch 11713) -2416> 2014-09-18 18:08:34.862360 7f4c9f251700 5 -- op tracker -- , seq: 86196, time: 2014-09-18 18:08:34.862224, event: throttled, op: pg_notify(2.410(12) epoch 11713) -2415> 2014-09-18 18:08:34.862364 7f4c9f251700 5 -- op tracker -- , seq: 86196, time: 2014-09-18 18:08:34.862294, event: all_read, op: pg_notify(2.410(12) epoch 11713) -2414> 2014-09-18 18:08:34.862368 7f4c9f251700 5 -- op tracker -- , seq: 86196, time: 2014-09-18 18:08:34.862347, event: dispatched, op: pg_notify(2.410(12) epoch 11713) -2413> 2014-09-18 18:08:34.862373 7f4c9f251700 5 -- op tracker -- , seq: 86196, time: 2014-09-18 18:08:34.862372, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11713) -2412> 2014-09-18 18:08:34.862382 7f4c9f251700 5 -- op tracker -- , seq: 86196, time: 2014-09-18 18:08:34.862380, event: started, op: pg_notify(2.410(12) epoch 11713) -2411> 2014-09-18 18:08:34.862400 7f4c9f251700 5 -- op tracker -- , seq: 86196, time: 2014-09-18 18:08:34.862398, event: done, op: pg_notify(2.410(12) epoch 11713) -2410> 2014-09-18 18:08:34.863515 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3239 ==== pg_notify(2.186(12) epoch 11713) v5 ==== 1386+0+0 (1793139549 0 0) 0xe796580 con 0x1395ec60 -2409> 2014-09-18 18:08:34.863534 7f4c9f251700 5 -- op tracker -- , seq: 86197, time: 2014-09-18 18:08:34.863447, event: header_read, op: pg_notify(2.186(12) epoch 11713) -2408> 2014-09-18 18:08:34.863542 7f4c9f251700 5 -- op tracker -- , seq: 86197, time: 2014-09-18 18:08:34.863447, event: throttled, op: pg_notify(2.186(12) epoch 11713) -2407> 2014-09-18 18:08:34.863547 7f4c9f251700 5 -- op tracker -- , seq: 86197, time: 2014-09-18 18:08:34.863480, event: all_read, op: pg_notify(2.186(12) epoch 11713) -2406> 2014-09-18 18:08:34.863553 7f4c9f251700 5 -- op tracker -- , seq: 86197, time: 2014-09-18 18:08:34.863530, event: dispatched, op: pg_notify(2.186(12) epoch 11713) -2405> 2014-09-18 18:08:34.863560 7f4c9f251700 5 -- op tracker -- , seq: 86197, time: 2014-09-18 18:08:34.863559, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11713) -2404> 2014-09-18 18:08:34.863568 7f4c9f251700 5 -- op tracker -- , seq: 86197, time: 2014-09-18 18:08:34.863567, event: started, op: pg_notify(2.186(12) epoch 11713) -2403> 2014-09-18 18:08:34.863586 7f4c9f251700 5 -- op tracker -- , seq: 86197, time: 2014-09-18 18:08:34.863585, event: done, op: pg_notify(2.186(12) epoch 11713) -2402> 2014-09-18 18:08:34.897689 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1283 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x2fba0640 con 0x1f102840 -2401> 2014-09-18 18:08:34.897709 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11711,11711], i have 11713, src has [1,11711] -2400> 2014-09-18 18:08:34.897721 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1284 ==== pg_notify(2.4f6(15) epoch 11711) v5 ==== 1543+0+0 (4230449922 0 0) 0x11b110c0 con 0x1f102840 -2399> 2014-09-18 18:08:34.897730 7f4c9f251700 5 -- op tracker -- , seq: 86198, time: 2014-09-18 18:08:34.897614, event: header_read, op: pg_notify(2.4f6(15) epoch 11711) -2398> 2014-09-18 18:08:34.897740 7f4c9f251700 5 -- op tracker -- , seq: 86198, time: 2014-09-18 18:08:34.897615, event: throttled, op: pg_notify(2.4f6(15) epoch 11711) -2397> 2014-09-18 18:08:34.897748 7f4c9f251700 5 -- op tracker -- , seq: 86198, time: 2014-09-18 18:08:34.897653, event: all_read, op: pg_notify(2.4f6(15) epoch 11711) -2396> 2014-09-18 18:08:34.897754 7f4c9f251700 5 -- op tracker -- , seq: 86198, time: 2014-09-18 18:08:34.897728, event: dispatched, op: pg_notify(2.4f6(15) epoch 11711) -2395> 2014-09-18 18:08:34.897760 7f4c9f251700 5 -- op tracker -- , seq: 86198, time: 2014-09-18 18:08:34.897759, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11711) -2394> 2014-09-18 18:08:34.897768 7f4c9f251700 5 -- op tracker -- , seq: 86198, time: 2014-09-18 18:08:34.897767, event: started, op: pg_notify(2.4f6(15) epoch 11711) -2393> 2014-09-18 18:08:34.897785 7f4c9f251700 5 -- op tracker -- , seq: 86198, time: 2014-09-18 18:08:34.897783, event: done, op: pg_notify(2.4f6(15) epoch 11711) -2392> 2014-09-18 18:08:34.923823 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1285 ==== osd_map(11712..11712 src has 1..11712) v3 ==== 320+0+0 (3723886124 0 0) 0xeee9440 con 0x1f102840 -2391> 2014-09-18 18:08:34.923842 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11712,11712], i have 11713, src has [1,11712] -2390> 2014-09-18 18:08:34.923850 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1286 ==== pg_notify(2.4f6(15) epoch 11712) v5 ==== 1543+0+0 (1499399387 0 0) 0x11b0eb40 con 0x1f102840 -2389> 2014-09-18 18:08:34.923860 7f4c9f251700 5 -- op tracker -- , seq: 86199, time: 2014-09-18 18:08:34.923783, event: header_read, op: pg_notify(2.4f6(15) epoch 11712) -2388> 2014-09-18 18:08:34.923868 7f4c9f251700 5 -- op tracker -- , seq: 86199, time: 2014-09-18 18:08:34.923783, event: throttled, op: pg_notify(2.4f6(15) epoch 11712) -2387> 2014-09-18 18:08:34.923872 7f4c9f251700 5 -- op tracker -- , seq: 86199, time: 2014-09-18 18:08:34.923829, event: all_read, op: pg_notify(2.4f6(15) epoch 11712) -2386> 2014-09-18 18:08:34.923876 7f4c9f251700 5 -- op tracker -- , seq: 86199, time: 2014-09-18 18:08:34.923857, event: dispatched, op: pg_notify(2.4f6(15) epoch 11712) -2385> 2014-09-18 18:08:34.923881 7f4c9f251700 5 -- op tracker -- , seq: 86199, time: 2014-09-18 18:08:34.923879, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11712) -2384> 2014-09-18 18:08:34.923886 7f4c9f251700 5 -- op tracker -- , seq: 86199, time: 2014-09-18 18:08:34.923885, event: started, op: pg_notify(2.4f6(15) epoch 11712) -2383> 2014-09-18 18:08:34.923913 7f4c9f251700 5 -- op tracker -- , seq: 86199, time: 2014-09-18 18:08:34.923911, event: done, op: pg_notify(2.4f6(15) epoch 11712) -2382> 2014-09-18 18:08:34.927306 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1287 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0xeeeb3c0 con 0x1f102840 -2381> 2014-09-18 18:08:34.927324 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2380> 2014-09-18 18:08:34.927344 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1288 ==== pg_notify(2.4f6(15) epoch 11713) v5 ==== 1543+0+0 (20032301 0 0) 0xe1a52a0 con 0x1f102840 -2379> 2014-09-18 18:08:34.927357 7f4c9f251700 5 -- op tracker -- , seq: 86200, time: 2014-09-18 18:08:34.927303, event: header_read, op: pg_notify(2.4f6(15) epoch 11713) -2378> 2014-09-18 18:08:34.927367 7f4c9f251700 5 -- op tracker -- , seq: 86200, time: 2014-09-18 18:08:34.927303, event: throttled, op: pg_notify(2.4f6(15) epoch 11713) -2377> 2014-09-18 18:08:34.927373 7f4c9f251700 5 -- op tracker -- , seq: 86200, time: 2014-09-18 18:08:34.927335, event: all_read, op: pg_notify(2.4f6(15) epoch 11713) -2376> 2014-09-18 18:08:34.927379 7f4c9f251700 5 -- op tracker -- , seq: 86200, time: 2014-09-18 18:08:34.927354, event: dispatched, op: pg_notify(2.4f6(15) epoch 11713) -2375> 2014-09-18 18:08:34.927386 7f4c9f251700 5 -- op tracker -- , seq: 86200, time: 2014-09-18 18:08:34.927385, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11713) -2374> 2014-09-18 18:08:34.927394 7f4c9f251700 5 -- op tracker -- , seq: 86200, time: 2014-09-18 18:08:34.927392, event: started, op: pg_notify(2.4f6(15) epoch 11713) -2373> 2014-09-18 18:08:34.927411 7f4c9f251700 5 -- op tracker -- , seq: 86200, time: 2014-09-18 18:08:34.927409, event: done, op: pg_notify(2.4f6(15) epoch 11713) -2372> 2014-09-18 18:08:34.933731 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3240 ==== pg_notify(2.74e(11) epoch 11713) v5 ==== 1303+0+0 (1237091594 0 0) 0xe7970c0 con 0x1395ec60 -2371> 2014-09-18 18:08:34.933751 7f4c9f251700 5 -- op tracker -- , seq: 86201, time: 2014-09-18 18:08:34.933624, event: header_read, op: pg_notify(2.74e(11) epoch 11713) -2370> 2014-09-18 18:08:34.933761 7f4c9f251700 5 -- op tracker -- , seq: 86201, time: 2014-09-18 18:08:34.933625, event: throttled, op: pg_notify(2.74e(11) epoch 11713) -2369> 2014-09-18 18:08:34.933768 7f4c9f251700 5 -- op tracker -- , seq: 86201, time: 2014-09-18 18:08:34.933685, event: all_read, op: pg_notify(2.74e(11) epoch 11713) -2368> 2014-09-18 18:08:34.933774 7f4c9f251700 5 -- op tracker -- , seq: 86201, time: 2014-09-18 18:08:34.933746, event: dispatched, op: pg_notify(2.74e(11) epoch 11713) -2367> 2014-09-18 18:08:34.933780 7f4c9f251700 5 -- op tracker -- , seq: 86201, time: 2014-09-18 18:08:34.933779, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11713) -2366> 2014-09-18 18:08:34.933789 7f4c9f251700 5 -- op tracker -- , seq: 86201, time: 2014-09-18 18:08:34.933787, event: started, op: pg_notify(2.74e(11) epoch 11713) -2365> 2014-09-18 18:08:34.933808 7f4c9f251700 5 -- op tracker -- , seq: 86201, time: 2014-09-18 18:08:34.933806, event: done, op: pg_notify(2.74e(11) epoch 11713) -2364> 2014-09-18 18:08:34.934821 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3241 ==== pg_notify(2.4b1(13),2.4c1(15),2.78f(54),2.4c6(15) epoch 11713) v5 ==== 8031+0+0 (3723675182 0 0) 0xe7961c0 con 0x1395ec60 -2363> 2014-09-18 18:08:34.934842 7f4c9f251700 5 -- op tracker -- , seq: 86202, time: 2014-09-18 18:08:34.934673, event: header_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(54),2.4c6(15) epoch 11713) -2362> 2014-09-18 18:08:34.934863 7f4c9f251700 5 -- op tracker -- , seq: 86202, time: 2014-09-18 18:08:34.934674, event: throttled, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(54),2.4c6(15) epoch 11713) -2361> 2014-09-18 18:08:34.934870 7f4c9f251700 5 -- op tracker -- , seq: 86202, time: 2014-09-18 18:08:34.934782, event: all_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(54),2.4c6(15) epoch 11713) -2360> 2014-09-18 18:08:34.934877 7f4c9f251700 5 -- op tracker -- , seq: 86202, time: 2014-09-18 18:08:34.934837, event: dispatched, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(54),2.4c6(15) epoch 11713) -2359> 2014-09-18 18:08:34.934884 7f4c9f251700 5 -- op tracker -- , seq: 86202, time: 2014-09-18 18:08:34.934882, event: waiting_for_osdmap, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(54),2.4c6(15) epoch 11713) -2358> 2014-09-18 18:08:34.934892 7f4c9f251700 5 -- op tracker -- , seq: 86202, time: 2014-09-18 18:08:34.934890, event: started, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(54),2.4c6(15) epoch 11713) -2357> 2014-09-18 18:08:34.934940 7f4c9f251700 5 -- op tracker -- , seq: 86202, time: 2014-09-18 18:08:34.934938, event: done, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(54),2.4c6(15) epoch 11713) -2356> 2014-09-18 18:08:34.935778 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3242 ==== pg_notify(2.799(9),2.4d0(10) epoch 11713) v5 ==== 2509+0+0 (286458628 0 0) 0x2b93fe00 con 0x1395ec60 -2355> 2014-09-18 18:08:34.935868 7f4c9f251700 5 -- op tracker -- , seq: 86203, time: 2014-09-18 18:08:34.935693, event: header_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11713) -2354> 2014-09-18 18:08:34.935884 7f4c9f251700 5 -- op tracker -- , seq: 86203, time: 2014-09-18 18:08:34.935697, event: throttled, op: pg_notify(2.799(9),2.4d0(10) epoch 11713) -2353> 2014-09-18 18:08:34.935890 7f4c9f251700 5 -- op tracker -- , seq: 86203, time: 2014-09-18 18:08:34.935741, event: all_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11713) -2352> 2014-09-18 18:08:34.935897 7f4c9f251700 5 -- op tracker -- , seq: 86203, time: 2014-09-18 18:08:34.935863, event: dispatched, op: pg_notify(2.799(9),2.4d0(10) epoch 11713) -2351> 2014-09-18 18:08:34.935904 7f4c9f251700 5 -- op tracker -- , seq: 86203, time: 2014-09-18 18:08:34.935902, event: waiting_for_osdmap, op: pg_notify(2.799(9),2.4d0(10) epoch 11713) -2350> 2014-09-18 18:08:34.935912 7f4c9f251700 5 -- op tracker -- , seq: 86203, time: 2014-09-18 18:08:34.935910, event: started, op: pg_notify(2.799(9),2.4d0(10) epoch 11713) -2349> 2014-09-18 18:08:34.935948 7f4c9f251700 5 -- op tracker -- , seq: 86203, time: 2014-09-18 18:08:34.935946, event: done, op: pg_notify(2.799(9),2.4d0(10) epoch 11713) -2348> 2014-09-18 18:08:35.041022 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3683 ==== pg_notify(2.1bb(9) epoch 11713) v5 ==== 1189+0+0 (316838599 0 0) 0xe797c00 con 0xcbecec0 -2347> 2014-09-18 18:08:35.041058 7f4c9f251700 5 -- op tracker -- , seq: 86204, time: 2014-09-18 18:08:35.040844, event: header_read, op: pg_notify(2.1bb(9) epoch 11713) -2346> 2014-09-18 18:08:35.041078 7f4c9f251700 5 -- op tracker -- , seq: 86204, time: 2014-09-18 18:08:35.040847, event: throttled, op: pg_notify(2.1bb(9) epoch 11713) -2345> 2014-09-18 18:08:35.041086 7f4c9f251700 5 -- op tracker -- , seq: 86204, time: 2014-09-18 18:08:35.040910, event: all_read, op: pg_notify(2.1bb(9) epoch 11713) -2344> 2014-09-18 18:08:35.041094 7f4c9f251700 5 -- op tracker -- , seq: 86204, time: 2014-09-18 18:08:35.041050, event: dispatched, op: pg_notify(2.1bb(9) epoch 11713) -2343> 2014-09-18 18:08:35.041112 7f4c9f251700 5 -- op tracker -- , seq: 86204, time: 2014-09-18 18:08:35.041110, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11713) -2342> 2014-09-18 18:08:35.041126 7f4c9f251700 5 -- op tracker -- , seq: 86204, time: 2014-09-18 18:08:35.041124, event: started, op: pg_notify(2.1bb(9) epoch 11713) -2341> 2014-09-18 18:08:35.041166 7f4c9f251700 5 -- op tracker -- , seq: 86204, time: 2014-09-18 18:08:35.041163, event: done, op: pg_notify(2.1bb(9) epoch 11713) -2340> 2014-09-18 18:08:35.043189 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3684 ==== pg_notify(2.78f(57) epoch 11713) v5 ==== 3773+0+0 (2653966157 0 0) 0xe797a20 con 0xcbecec0 -2339> 2014-09-18 18:08:35.043209 7f4c9f251700 5 -- op tracker -- , seq: 86205, time: 2014-09-18 18:08:35.043033, event: header_read, op: pg_notify(2.78f(57) epoch 11713) -2338> 2014-09-18 18:08:35.043218 7f4c9f251700 5 -- op tracker -- , seq: 86205, time: 2014-09-18 18:08:35.043034, event: throttled, op: pg_notify(2.78f(57) epoch 11713) -2337> 2014-09-18 18:08:35.043222 7f4c9f251700 5 -- op tracker -- , seq: 86205, time: 2014-09-18 18:08:35.043090, event: all_read, op: pg_notify(2.78f(57) epoch 11713) -2336> 2014-09-18 18:08:35.043226 7f4c9f251700 5 -- op tracker -- , seq: 86205, time: 2014-09-18 18:08:35.043205, event: dispatched, op: pg_notify(2.78f(57) epoch 11713) -2335> 2014-09-18 18:08:35.043231 7f4c9f251700 5 -- op tracker -- , seq: 86205, time: 2014-09-18 18:08:35.043230, event: waiting_for_osdmap, op: pg_notify(2.78f(57) epoch 11713) -2334> 2014-09-18 18:08:35.043237 7f4c9f251700 5 -- op tracker -- , seq: 86205, time: 2014-09-18 18:08:35.043235, event: started, op: pg_notify(2.78f(57) epoch 11713) -2333> 2014-09-18 18:08:35.043269 7f4c9f251700 5 -- op tracker -- , seq: 86205, time: 2014-09-18 18:08:35.043267, event: done, op: pg_notify(2.78f(57) epoch 11713) -2332> 2014-09-18 18:08:35.043297 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.317607 5 0.000209 -2331> 2014-09-18 18:08:35.043431 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -2330> 2014-09-18 18:08:35.043480 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000049 0 0.000000 -2329> 2014-09-18 18:08:35.043495 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -2328> 2014-09-18 18:08:35.043509 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000014 0 0.000000 -2327> 2014-09-18 18:08:35.043521 7f4c97241700 5 osd.42 pg_epoch: 11713 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -2326> 2014-09-18 18:08:35.045581 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3685 ==== pg_notify(2.78f(57) epoch 11713) v5 ==== 3773+0+0 (2653966157 0 0) 0xe794960 con 0xcbecec0 -2325> 2014-09-18 18:08:35.045600 7f4c9f251700 5 -- op tracker -- , seq: 86206, time: 2014-09-18 18:08:35.045438, event: header_read, op: pg_notify(2.78f(57) epoch 11713) -2324> 2014-09-18 18:08:35.045611 7f4c9f251700 5 -- op tracker -- , seq: 86206, time: 2014-09-18 18:08:35.045439, event: throttled, op: pg_notify(2.78f(57) epoch 11713) -2323> 2014-09-18 18:08:35.045617 7f4c9f251700 5 -- op tracker -- , seq: 86206, time: 2014-09-18 18:08:35.045552, event: all_read, op: pg_notify(2.78f(57) epoch 11713) -2322> 2014-09-18 18:08:35.045623 7f4c9f251700 5 -- op tracker -- , seq: 86206, time: 2014-09-18 18:08:35.045596, event: dispatched, op: pg_notify(2.78f(57) epoch 11713) -2321> 2014-09-18 18:08:35.045630 7f4c9f251700 5 -- op tracker -- , seq: 86206, time: 2014-09-18 18:08:35.045628, event: waiting_for_osdmap, op: pg_notify(2.78f(57) epoch 11713) -2320> 2014-09-18 18:08:35.045647 7f4c9f251700 5 -- op tracker -- , seq: 86206, time: 2014-09-18 18:08:35.045645, event: started, op: pg_notify(2.78f(57) epoch 11713) -2319> 2014-09-18 18:08:35.045664 7f4c9f251700 5 -- op tracker -- , seq: 86206, time: 2014-09-18 18:08:35.045663, event: done, op: pg_notify(2.78f(57) epoch 11713) -2318> 2014-09-18 18:08:35.251051 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5110 ==== pg_notify(2.40a(162) epoch 11710) v5 ==== 9736+0+0 (3880184790 0 0) 0xf0acd00 con 0x1088aec0 -2317> 2014-09-18 18:08:35.251078 7f4c9f251700 5 -- op tracker -- , seq: 86207, time: 2014-09-18 18:08:35.250851, event: header_read, op: pg_notify(2.40a(162) epoch 11710) -2316> 2014-09-18 18:08:35.251090 7f4c9f251700 5 -- op tracker -- , seq: 86207, time: 2014-09-18 18:08:35.250856, event: throttled, op: pg_notify(2.40a(162) epoch 11710) -2315> 2014-09-18 18:08:35.251097 7f4c9f251700 5 -- op tracker -- , seq: 86207, time: 2014-09-18 18:08:35.251030, event: all_read, op: pg_notify(2.40a(162) epoch 11710) -2314> 2014-09-18 18:08:35.251103 7f4c9f251700 5 -- op tracker -- , seq: 86207, time: 2014-09-18 18:08:35.251072, event: dispatched, op: pg_notify(2.40a(162) epoch 11710) -2313> 2014-09-18 18:08:35.251108 7f4c9f251700 5 -- op tracker -- , seq: 86207, time: 2014-09-18 18:08:35.251107, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11710) -2312> 2014-09-18 18:08:35.251117 7f4c9f251700 5 -- op tracker -- , seq: 86207, time: 2014-09-18 18:08:35.251116, event: started, op: pg_notify(2.40a(162) epoch 11710) -2311> 2014-09-18 18:08:35.251141 7f4c9f251700 5 -- op tracker -- , seq: 86207, time: 2014-09-18 18:08:35.251139, event: done, op: pg_notify(2.40a(162) epoch 11710) -2310> 2014-09-18 18:08:35.251202 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5111 ==== pg_notify(2.11b(8),2.120(11) epoch 11710) v5 ==== 2507+0+0 (771233451 0 0) 0xddf5c00 con 0x1088aec0 -2309> 2014-09-18 18:08:35.251214 7f4c9f251700 5 -- op tracker -- , seq: 86208, time: 2014-09-18 18:08:35.251048, event: header_read, op: pg_notify(2.11b(8),2.120(11) epoch 11710) -2308> 2014-09-18 18:08:35.251221 7f4c9f251700 5 -- op tracker -- , seq: 86208, time: 2014-09-18 18:08:35.251048, event: throttled, op: pg_notify(2.11b(8),2.120(11) epoch 11710) -2307> 2014-09-18 18:08:35.251228 7f4c9f251700 5 -- op tracker -- , seq: 86208, time: 2014-09-18 18:08:35.251095, event: all_read, op: pg_notify(2.11b(8),2.120(11) epoch 11710) -2306> 2014-09-18 18:08:35.251234 7f4c9f251700 5 -- op tracker -- , seq: 86208, time: 2014-09-18 18:08:35.251211, event: dispatched, op: pg_notify(2.11b(8),2.120(11) epoch 11710) -2305> 2014-09-18 18:08:35.251241 7f4c9f251700 5 -- op tracker -- , seq: 86208, time: 2014-09-18 18:08:35.251239, event: waiting_for_osdmap, op: pg_notify(2.11b(8),2.120(11) epoch 11710) -2304> 2014-09-18 18:08:35.251249 7f4c9f251700 5 -- op tracker -- , seq: 86208, time: 2014-09-18 18:08:35.251247, event: started, op: pg_notify(2.11b(8),2.120(11) epoch 11710) -2303> 2014-09-18 18:08:35.251276 7f4c9f251700 5 -- op tracker -- , seq: 86208, time: 2014-09-18 18:08:35.251275, event: done, op: pg_notify(2.11b(8),2.120(11) epoch 11710) -2302> 2014-09-18 18:08:35.251672 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5112 ==== pg_notify(2.179(10) epoch 11710) v5 ==== 1276+0+0 (2566695766 0 0) 0xddf32c0 con 0x1088aec0 -2301> 2014-09-18 18:08:35.251692 7f4c9f251700 5 -- op tracker -- , seq: 86209, time: 2014-09-18 18:08:35.251592, event: header_read, op: pg_notify(2.179(10) epoch 11710) -2300> 2014-09-18 18:08:35.251702 7f4c9f251700 5 -- op tracker -- , seq: 86209, time: 2014-09-18 18:08:35.251593, event: throttled, op: pg_notify(2.179(10) epoch 11710) -2299> 2014-09-18 18:08:35.251709 7f4c9f251700 5 -- op tracker -- , seq: 86209, time: 2014-09-18 18:08:35.251630, event: all_read, op: pg_notify(2.179(10) epoch 11710) -2298> 2014-09-18 18:08:35.251714 7f4c9f251700 5 -- op tracker -- , seq: 86209, time: 2014-09-18 18:08:35.251687, event: dispatched, op: pg_notify(2.179(10) epoch 11710) -2297> 2014-09-18 18:08:35.251721 7f4c9f251700 5 -- op tracker -- , seq: 86209, time: 2014-09-18 18:08:35.251719, event: waiting_for_osdmap, op: pg_notify(2.179(10) epoch 11710) -2296> 2014-09-18 18:08:35.251729 7f4c9f251700 5 -- op tracker -- , seq: 86209, time: 2014-09-18 18:08:35.251727, event: started, op: pg_notify(2.179(10) epoch 11710) -2295> 2014-09-18 18:08:35.251747 7f4c9f251700 5 -- op tracker -- , seq: 86209, time: 2014-09-18 18:08:35.251745, event: done, op: pg_notify(2.179(10) epoch 11710) -2294> 2014-09-18 18:08:35.251846 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5113 ==== pg_notify(2.410(12) epoch 11710) v5 ==== 1338+0+0 (1510378147 0 0) 0xddf5660 con 0x1088aec0 -2293> 2014-09-18 18:08:35.251858 7f4c9f251700 5 -- op tracker -- , seq: 86210, time: 2014-09-18 18:08:35.251800, event: header_read, op: pg_notify(2.410(12) epoch 11710) -2292> 2014-09-18 18:08:35.251875 7f4c9f251700 5 -- op tracker -- , seq: 86210, time: 2014-09-18 18:08:35.251800, event: throttled, op: pg_notify(2.410(12) epoch 11710) -2291> 2014-09-18 18:08:35.251881 7f4c9f251700 5 -- op tracker -- , seq: 86210, time: 2014-09-18 18:08:35.251835, event: all_read, op: pg_notify(2.410(12) epoch 11710) -2290> 2014-09-18 18:08:35.251887 7f4c9f251700 5 -- op tracker -- , seq: 86210, time: 2014-09-18 18:08:35.251855, event: dispatched, op: pg_notify(2.410(12) epoch 11710) -2289> 2014-09-18 18:08:35.251898 7f4c9f251700 5 -- op tracker -- , seq: 86210, time: 2014-09-18 18:08:35.251892, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11710) -2288> 2014-09-18 18:08:35.251906 7f4c9f251700 5 -- op tracker -- , seq: 86210, time: 2014-09-18 18:08:35.251905, event: started, op: pg_notify(2.410(12) epoch 11710) -2287> 2014-09-18 18:08:35.252041 7f4c9f251700 5 -- op tracker -- , seq: 86210, time: 2014-09-18 18:08:35.252039, event: done, op: pg_notify(2.410(12) epoch 11710) -2286> 2014-09-18 18:08:35.252932 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5114 ==== pg_notify(2.1a2(20) epoch 11710) v5 ==== 1798+0+0 (2262187663 0 0) 0xddf34a0 con 0x1088aec0 -2285> 2014-09-18 18:08:35.252961 7f4c9f251700 5 -- op tracker -- , seq: 86211, time: 2014-09-18 18:08:35.252872, event: header_read, op: pg_notify(2.1a2(20) epoch 11710) -2284> 2014-09-18 18:08:35.252973 7f4c9f251700 5 -- op tracker -- , seq: 86211, time: 2014-09-18 18:08:35.252874, event: throttled, op: pg_notify(2.1a2(20) epoch 11710) -2283> 2014-09-18 18:08:35.252980 7f4c9f251700 5 -- op tracker -- , seq: 86211, time: 2014-09-18 18:08:35.252916, event: all_read, op: pg_notify(2.1a2(20) epoch 11710) -2282> 2014-09-18 18:08:35.252986 7f4c9f251700 5 -- op tracker -- , seq: 86211, time: 2014-09-18 18:08:35.252948, event: dispatched, op: pg_notify(2.1a2(20) epoch 11710) -2281> 2014-09-18 18:08:35.252991 7f4c9f251700 5 -- op tracker -- , seq: 86211, time: 2014-09-18 18:08:35.252990, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11710) -2280> 2014-09-18 18:08:35.252999 7f4c9f251700 5 -- op tracker -- , seq: 86211, time: 2014-09-18 18:08:35.252998, event: started, op: pg_notify(2.1a2(20) epoch 11710) -2279> 2014-09-18 18:08:35.253103 7f4c9f251700 5 -- op tracker -- , seq: 86211, time: 2014-09-18 18:08:35.253101, event: done, op: pg_notify(2.1a2(20) epoch 11710) -2278> 2014-09-18 18:08:35.254332 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5115 ==== pg_notify(2.4c1(15) epoch 11710) v5 ==== 1507+0+0 (1706917014 0 0) 0x21cecb20 con 0x1088aec0 -2277> 2014-09-18 18:08:35.254346 7f4c9f251700 5 -- op tracker -- , seq: 86212, time: 2014-09-18 18:08:35.254264, event: header_read, op: pg_notify(2.4c1(15) epoch 11710) -2276> 2014-09-18 18:08:35.254355 7f4c9f251700 5 -- op tracker -- , seq: 86212, time: 2014-09-18 18:08:35.254266, event: throttled, op: pg_notify(2.4c1(15) epoch 11710) -2275> 2014-09-18 18:08:35.254361 7f4c9f251700 5 -- op tracker -- , seq: 86212, time: 2014-09-18 18:08:35.254319, event: all_read, op: pg_notify(2.4c1(15) epoch 11710) -2274> 2014-09-18 18:08:35.254367 7f4c9f251700 5 -- op tracker -- , seq: 86212, time: 2014-09-18 18:08:35.254343, event: dispatched, op: pg_notify(2.4c1(15) epoch 11710) -2273> 2014-09-18 18:08:35.254373 7f4c9f251700 5 -- op tracker -- , seq: 86212, time: 2014-09-18 18:08:35.254372, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11710) -2272> 2014-09-18 18:08:35.254381 7f4c9f251700 5 -- op tracker -- , seq: 86212, time: 2014-09-18 18:08:35.254379, event: started, op: pg_notify(2.4c1(15) epoch 11710) -2271> 2014-09-18 18:08:35.254399 7f4c9f251700 5 -- op tracker -- , seq: 86212, time: 2014-09-18 18:08:35.254397, event: done, op: pg_notify(2.4c1(15) epoch 11710) -2270> 2014-09-18 18:08:35.282651 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1734 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x14409440 con 0x4900000 -2269> 2014-09-18 18:08:35.282678 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2268> 2014-09-18 18:08:35.282759 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1735 ==== pg_notify(2.e0(79) epoch 11713) v5 ==== 5119+0+0 (3299020732 0 0) 0xf1ead20 con 0x4900000 -2267> 2014-09-18 18:08:35.282772 7f4c9f251700 5 -- op tracker -- , seq: 86213, time: 2014-09-18 18:08:35.282655, event: header_read, op: pg_notify(2.e0(79) epoch 11713) -2266> 2014-09-18 18:08:35.282782 7f4c9f251700 5 -- op tracker -- , seq: 86213, time: 2014-09-18 18:08:35.282656, event: throttled, op: pg_notify(2.e0(79) epoch 11713) -2265> 2014-09-18 18:08:35.282791 7f4c9f251700 5 -- op tracker -- , seq: 86213, time: 2014-09-18 18:08:35.282746, event: all_read, op: pg_notify(2.e0(79) epoch 11713) -2264> 2014-09-18 18:08:35.282799 7f4c9f251700 5 -- op tracker -- , seq: 86213, time: 2014-09-18 18:08:35.282768, event: dispatched, op: pg_notify(2.e0(79) epoch 11713) -2263> 2014-09-18 18:08:35.282807 7f4c9f251700 5 -- op tracker -- , seq: 86213, time: 2014-09-18 18:08:35.282805, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11713) -2262> 2014-09-18 18:08:35.282816 7f4c9f251700 5 -- op tracker -- , seq: 86213, time: 2014-09-18 18:08:35.282814, event: started, op: pg_notify(2.e0(79) epoch 11713) -2261> 2014-09-18 18:08:35.282836 7f4c9f251700 5 -- op tracker -- , seq: 86213, time: 2014-09-18 18:08:35.282835, event: done, op: pg_notify(2.e0(79) epoch 11713) -2260> 2014-09-18 18:08:35.408864 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.27 10.242.42.174:0/21548 150 ==== osd_ping(ping e11713 stamp 2014-09-18 18:06:41.239537) v2 ==== 47+0+0 (2374766289 0 0) 0x146e1a20 con 0x107358c0 -2259> 2014-09-18 18:08:35.408913 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.27 10.242.42.174:0/21548 150 ==== osd_ping(ping e11713 stamp 2014-09-18 18:06:41.239537) v2 ==== 47+0+0 (2374766289 0 0) 0xf2723c0 con 0x101cb4a0 -2258> 2014-09-18 18:08:35.408923 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.174:0/21548 -- osd_ping(ping_reply e11713 stamp 2014-09-18 18:06:41.239537) v2 -- ?+0 0xfae7e00 con 0x107358c0 -2257> 2014-09-18 18:08:35.408966 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.174:0/21548 -- osd_ping(ping_reply e11713 stamp 2014-09-18 18:06:41.239537) v2 -- ?+0 0x2b989840 con 0x101cb4a0 -2256> 2014-09-18 18:08:35.453987 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.39 10.242.42.172:0/2660 1590 ==== osd_ping(ping e11713 stamp 2014-09-18 23:37:24.558158) v2 ==== 47+0+0 (3363720892 0 0) 0x1393c580 con 0x4750840 -2255> 2014-09-18 18:08:35.454017 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11713 stamp 2014-09-18 23:37:24.558158) v2 -- ?+0 0xf2723c0 con 0x4750840 -2254> 2014-09-18 18:08:35.454031 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.39 10.242.42.172:0/2660 1590 ==== osd_ping(ping e11713 stamp 2014-09-18 23:37:24.558158) v2 ==== 47+0+0 (3363720892 0 0) 0x11b11480 con 0xec39340 -2253> 2014-09-18 18:08:35.454055 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/2660 -- osd_ping(ping_reply e11713 stamp 2014-09-18 23:37:24.558158) v2 -- ?+0 0x146e1a20 con 0xec39340 -2252> 2014-09-18 18:08:35.584467 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.56 10.242.42.172:0/36458 654 ==== osd_ping(ping e11713 stamp 2014-09-18 23:37:24.688617) v2 ==== 47+0+0 (2895887150 0 0) 0xf0e21e0 con 0x14392c60 -2251> 2014-09-18 18:08:35.584524 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/36458 -- osd_ping(ping_reply e11713 stamp 2014-09-18 23:37:24.688617) v2 -- ?+0 0x1393c580 con 0x14392c60 -2250> 2014-09-18 18:08:35.584925 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.56 10.242.42.172:0/36458 654 ==== osd_ping(ping e11713 stamp 2014-09-18 23:37:24.688617) v2 ==== 47+0+0 (2895887150 0 0) 0x21d15840 con 0x10a81700 -2249> 2014-09-18 18:08:35.584953 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/36458 -- osd_ping(ping_reply e11713 stamp 2014-09-18 23:37:24.688617) v2 -- ?+0 0x11b11480 con 0x10a81700 -2248> 2014-09-18 18:08:35.621423 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 348 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0xe9381c0 con 0x12562260 -2247> 2014-09-18 18:08:35.621454 7f4c9f251700 3 osd.42 11713 handle_osd_map epochs [11713,11713], i have 11713, src has [1,11713] -2246> 2014-09-18 18:08:35.621463 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 349 ==== pg_notify(2.719(17) epoch 11713) v5 ==== 1601+0+0 (222628274 0 0) 0x1173bde0 con 0x12562260 -2245> 2014-09-18 18:08:35.621475 7f4c9f251700 5 -- op tracker -- , seq: 86214, time: 2014-09-18 18:08:35.621365, event: header_read, op: pg_notify(2.719(17) epoch 11713) -2244> 2014-09-18 18:08:35.621486 7f4c9f251700 5 -- op tracker -- , seq: 86214, time: 2014-09-18 18:08:35.621366, event: throttled, op: pg_notify(2.719(17) epoch 11713) -2243> 2014-09-18 18:08:35.621493 7f4c9f251700 5 -- op tracker -- , seq: 86214, time: 2014-09-18 18:08:35.621410, event: all_read, op: pg_notify(2.719(17) epoch 11713) -2242> 2014-09-18 18:08:35.621499 7f4c9f251700 5 -- op tracker -- , seq: 86214, time: 2014-09-18 18:08:35.621471, event: dispatched, op: pg_notify(2.719(17) epoch 11713) -2241> 2014-09-18 18:08:35.621505 7f4c9f251700 5 -- op tracker -- , seq: 86214, time: 2014-09-18 18:08:35.621504, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11713) -2240> 2014-09-18 18:08:35.621514 7f4c9f251700 5 -- op tracker -- , seq: 86214, time: 2014-09-18 18:08:35.621513, event: started, op: pg_notify(2.719(17) epoch 11713) -2239> 2014-09-18 18:08:35.621537 7f4c9f251700 5 -- op tracker -- , seq: 86214, time: 2014-09-18 18:08:35.621536, event: done, op: pg_notify(2.719(17) epoch 11713) -2238> 2014-09-18 18:08:35.622358 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.52 10.242.42.168:0/16848 131 ==== osd_ping(ping e11713 stamp 2014-09-18 18:08:35.489020) v2 ==== 47+0+0 (3702801644 0 0) 0x2b929840 con 0xcc49b80 -2237> 2014-09-18 18:08:35.622397 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.52 10.242.42.168:0/16848 131 ==== osd_ping(ping e11713 stamp 2014-09-18 18:08:35.489020) v2 ==== 47+0+0 (3702801644 0 0) 0x1213b480 con 0x101cc520 -2236> 2014-09-18 18:08:35.622383 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16848 -- osd_ping(ping_reply e11713 stamp 2014-09-18 18:08:35.489020) v2 -- ?+0 0x21d15840 con 0xcc49b80 -2235> 2014-09-18 18:08:35.622459 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16848 -- osd_ping(ping_reply e11713 stamp 2014-09-18 18:08:35.489020) v2 -- ?+0 0xf0e21e0 con 0x101cc520 -2234> 2014-09-18 18:08:35.644499 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.37 10.242.42.168:0/18319 108 ==== osd_ping(ping e11713 stamp 2014-09-18 18:08:35.511085) v2 ==== 47+0+0 (368683894 0 0) 0x11fd2ee0 con 0x110bb860 -2233> 2014-09-18 18:08:35.644522 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.37 10.242.42.168:0/18319 108 ==== osd_ping(ping e11713 stamp 2014-09-18 18:08:35.511085) v2 ==== 47+0+0 (368683894 0 0) 0x1421ed20 con 0x11ffdce0 -2232> 2014-09-18 18:08:35.644533 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11713 stamp 2014-09-18 18:08:35.511085) v2 -- ?+0 0x2b929840 con 0x110bb860 -2231> 2014-09-18 18:08:35.644563 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11713 stamp 2014-09-18 18:08:35.511085) v2 -- ?+0 0x1213b480 con 0x11ffdce0 -2230> 2014-09-18 18:08:35.678354 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 32 ==== osd_map(11712..11714 src has 1..11714) v3 ==== 1888+0+0 (4177877393 0 0) 0x15e32f40 con 0x111771e0 -2229> 2014-09-18 18:08:35.678447 7f4ca0a54700 3 osd.42 11713 handle_osd_map epochs [11712,11714], i have 11713, src has [1,11714] -2228> 2014-09-18 18:08:35.681575 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1050 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0xcc02640 con 0x14358260 -2227> 2014-09-18 18:08:35.682453 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11714..11714 src has 1..11714) v3 -- ?+0 0x11175600 con 0x1395edc0 -2226> 2014-09-18 18:08:35.682530 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11714) v5 -- ?+0 0x141821e0 con 0x1395edc0 -2225> 2014-09-18 18:08:35.683223 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2224> 2014-09-18 18:08:35.683248 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1274 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0xeee9200 con 0xe8db760 -2223> 2014-09-18 18:08:35.683263 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2222> 2014-09-18 18:08:35.683270 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 796 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x2d7fa480 con 0xe9a27e0 -2221> 2014-09-18 18:08:35.683279 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2220> 2014-09-18 18:08:35.683284 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1051 ==== pg_notify(2.719(17) epoch 11714) v5 ==== 1601+0+0 (1974787217 0 0) 0xf7301c0 con 0x14358260 -2219> 2014-09-18 18:08:35.683291 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11714..11714 src has 1..11714) v3 -- ?+0 0xe338880 con 0x140f95a0 -2218> 2014-09-18 18:08:35.683297 7f4c9f251700 5 -- op tracker -- , seq: 86215, time: 2014-09-18 18:08:35.681565, event: header_read, op: pg_notify(2.719(17) epoch 11714) -2217> 2014-09-18 18:08:35.683326 7f4c9f251700 5 -- op tracker -- , seq: 86215, time: 2014-09-18 18:08:35.681566, event: throttled, op: pg_notify(2.719(17) epoch 11714) -2216> 2014-09-18 18:08:35.683338 7f4c9f251700 5 -- op tracker -- , seq: 86215, time: 2014-09-18 18:08:35.681617, event: all_read, op: pg_notify(2.719(17) epoch 11714) -2215> 2014-09-18 18:08:35.683345 7f4c9f251700 5 -- op tracker -- , seq: 86215, time: 2014-09-18 18:08:35.683292, event: dispatched, op: pg_notify(2.719(17) epoch 11714) -2214> 2014-09-18 18:08:35.683352 7f4c9f251700 5 -- op tracker -- , seq: 86215, time: 2014-09-18 18:08:35.683350, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11714) -2213> 2014-09-18 18:08:35.683361 7f4c9f251700 5 -- op tracker -- , seq: 86215, time: 2014-09-18 18:08:35.683360, event: started, op: pg_notify(2.719(17) epoch 11714) -2212> 2014-09-18 18:08:35.683381 7f4c9f251700 5 -- op tracker -- , seq: 86215, time: 2014-09-18 18:08:35.683379, event: done, op: pg_notify(2.719(17) epoch 11714) -2211> 2014-09-18 18:08:35.683399 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(10) epoch 11714) v5 -- ?+0 0xe795680 con 0x140f95a0 -2210> 2014-09-18 18:08:35.683954 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(13) epoch 11714) v5 -- ?+0 0xe2870e0 con 0x140f95a0 -2209> 2014-09-18 18:08:35.684383 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.640863 3 0.000260 -2208> 2014-09-18 18:08:35.684404 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.958724 0 0.000000 -2207> 2014-09-18 18:08:35.684416 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Started/Primary 0.958744 0 0.000000 -2206> 2014-09-18 18:08:35.684426 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Started 0.958790 0 0.000000 -2205> 2014-09-18 18:08:35.684436 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11713/11713/11258) [27,42]/[42,35] r=0 lpr=11713 pi=5697-11712/156 bft=27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Reset -2204> 2014-09-18 18:08:35.684474 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000037 1 0.000091 -2203> 2014-09-18 18:08:35.684486 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started -2202> 2014-09-18 18:08:35.684523 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Start -2201> 2014-09-18 18:08:35.684603 7f4c97241700 1 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -2200> 2014-09-18 18:08:35.684614 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000091 0 0.000000 -2199> 2014-09-18 18:08:35.684624 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -2198> 2014-09-18 18:08:35.684633 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -2197> 2014-09-18 18:08:35.684642 7f4c97241700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -2196> 2014-09-18 18:08:35.683399 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 945 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x2d7fc1c0 con 0x1395f8c0 -2195> 2014-09-18 18:08:35.684794 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2194> 2014-09-18 18:08:35.684806 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 53 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x22e178c0 con 0x1395f340 -2193> 2014-09-18 18:08:35.684839 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2192> 2014-09-18 18:08:35.684850 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1275 ==== pg_notify(2.17e(17) epoch 11714) v5 ==== 1593+0+0 (1420404463 0 0) 0xf0dad20 con 0xe8db760 -2191> 2014-09-18 18:08:35.684865 7f4c9f251700 5 -- op tracker -- , seq: 86216, time: 2014-09-18 18:08:35.681590, event: header_read, op: pg_notify(2.17e(17) epoch 11714) -2190> 2014-09-18 18:08:35.684878 7f4c9f251700 5 -- op tracker -- , seq: 86216, time: 2014-09-18 18:08:35.681591, event: throttled, op: pg_notify(2.17e(17) epoch 11714) -2189> 2014-09-18 18:08:35.684887 7f4c9f251700 5 -- op tracker -- , seq: 86216, time: 2014-09-18 18:08:35.681659, event: all_read, op: pg_notify(2.17e(17) epoch 11714) -2188> 2014-09-18 18:08:35.684893 7f4c9f251700 5 -- op tracker -- , seq: 86216, time: 2014-09-18 18:08:35.684860, event: dispatched, op: pg_notify(2.17e(17) epoch 11714) -2187> 2014-09-18 18:08:35.684903 7f4c9f251700 5 -- op tracker -- , seq: 86216, time: 2014-09-18 18:08:35.684900, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11714) -2186> 2014-09-18 18:08:35.684913 7f4c9f251700 5 -- op tracker -- , seq: 86216, time: 2014-09-18 18:08:35.684911, event: started, op: pg_notify(2.17e(17) epoch 11714) -2185> 2014-09-18 18:08:35.684951 7f4c9f251700 5 -- op tracker -- , seq: 86216, time: 2014-09-18 18:08:35.684948, event: done, op: pg_notify(2.17e(17) epoch 11714) -2184> 2014-09-18 18:08:35.684973 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 797 ==== pg_notify(2.2d8(173) epoch 11714) v5 ==== 10757+0+0 (4135927104 0 0) 0x14221a20 con 0xe9a27e0 -2183> 2014-09-18 18:08:35.684991 7f4c9f251700 5 -- op tracker -- , seq: 86217, time: 2014-09-18 18:08:35.682521, event: header_read, op: pg_notify(2.2d8(173) epoch 11714) -2182> 2014-09-18 18:08:35.684999 7f4c9f251700 5 -- op tracker -- , seq: 86217, time: 2014-09-18 18:08:35.682522, event: throttled, op: pg_notify(2.2d8(173) epoch 11714) -2181> 2014-09-18 18:08:35.685019 7f4c9f251700 5 -- op tracker -- , seq: 86217, time: 2014-09-18 18:08:35.682771, event: all_read, op: pg_notify(2.2d8(173) epoch 11714) -2180> 2014-09-18 18:08:35.685045 7f4c9f251700 5 -- op tracker -- , seq: 86217, time: 2014-09-18 18:08:35.684986, event: dispatched, op: pg_notify(2.2d8(173) epoch 11714) -2179> 2014-09-18 18:08:35.685053 7f4c9f251700 5 -- op tracker -- , seq: 86217, time: 2014-09-18 18:08:35.685050, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11714) -2178> 2014-09-18 18:08:35.685062 7f4c9f251700 5 -- op tracker -- , seq: 86217, time: 2014-09-18 18:08:35.685059, event: started, op: pg_notify(2.2d8(173) epoch 11714) -2177> 2014-09-18 18:08:35.685081 7f4c9f251700 5 -- op tracker -- , seq: 86217, time: 2014-09-18 18:08:35.685078, event: done, op: pg_notify(2.2d8(173) epoch 11714) -2176> 2014-09-18 18:08:35.685118 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3686 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0xcc00240 con 0xcbecec0 -2175> 2014-09-18 18:08:35.685150 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2174> 2014-09-18 18:08:35.685156 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 946 ==== pg_notify(2.78f(157) epoch 11714) v5 ==== 9441+0+0 (3961209727 0 0) 0xddf2f00 con 0x1395f8c0 -2173> 2014-09-18 18:08:35.685166 7f4c9f251700 5 -- op tracker -- , seq: 86218, time: 2014-09-18 18:08:35.683395, event: header_read, op: pg_notify(2.78f(157) epoch 11714) -2172> 2014-09-18 18:08:35.685187 7f4c9f251700 5 -- op tracker -- , seq: 86218, time: 2014-09-18 18:08:35.683396, event: throttled, op: pg_notify(2.78f(157) epoch 11714) -2171> 2014-09-18 18:08:35.685194 7f4c9f251700 5 -- op tracker -- , seq: 86218, time: 2014-09-18 18:08:35.683561, event: all_read, op: pg_notify(2.78f(157) epoch 11714) -2170> 2014-09-18 18:08:35.685200 7f4c9f251700 5 -- op tracker -- , seq: 86218, time: 2014-09-18 18:08:35.685163, event: dispatched, op: pg_notify(2.78f(157) epoch 11714) -2169> 2014-09-18 18:08:35.685208 7f4c9f251700 5 -- op tracker -- , seq: 86218, time: 2014-09-18 18:08:35.685206, event: waiting_for_osdmap, op: pg_notify(2.78f(157) epoch 11714) -2168> 2014-09-18 18:08:35.685215 7f4c9f251700 5 -- op tracker -- , seq: 86218, time: 2014-09-18 18:08:35.685213, event: started, op: pg_notify(2.78f(157) epoch 11714) -2167> 2014-09-18 18:08:35.685276 7f4c9f251700 5 -- op tracker -- , seq: 86218, time: 2014-09-18 18:08:35.685274, event: done, op: pg_notify(2.78f(157) epoch 11714) -2166> 2014-09-18 18:08:35.685299 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -2165> 2014-09-18 18:08:35.685304 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_alive(want up_thru 11714 have 11714) v1 -- ?+0 0xf0b6780 con 0x111771e0 -2164> 2014-09-18 18:08:35.685298 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 54 ==== pg_notify(2.1d8(9) epoch 11714) v5 ==== 1261+0+0 (1185477780 0 0) 0xf9732c0 con 0x1395f340 -2163> 2014-09-18 18:08:35.685329 7f4c9f251700 5 -- op tracker -- , seq: 86219, time: 2014-09-18 18:08:35.681647, event: header_read, op: pg_notify(2.1d8(9) epoch 11714) -2162> 2014-09-18 18:08:35.685338 7f4c9f251700 5 -- op tracker -- , seq: 86219, time: 2014-09-18 18:08:35.681648, event: throttled, op: pg_notify(2.1d8(9) epoch 11714) -2161> 2014-09-18 18:08:35.685345 7f4c9f251700 5 -- op tracker -- , seq: 86219, time: 2014-09-18 18:08:35.681682, event: all_read, op: pg_notify(2.1d8(9) epoch 11714) -2160> 2014-09-18 18:08:35.685351 7f4c9f251700 5 -- op tracker -- , seq: 86219, time: 2014-09-18 18:08:35.685324, event: dispatched, op: pg_notify(2.1d8(9) epoch 11714) -2159> 2014-09-18 18:08:35.685358 7f4c9f251700 5 -- op tracker -- , seq: 86219, time: 2014-09-18 18:08:35.685356, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11714) -2158> 2014-09-18 18:08:35.685379 7f4c9f251700 5 -- op tracker -- , seq: 86219, time: 2014-09-18 18:08:35.685365, event: started, op: pg_notify(2.1d8(9) epoch 11714) -2157> 2014-09-18 18:08:35.685399 7f4c9f251700 5 -- op tracker -- , seq: 86219, time: 2014-09-18 18:08:35.685397, event: done, op: pg_notify(2.1d8(9) epoch 11714) -2156> 2014-09-18 18:08:35.685428 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 547 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0xeeea880 con 0x1395ef20 -2155> 2014-09-18 18:08:35.685434 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6857/39817 -- osd_map(11710..11714 src has 1..11714) v3 -- ?+0 0xe267f80 con 0x12b3b2e0 -2154> 2014-09-18 18:08:35.685441 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2153> 2014-09-18 18:08:35.685446 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3687 ==== pg_notify(2.4f6(12) epoch 11714) v5 ==== 1382+0+0 (942984382 0 0) 0xe797840 con 0xcbecec0 -2152> 2014-09-18 18:08:35.685458 7f4c9f251700 5 -- op tracker -- , seq: 86220, time: 2014-09-18 18:08:35.684524, event: header_read, op: pg_notify(2.4f6(12) epoch 11714) -2151> 2014-09-18 18:08:35.685464 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6857/39817 -- pg_query(2.78f epoch 11714) v3 -- ?+0 0xf0dc580 con 0x12b3b2e0 -2150> 2014-09-18 18:08:35.685467 7f4c9f251700 5 -- op tracker -- , seq: 86220, time: 2014-09-18 18:08:35.684525, event: throttled, op: pg_notify(2.4f6(12) epoch 11714) -2149> 2014-09-18 18:08:35.685473 7f4c9f251700 5 -- op tracker -- , seq: 86220, time: 2014-09-18 18:08:35.684563, event: all_read, op: pg_notify(2.4f6(12) epoch 11714) -2148> 2014-09-18 18:08:35.685480 7f4c9f251700 5 -- op tracker -- , seq: 86220, time: 2014-09-18 18:08:35.685454, event: dispatched, op: pg_notify(2.4f6(12) epoch 11714) -2147> 2014-09-18 18:08:35.685488 7f4c9f251700 5 -- op tracker -- , seq: 86220, time: 2014-09-18 18:08:35.685486, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11714) -2146> 2014-09-18 18:08:35.685496 7f4c9f251700 5 -- op tracker -- , seq: 86220, time: 2014-09-18 18:08:35.685494, event: started, op: pg_notify(2.4f6(12) epoch 11714) -2145> 2014-09-18 18:08:35.685521 7f4c9f251700 5 -- op tracker -- , seq: 86220, time: 2014-09-18 18:08:35.685519, event: done, op: pg_notify(2.4f6(12) epoch 11714) -2144> 2014-09-18 18:08:35.685534 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 548 ==== pg_notify(2.4d0(10) epoch 11714) v5 ==== 1292+0+0 (1134593857 0 0) 0xf7a7860 con 0x1395ef20 -2143> 2014-09-18 18:08:35.685547 7f4c9f251700 5 -- op tracker -- , seq: 86221, time: 2014-09-18 18:08:35.685135, event: header_read, op: pg_notify(2.4d0(10) epoch 11714) -2142> 2014-09-18 18:08:35.685554 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11714..11714 src has 1..11714) v3 -- ?+0 0xef59a80 con 0x1395f8c0 -2141> 2014-09-18 18:08:35.685562 7f4c9f251700 5 -- op tracker -- , seq: 86221, time: 2014-09-18 18:08:35.685136, event: throttled, op: pg_notify(2.4d0(10) epoch 11714) -2140> 2014-09-18 18:08:35.685570 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.78f epoch 11714) v3 -- ?+0 0xf0dbfe0 con 0x1395f8c0 -2139> 2014-09-18 18:08:35.685577 7f4c9f251700 5 -- op tracker -- , seq: 86221, time: 2014-09-18 18:08:35.685170, event: all_read, op: pg_notify(2.4d0(10) epoch 11714) -2138> 2014-09-18 18:08:35.685584 7f4c9f251700 5 -- op tracker -- , seq: 86221, time: 2014-09-18 18:08:35.685543, event: dispatched, op: pg_notify(2.4d0(10) epoch 11714) -2137> 2014-09-18 18:08:35.685591 7f4c9f251700 5 -- op tracker -- , seq: 86221, time: 2014-09-18 18:08:35.685588, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11714) -2136> 2014-09-18 18:08:35.685600 7f4c9f251700 5 -- op tracker -- , seq: 86221, time: 2014-09-18 18:08:35.685597, event: started, op: pg_notify(2.4d0(10) epoch 11714) -2135> 2014-09-18 18:08:35.685606 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11714..11714 src has 1..11714) v3 -- ?+0 0xef593c0 con 0xcbecec0 -2134> 2014-09-18 18:08:35.685615 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_query(2.78f epoch 11714) v3 -- ?+0 0xf0db4a0 con 0xcbecec0 -2133> 2014-09-18 18:08:35.685787 7f4c9f251700 5 -- op tracker -- , seq: 86221, time: 2014-09-18 18:08:35.685783, event: done, op: pg_notify(2.4d0(10) epoch 11714) -2132> 2014-09-18 18:08:35.685823 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11714..11714 src has 1..11714) v3 -- ?+0 0x2db45b00 con 0x1395f1e0 -2131> 2014-09-18 18:08:35.685846 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(23) epoch 11714) v5 -- ?+0 0xf1e6d20 con 0x1395f1e0 -2130> 2014-09-18 18:08:35.686247 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2606 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x1440a640 con 0x4900b00 -2129> 2014-09-18 18:08:35.686264 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2128> 2014-09-18 18:08:35.686314 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11714..11714 src has 1..11714) v3 -- ?+0 0x11bb7b00 con 0x1088aec0 -2127> 2014-09-18 18:08:35.686331 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.4e8(8) epoch 11714) v5 -- ?+0 0x21e105a0 con 0x1088aec0 -2126> 2014-09-18 18:08:35.686362 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2607 ==== pg_notify(2.71c(12) epoch 11714) v5 ==== 1424+0+0 (1744963930 0 0) 0xf05af00 con 0x4900b00 -2125> 2014-09-18 18:08:35.686393 7f4c9f251700 5 -- op tracker -- , seq: 86222, time: 2014-09-18 18:08:35.686258, event: header_read, op: pg_notify(2.71c(12) epoch 11714) -2124> 2014-09-18 18:08:35.686406 7f4c9f251700 5 -- op tracker -- , seq: 86222, time: 2014-09-18 18:08:35.686259, event: throttled, op: pg_notify(2.71c(12) epoch 11714) -2123> 2014-09-18 18:08:35.686416 7f4c9f251700 5 -- op tracker -- , seq: 86222, time: 2014-09-18 18:08:35.686346, event: all_read, op: pg_notify(2.71c(12) epoch 11714) -2122> 2014-09-18 18:08:35.686426 7f4c9f251700 5 -- op tracker -- , seq: 86222, time: 2014-09-18 18:08:35.686387, event: dispatched, op: pg_notify(2.71c(12) epoch 11714) -2121> 2014-09-18 18:08:35.686447 7f4c9f251700 5 -- op tracker -- , seq: 86222, time: 2014-09-18 18:08:35.686445, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11714) -2120> 2014-09-18 18:08:35.686457 7f4c9f251700 5 -- op tracker -- , seq: 86222, time: 2014-09-18 18:08:35.686455, event: started, op: pg_notify(2.71c(12) epoch 11714) -2119> 2014-09-18 18:08:35.686474 7f4c9f251700 5 -- op tracker -- , seq: 86222, time: 2014-09-18 18:08:35.686473, event: done, op: pg_notify(2.71c(12) epoch 11714) -2118> 2014-09-18 18:08:35.686499 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11714..11714 src has 1..11714) v3 -- ?+0 0x24dbcd80 con 0x23a74f20 -2117> 2014-09-18 18:08:35.686516 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11714) v5 -- ?+0 0xf0abc20 con 0x23a74f20 -2116> 2014-09-18 18:08:35.687159 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2021 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0xe2eed00 con 0xe2d9a20 -2115> 2014-09-18 18:08:35.687183 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2114> 2014-09-18 18:08:35.687233 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2022 ==== pg_notify(2.26d(9) epoch 11714) v5 ==== 1257+0+0 (2128615603 0 0) 0xf1eb4a0 con 0xe2d9a20 -2113> 2014-09-18 18:08:35.687243 7f4c9f251700 5 -- op tracker -- , seq: 86223, time: 2014-09-18 18:08:35.687194, event: header_read, op: pg_notify(2.26d(9) epoch 11714) -2112> 2014-09-18 18:08:35.687252 7f4c9f251700 5 -- op tracker -- , seq: 86223, time: 2014-09-18 18:08:35.687195, event: throttled, op: pg_notify(2.26d(9) epoch 11714) -2111> 2014-09-18 18:08:35.687258 7f4c9f251700 5 -- op tracker -- , seq: 86223, time: 2014-09-18 18:08:35.687222, event: all_read, op: pg_notify(2.26d(9) epoch 11714) -2110> 2014-09-18 18:08:35.687264 7f4c9f251700 5 -- op tracker -- , seq: 86223, time: 2014-09-18 18:08:35.687240, event: dispatched, op: pg_notify(2.26d(9) epoch 11714) -2109> 2014-09-18 18:08:35.687270 7f4c9f251700 5 -- op tracker -- , seq: 86223, time: 2014-09-18 18:08:35.687269, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11714) -2108> 2014-09-18 18:08:35.687278 7f4c9f251700 5 -- op tracker -- , seq: 86223, time: 2014-09-18 18:08:35.687276, event: started, op: pg_notify(2.26d(9) epoch 11714) -2107> 2014-09-18 18:08:35.687295 7f4c9f251700 5 -- op tracker -- , seq: 86223, time: 2014-09-18 18:08:35.687293, event: done, op: pg_notify(2.26d(9) epoch 11714) -2106> 2014-09-18 18:08:35.689306 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1206 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x2db44fc0 con 0x1088b440 -2105> 2014-09-18 18:08:35.689321 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2104> 2014-09-18 18:08:35.689468 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3688 ==== pg_notify(2.3b6(10) epoch 11714) v5 ==== 1260+0+0 (361141517 0 0) 0xf0db4a0 con 0xcbecec0 -2103> 2014-09-18 18:08:35.689479 7f4c9f251700 5 -- op tracker -- , seq: 86224, time: 2014-09-18 18:08:35.689417, event: header_read, op: pg_notify(2.3b6(10) epoch 11714) -2102> 2014-09-18 18:08:35.689488 7f4c9f251700 5 -- op tracker -- , seq: 86224, time: 2014-09-18 18:08:35.689418, event: throttled, op: pg_notify(2.3b6(10) epoch 11714) -2101> 2014-09-18 18:08:35.689494 7f4c9f251700 5 -- op tracker -- , seq: 86224, time: 2014-09-18 18:08:35.689457, event: all_read, op: pg_notify(2.3b6(10) epoch 11714) -2100> 2014-09-18 18:08:35.689500 7f4c9f251700 5 -- op tracker -- , seq: 86224, time: 2014-09-18 18:08:35.689476, event: dispatched, op: pg_notify(2.3b6(10) epoch 11714) -2099> 2014-09-18 18:08:35.689506 7f4c9f251700 5 -- op tracker -- , seq: 86224, time: 2014-09-18 18:08:35.689505, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11714) -2098> 2014-09-18 18:08:35.689514 7f4c9f251700 5 -- op tracker -- , seq: 86224, time: 2014-09-18 18:08:35.689513, event: started, op: pg_notify(2.3b6(10) epoch 11714) -2097> 2014-09-18 18:08:35.689532 7f4c9f251700 5 -- op tracker -- , seq: 86224, time: 2014-09-18 18:08:35.689531, event: done, op: pg_notify(2.3b6(10) epoch 11714) -2096> 2014-09-18 18:08:35.689572 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3314 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x10d16480 con 0x4900580 -2095> 2014-09-18 18:08:35.689584 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -2094> 2014-09-18 18:08:35.689658 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1207 ==== pg_notify(2.2d8(173) epoch 11714) v5 ==== 10757+0+0 (4135927104 0 0) 0xe76c1e0 con 0x1088b440 -2093> 2014-09-18 18:08:35.689669 7f4c9f251700 5 -- op tracker -- , seq: 86225, time: 2014-09-18 18:08:35.689480, event: header_read, op: pg_notify(2.2d8(173) epoch 11714) -2092> 2014-09-18 18:08:35.689676 7f4c9f251700 5 -- op tracker -- , seq: 86225, time: 2014-09-18 18:08:35.689481, event: throttled, op: pg_notify(2.2d8(173) epoch 11714) -2091> 2014-09-18 18:08:35.689682 7f4c9f251700 5 -- op tracker -- , seq: 86225, time: 2014-09-18 18:08:35.689645, event: all_read, op: pg_notify(2.2d8(173) epoch 11714) -2090> 2014-09-18 18:08:35.689688 7f4c9f251700 5 -- op tracker -- , seq: 86225, time: 2014-09-18 18:08:35.689666, event: dispatched, op: pg_notify(2.2d8(173) epoch 11714) -2089> 2014-09-18 18:08:35.689695 7f4c9f251700 5 -- op tracker -- , seq: 86225, time: 2014-09-18 18:08:35.689694, event: waiting_for_osdmap, op: pg_notify(2.2d8(173) epoch 11714) -2088> 2014-09-18 18:08:35.689702 7f4c9f251700 5 -- op tracker -- , seq: 86225, time: 2014-09-18 18:08:35.689701, event: started, op: pg_notify(2.2d8(173) epoch 11714) -2087> 2014-09-18 18:08:35.689718 7f4c9f251700 5 -- op tracker -- , seq: 86225, time: 2014-09-18 18:08:35.689716, event: done, op: pg_notify(2.2d8(173) epoch 11714) -2086> 2014-09-18 18:08:35.689843 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3315 ==== pg_notify(2.4f6(184) epoch 11714) v5 ==== 11296+0+0 (1475813835 0 0) 0xec68f00 con 0x4900580 -2085> 2014-09-18 18:08:35.689862 7f4c9f251700 5 -- op tracker -- , seq: 86226, time: 2014-09-18 18:08:35.689667, event: header_read, op: pg_notify(2.4f6(184) epoch 11714) -2084> 2014-09-18 18:08:35.689874 7f4c9f251700 5 -- op tracker -- , seq: 86226, time: 2014-09-18 18:08:35.689668, event: throttled, op: pg_notify(2.4f6(184) epoch 11714) -2083> 2014-09-18 18:08:35.689882 7f4c9f251700 5 -- op tracker -- , seq: 86226, time: 2014-09-18 18:08:35.689830, event: all_read, op: pg_notify(2.4f6(184) epoch 11714) -2082> 2014-09-18 18:08:35.689892 7f4c9f251700 5 -- op tracker -- , seq: 86226, time: 2014-09-18 18:08:35.689857, event: dispatched, op: pg_notify(2.4f6(184) epoch 11714) -2081> 2014-09-18 18:08:35.689900 7f4c9f251700 5 -- op tracker -- , seq: 86226, time: 2014-09-18 18:08:35.689898, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11714) -2080> 2014-09-18 18:08:35.689907 7f4c9f251700 5 -- op tracker -- , seq: 86226, time: 2014-09-18 18:08:35.689906, event: started, op: pg_notify(2.4f6(184) epoch 11714) -2079> 2014-09-18 18:08:35.689928 7f4c9f251700 5 -- op tracker -- , seq: 86226, time: 2014-09-18 18:08:35.689926, event: done, op: pg_notify(2.4f6(184) epoch 11714) -2078> 2014-09-18 18:08:35.690655 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3316 ==== pg_notify(2.17e(17) epoch 11714) v5 ==== 1593+0+0 (1420404463 0 0) 0xec6a940 con 0x4900580 -2077> 2014-09-18 18:08:35.690668 7f4c9f251700 5 -- op tracker -- , seq: 86227, time: 2014-09-18 18:08:35.690604, event: header_read, op: pg_notify(2.17e(17) epoch 11714) -2076> 2014-09-18 18:08:35.690676 7f4c9f251700 5 -- op tracker -- , seq: 86227, time: 2014-09-18 18:08:35.690604, event: throttled, op: pg_notify(2.17e(17) epoch 11714) -2075> 2014-09-18 18:08:35.690682 7f4c9f251700 5 -- op tracker -- , seq: 86227, time: 2014-09-18 18:08:35.690644, event: all_read, op: pg_notify(2.17e(17) epoch 11714) -2074> 2014-09-18 18:08:35.690688 7f4c9f251700 5 -- op tracker -- , seq: 86227, time: 2014-09-18 18:08:35.690665, event: dispatched, op: pg_notify(2.17e(17) epoch 11714) -2073> 2014-09-18 18:08:35.690695 7f4c9f251700 5 -- op tracker -- , seq: 86227, time: 2014-09-18 18:08:35.690693, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11714) -2072> 2014-09-18 18:08:35.690702 7f4c9f251700 5 -- op tracker -- , seq: 86227, time: 2014-09-18 18:08:35.690700, event: started, op: pg_notify(2.17e(17) epoch 11714) -2071> 2014-09-18 18:08:35.690719 7f4c9f251700 5 -- op tracker -- , seq: 86227, time: 2014-09-18 18:08:35.690717, event: done, op: pg_notify(2.17e(17) epoch 11714) -2070> 2014-09-18 18:08:35.691240 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3317 ==== pg_notify(2.1bb(9) epoch 11714) v5 ==== 1189+0+0 (3396298614 0 0) 0xf2b23c0 con 0x4900580 -2069> 2014-09-18 18:08:35.691256 7f4c9f251700 5 -- op tracker -- , seq: 86228, time: 2014-09-18 18:08:35.691181, event: header_read, op: pg_notify(2.1bb(9) epoch 11714) -2068> 2014-09-18 18:08:35.691266 7f4c9f251700 5 -- op tracker -- , seq: 86228, time: 2014-09-18 18:08:35.691182, event: throttled, op: pg_notify(2.1bb(9) epoch 11714) -2067> 2014-09-18 18:08:35.691276 7f4c9f251700 5 -- op tracker -- , seq: 86228, time: 2014-09-18 18:08:35.691224, event: all_read, op: pg_notify(2.1bb(9) epoch 11714) -2066> 2014-09-18 18:08:35.691286 7f4c9f251700 5 -- op tracker -- , seq: 86228, time: 2014-09-18 18:08:35.691253, event: dispatched, op: pg_notify(2.1bb(9) epoch 11714) -2065> 2014-09-18 18:08:35.691294 7f4c9f251700 5 -- op tracker -- , seq: 86228, time: 2014-09-18 18:08:35.691292, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11714) -2064> 2014-09-18 18:08:35.691304 7f4c9f251700 5 -- op tracker -- , seq: 86228, time: 2014-09-18 18:08:35.691301, event: started, op: pg_notify(2.1bb(9) epoch 11714) -2063> 2014-09-18 18:08:35.691325 7f4c9f251700 5 -- op tracker -- , seq: 86228, time: 2014-09-18 18:08:35.691323, event: done, op: pg_notify(2.1bb(9) epoch 11714) -2062> 2014-09-18 18:08:35.692628 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3689 ==== pg_notify(2.517(11) epoch 11714) v5 ==== 1343+0+0 (4118665519 0 0) 0xe7963a0 con 0xcbecec0 -2061> 2014-09-18 18:08:35.692645 7f4c9f251700 5 -- op tracker -- , seq: 86229, time: 2014-09-18 18:08:35.692575, event: header_read, op: pg_notify(2.517(11) epoch 11714) -2060> 2014-09-18 18:08:35.692655 7f4c9f251700 5 -- op tracker -- , seq: 86229, time: 2014-09-18 18:08:35.692576, event: throttled, op: pg_notify(2.517(11) epoch 11714) -2059> 2014-09-18 18:08:35.692661 7f4c9f251700 5 -- op tracker -- , seq: 86229, time: 2014-09-18 18:08:35.692616, event: all_read, op: pg_notify(2.517(11) epoch 11714) -2058> 2014-09-18 18:08:35.692667 7f4c9f251700 5 -- op tracker -- , seq: 86229, time: 2014-09-18 18:08:35.692641, event: dispatched, op: pg_notify(2.517(11) epoch 11714) -2057> 2014-09-18 18:08:35.692673 7f4c9f251700 5 -- op tracker -- , seq: 86229, time: 2014-09-18 18:08:35.692672, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11714) -2056> 2014-09-18 18:08:35.692681 7f4c9f251700 5 -- op tracker -- , seq: 86229, time: 2014-09-18 18:08:35.692680, event: started, op: pg_notify(2.517(11) epoch 11714) -2055> 2014-09-18 18:08:35.692698 7f4c9f251700 5 -- op tracker -- , seq: 86229, time: 2014-09-18 18:08:35.692696, event: done, op: pg_notify(2.517(11) epoch 11714) -2054> 2014-09-18 18:08:35.694067 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3690 ==== pg_notify(2.2c7(139) epoch 11714) v5 ==== 8651+0+0 (2123783653 0 0) 0xe794f00 con 0xcbecec0 -2053> 2014-09-18 18:08:35.694081 7f4c9f251700 5 -- op tracker -- , seq: 86230, time: 2014-09-18 18:08:35.693938, event: header_read, op: pg_notify(2.2c7(139) epoch 11714) -2052> 2014-09-18 18:08:35.694090 7f4c9f251700 5 -- op tracker -- , seq: 86230, time: 2014-09-18 18:08:35.693939, event: throttled, op: pg_notify(2.2c7(139) epoch 11714) -2051> 2014-09-18 18:08:35.694096 7f4c9f251700 5 -- op tracker -- , seq: 86230, time: 2014-09-18 18:08:35.694047, event: all_read, op: pg_notify(2.2c7(139) epoch 11714) -2050> 2014-09-18 18:08:35.694103 7f4c9f251700 5 -- op tracker -- , seq: 86230, time: 2014-09-18 18:08:35.694078, event: dispatched, op: pg_notify(2.2c7(139) epoch 11714) -2049> 2014-09-18 18:08:35.694108 7f4c9f251700 5 -- op tracker -- , seq: 86230, time: 2014-09-18 18:08:35.694106, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11714) -2048> 2014-09-18 18:08:35.694124 7f4c9f251700 5 -- op tracker -- , seq: 86230, time: 2014-09-18 18:08:35.694123, event: started, op: pg_notify(2.2c7(139) epoch 11714) -2047> 2014-09-18 18:08:35.694145 7f4c9f251700 5 -- op tracker -- , seq: 86230, time: 2014-09-18 18:08:35.694143, event: done, op: pg_notify(2.2c7(139) epoch 11714) -2046> 2014-09-18 18:08:35.694247 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 1 ==== pg_notify(2.120(15) epoch 11714) v5 ==== 1535+0+0 (652212397 0 0) 0xf0dc580 con 0x12b3b2e0 -2045> 2014-09-18 18:08:35.694260 7f4c9f251700 5 -- op tracker -- , seq: 86231, time: 2014-09-18 18:08:35.694068, event: header_read, op: pg_notify(2.120(15) epoch 11714) -2044> 2014-09-18 18:08:35.694268 7f4c9f251700 5 -- op tracker -- , seq: 86231, time: 2014-09-18 18:08:35.694069, event: throttled, op: pg_notify(2.120(15) epoch 11714) -2043> 2014-09-18 18:08:35.694275 7f4c9f251700 5 -- op tracker -- , seq: 86231, time: 2014-09-18 18:08:35.694128, event: all_read, op: pg_notify(2.120(15) epoch 11714) -2042> 2014-09-18 18:08:35.694281 7f4c9f251700 5 -- op tracker -- , seq: 86231, time: 2014-09-18 18:08:35.694257, event: dispatched, op: pg_notify(2.120(15) epoch 11714) -2041> 2014-09-18 18:08:35.694288 7f4c9f251700 5 -- op tracker -- , seq: 86231, time: 2014-09-18 18:08:35.694286, event: waiting_for_osdmap, op: pg_notify(2.120(15) epoch 11714) -2040> 2014-09-18 18:08:35.694295 7f4c9f251700 5 -- op tracker -- , seq: 86231, time: 2014-09-18 18:08:35.694293, event: started, op: pg_notify(2.120(15) epoch 11714) -2039> 2014-09-18 18:08:35.694311 7f4c9f251700 5 -- op tracker -- , seq: 86231, time: 2014-09-18 18:08:35.694309, event: done, op: pg_notify(2.120(15) epoch 11714) -2038> 2014-09-18 18:08:35.695501 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 2 ==== pg_notify(2.40a(162) epoch 11714) v5 ==== 9736+0+0 (2801459469 0 0) 0xf0abe00 con 0x12b3b2e0 -2037> 2014-09-18 18:08:35.695516 7f4c9f251700 5 -- op tracker -- , seq: 86232, time: 2014-09-18 18:08:35.695365, event: header_read, op: pg_notify(2.40a(162) epoch 11714) -2036> 2014-09-18 18:08:35.695524 7f4c9f251700 5 -- op tracker -- , seq: 86232, time: 2014-09-18 18:08:35.695366, event: throttled, op: pg_notify(2.40a(162) epoch 11714) -2035> 2014-09-18 18:08:35.695530 7f4c9f251700 5 -- op tracker -- , seq: 86232, time: 2014-09-18 18:08:35.695488, event: all_read, op: pg_notify(2.40a(162) epoch 11714) -2034> 2014-09-18 18:08:35.695536 7f4c9f251700 5 -- op tracker -- , seq: 86232, time: 2014-09-18 18:08:35.695512, event: dispatched, op: pg_notify(2.40a(162) epoch 11714) -2033> 2014-09-18 18:08:35.695542 7f4c9f251700 5 -- op tracker -- , seq: 86232, time: 2014-09-18 18:08:35.695541, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11714) -2032> 2014-09-18 18:08:35.695550 7f4c9f251700 5 -- op tracker -- , seq: 86232, time: 2014-09-18 18:08:35.695548, event: started, op: pg_notify(2.40a(162) epoch 11714) -2031> 2014-09-18 18:08:35.695566 7f4c9f251700 5 -- op tracker -- , seq: 86232, time: 2014-09-18 18:08:35.695565, event: done, op: pg_notify(2.40a(162) epoch 11714) -2030> 2014-09-18 18:08:35.697995 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3691 ==== pg_notify(2.40a(162) epoch 11714) v5 ==== 9736+0+0 (1486794795 0 0) 0x19f23e00 con 0xcbecec0 -2029> 2014-09-18 18:08:35.698012 7f4c9f251700 5 -- op tracker -- , seq: 86233, time: 2014-09-18 18:08:35.697800, event: header_read, op: pg_notify(2.40a(162) epoch 11714) -2028> 2014-09-18 18:08:35.698022 7f4c9f251700 5 -- op tracker -- , seq: 86233, time: 2014-09-18 18:08:35.697801, event: throttled, op: pg_notify(2.40a(162) epoch 11714) -2027> 2014-09-18 18:08:35.698028 7f4c9f251700 5 -- op tracker -- , seq: 86233, time: 2014-09-18 18:08:35.697972, event: all_read, op: pg_notify(2.40a(162) epoch 11714) -2026> 2014-09-18 18:08:35.698034 7f4c9f251700 5 -- op tracker -- , seq: 86233, time: 2014-09-18 18:08:35.698008, event: dispatched, op: pg_notify(2.40a(162) epoch 11714) -2025> 2014-09-18 18:08:35.698049 7f4c9f251700 5 -- op tracker -- , seq: 86233, time: 2014-09-18 18:08:35.698047, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11714) -2024> 2014-09-18 18:08:35.698056 7f4c9f251700 5 -- op tracker -- , seq: 86233, time: 2014-09-18 18:08:35.698055, event: started, op: pg_notify(2.40a(162) epoch 11714) -2023> 2014-09-18 18:08:35.698074 7f4c9f251700 5 -- op tracker -- , seq: 86233, time: 2014-09-18 18:08:35.698072, event: done, op: pg_notify(2.40a(162) epoch 11714) -2022> 2014-09-18 18:08:35.699117 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3692 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11714) v5 ==== 6174+0+0 (2384936793 0 0) 0x19f24b20 con 0xcbecec0 -2021> 2014-09-18 18:08:35.699141 7f4c9f251700 5 -- op tracker -- , seq: 86234, time: 2014-09-18 18:08:35.699026, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11714) -2020> 2014-09-18 18:08:35.699181 7f4c9f251700 5 -- op tracker -- , seq: 86234, time: 2014-09-18 18:08:35.699027, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11714) -2019> 2014-09-18 18:08:35.699188 7f4c9f251700 5 -- op tracker -- , seq: 86234, time: 2014-09-18 18:08:35.699105, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11714) -2018> 2014-09-18 18:08:35.699195 7f4c9f251700 5 -- op tracker -- , seq: 86234, time: 2014-09-18 18:08:35.699137, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11714) -2017> 2014-09-18 18:08:35.699210 7f4c9f251700 5 -- op tracker -- , seq: 86234, time: 2014-09-18 18:08:35.699208, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11714) -2016> 2014-09-18 18:08:35.699218 7f4c9f251700 5 -- op tracker -- , seq: 86234, time: 2014-09-18 18:08:35.699216, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11714) -2015> 2014-09-18 18:08:35.699255 7f4c9f251700 5 -- op tracker -- , seq: 86234, time: 2014-09-18 18:08:35.699253, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11714) -2014> 2014-09-18 18:08:35.699676 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 947 ==== pg_notify(2.78f(157) epoch 11714) v5 ==== 9441+0+0 (3961209727 0 0) 0xf0dbfe0 con 0x1395f8c0 -2013> 2014-09-18 18:08:35.699693 7f4c9f251700 5 -- op tracker -- , seq: 86235, time: 2014-09-18 18:08:35.699518, event: header_read, op: pg_notify(2.78f(157) epoch 11714) -2012> 2014-09-18 18:08:35.699701 7f4c9f251700 5 -- op tracker -- , seq: 86235, time: 2014-09-18 18:08:35.699519, event: throttled, op: pg_notify(2.78f(157) epoch 11714) -2011> 2014-09-18 18:08:35.699707 7f4c9f251700 5 -- op tracker -- , seq: 86235, time: 2014-09-18 18:08:35.699651, event: all_read, op: pg_notify(2.78f(157) epoch 11714) -2010> 2014-09-18 18:08:35.699713 7f4c9f251700 5 -- op tracker -- , seq: 86235, time: 2014-09-18 18:08:35.699689, event: dispatched, op: pg_notify(2.78f(157) epoch 11714) -2009> 2014-09-18 18:08:35.699719 7f4c9f251700 5 -- op tracker -- , seq: 86235, time: 2014-09-18 18:08:35.699718, event: waiting_for_osdmap, op: pg_notify(2.78f(157) epoch 11714) -2008> 2014-09-18 18:08:35.699727 7f4c9f251700 5 -- op tracker -- , seq: 86235, time: 2014-09-18 18:08:35.699726, event: started, op: pg_notify(2.78f(157) epoch 11714) -2007> 2014-09-18 18:08:35.699744 7f4c9f251700 5 -- op tracker -- , seq: 86235, time: 2014-09-18 18:08:35.699743, event: done, op: pg_notify(2.78f(157) epoch 11714) -2006> 2014-09-18 18:08:35.722796 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 3 ==== pg_notify(2.1a2(20) epoch 11714) v5 ==== 1798+0+0 (1975998670 0 0) 0xec68f00 con 0x12b3b2e0 -2005> 2014-09-18 18:08:35.722816 7f4c9f251700 5 -- op tracker -- , seq: 86236, time: 2014-09-18 18:08:35.722697, event: header_read, op: pg_notify(2.1a2(20) epoch 11714) -2004> 2014-09-18 18:08:35.722836 7f4c9f251700 5 -- op tracker -- , seq: 86236, time: 2014-09-18 18:08:35.722698, event: throttled, op: pg_notify(2.1a2(20) epoch 11714) -2003> 2014-09-18 18:08:35.722842 7f4c9f251700 5 -- op tracker -- , seq: 86236, time: 2014-09-18 18:08:35.722758, event: all_read, op: pg_notify(2.1a2(20) epoch 11714) -2002> 2014-09-18 18:08:35.722848 7f4c9f251700 5 -- op tracker -- , seq: 86236, time: 2014-09-18 18:08:35.722811, event: dispatched, op: pg_notify(2.1a2(20) epoch 11714) -2001> 2014-09-18 18:08:35.722855 7f4c9f251700 5 -- op tracker -- , seq: 86236, time: 2014-09-18 18:08:35.722853, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11714) -2000> 2014-09-18 18:08:35.722863 7f4c9f251700 5 -- op tracker -- , seq: 86236, time: 2014-09-18 18:08:35.722861, event: started, op: pg_notify(2.1a2(20) epoch 11714) -1999> 2014-09-18 18:08:35.722883 7f4c9f251700 5 -- op tracker -- , seq: 86236, time: 2014-09-18 18:08:35.722881, event: done, op: pg_notify(2.1a2(20) epoch 11714) -1998> 2014-09-18 18:08:35.726774 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 4 ==== pg_notify(2.78f(157),2.4c6(15) epoch 11714) v5 ==== 10940+0+0 (3019743961 0 0) 0xe76c1e0 con 0x12b3b2e0 -1997> 2014-09-18 18:08:35.726791 7f4c9f251700 5 -- op tracker -- , seq: 86237, time: 2014-09-18 18:08:35.726561, event: header_read, op: pg_notify(2.78f(157),2.4c6(15) epoch 11714) -1996> 2014-09-18 18:08:35.726801 7f4c9f251700 5 -- op tracker -- , seq: 86237, time: 2014-09-18 18:08:35.726562, event: throttled, op: pg_notify(2.78f(157),2.4c6(15) epoch 11714) -1995> 2014-09-18 18:08:35.726808 7f4c9f251700 5 -- op tracker -- , seq: 86237, time: 2014-09-18 18:08:35.726737, event: all_read, op: pg_notify(2.78f(157),2.4c6(15) epoch 11714) -1994> 2014-09-18 18:08:35.726814 7f4c9f251700 5 -- op tracker -- , seq: 86237, time: 2014-09-18 18:08:35.726787, event: dispatched, op: pg_notify(2.78f(157),2.4c6(15) epoch 11714) -1993> 2014-09-18 18:08:35.726821 7f4c9f251700 5 -- op tracker -- , seq: 86237, time: 2014-09-18 18:08:35.726819, event: waiting_for_osdmap, op: pg_notify(2.78f(157),2.4c6(15) epoch 11714) -1992> 2014-09-18 18:08:35.726829 7f4c9f251700 5 -- op tracker -- , seq: 86237, time: 2014-09-18 18:08:35.726827, event: started, op: pg_notify(2.78f(157),2.4c6(15) epoch 11714) -1991> 2014-09-18 18:08:35.726856 7f4c9f251700 5 -- op tracker -- , seq: 86237, time: 2014-09-18 18:08:35.726854, event: done, op: pg_notify(2.78f(157),2.4c6(15) epoch 11714) -1990> 2014-09-18 18:08:35.735210 7f4caae41700 5 osd.42 11714 tick -1989> 2014-09-18 18:08:35.752632 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 5 ==== pg_notify(2.78f(157) epoch 11714) v5 ==== 9441+0+0 (3961209727 0 0) 0xf0ab4a0 con 0x12b3b2e0 -1988> 2014-09-18 18:08:35.752653 7f4c9f251700 5 -- op tracker -- , seq: 86238, time: 2014-09-18 18:08:35.752412, event: header_read, op: pg_notify(2.78f(157) epoch 11714) -1987> 2014-09-18 18:08:35.752664 7f4c9f251700 5 -- op tracker -- , seq: 86238, time: 2014-09-18 18:08:35.752414, event: throttled, op: pg_notify(2.78f(157) epoch 11714) -1986> 2014-09-18 18:08:35.752678 7f4c9f251700 5 -- op tracker -- , seq: 86238, time: 2014-09-18 18:08:35.752592, event: all_read, op: pg_notify(2.78f(157) epoch 11714) -1985> 2014-09-18 18:08:35.752686 7f4c9f251700 5 -- op tracker -- , seq: 86238, time: 2014-09-18 18:08:35.752649, event: dispatched, op: pg_notify(2.78f(157) epoch 11714) -1984> 2014-09-18 18:08:35.752696 7f4c9f251700 5 -- op tracker -- , seq: 86238, time: 2014-09-18 18:08:35.752694, event: waiting_for_osdmap, op: pg_notify(2.78f(157) epoch 11714) -1983> 2014-09-18 18:08:35.752705 7f4c9f251700 5 -- op tracker -- , seq: 86238, time: 2014-09-18 18:08:35.752704, event: started, op: pg_notify(2.78f(157) epoch 11714) -1982> 2014-09-18 18:08:35.752726 7f4c9f251700 5 -- op tracker -- , seq: 86238, time: 2014-09-18 18:08:35.752723, event: done, op: pg_notify(2.78f(157) epoch 11714) -1981> 2014-09-18 18:08:35.774751 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1289 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0xeee98c0 con 0x1f102840 -1980> 2014-09-18 18:08:35.774768 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -1979> 2014-09-18 18:08:35.774800 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1290 ==== pg_info(1 pgs e11710:2.519) v4 ==== 718+0+0 (4272558907 0 0) 0xe1a5840 con 0x1f102840 -1978> 2014-09-18 18:08:35.774812 7f4c9f251700 5 -- op tracker -- , seq: 86239, time: 2014-09-18 18:08:35.774757, event: header_read, op: pg_info(1 pgs e11710:2.519) -1977> 2014-09-18 18:08:35.774821 7f4c9f251700 5 -- op tracker -- , seq: 86239, time: 2014-09-18 18:08:35.774758, event: throttled, op: pg_info(1 pgs e11710:2.519) -1976> 2014-09-18 18:08:35.774829 7f4c9f251700 5 -- op tracker -- , seq: 86239, time: 2014-09-18 18:08:35.774790, event: all_read, op: pg_info(1 pgs e11710:2.519) -1975> 2014-09-18 18:08:35.774837 7f4c9f251700 5 -- op tracker -- , seq: 86239, time: 2014-09-18 18:08:35.774808, event: dispatched, op: pg_info(1 pgs e11710:2.519) -1974> 2014-09-18 18:08:35.774844 7f4c9f251700 5 -- op tracker -- , seq: 86239, time: 2014-09-18 18:08:35.774843, event: waiting_for_osdmap, op: pg_info(1 pgs e11710:2.519) -1973> 2014-09-18 18:08:35.774854 7f4c9f251700 5 -- op tracker -- , seq: 86239, time: 2014-09-18 18:08:35.774852, event: started, op: pg_info(1 pgs e11710:2.519) -1972> 2014-09-18 18:08:35.774895 7f4c9f251700 5 -- op tracker -- , seq: 86239, time: 2014-09-18 18:08:35.774893, event: done, op: pg_info(1 pgs e11710:2.519) -1971> 2014-09-18 18:08:35.786933 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 702 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x11bb6d80 con 0x1f103e40 -1970> 2014-09-18 18:08:35.786946 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11713,11713], i have 11714, src has [1,11713] -1969> 2014-09-18 18:08:35.786985 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 703 ==== pg_notify(2.4f6(15) epoch 11713) v5 ==== 1543+0+0 (4116341755 0 0) 0xe958d00 con 0x1f103e40 -1968> 2014-09-18 18:08:35.786996 7f4c9f251700 5 -- op tracker -- , seq: 86240, time: 2014-09-18 18:08:35.786932, event: header_read, op: pg_notify(2.4f6(15) epoch 11713) -1967> 2014-09-18 18:08:35.787004 7f4c9f251700 5 -- op tracker -- , seq: 86240, time: 2014-09-18 18:08:35.786932, event: throttled, op: pg_notify(2.4f6(15) epoch 11713) -1966> 2014-09-18 18:08:35.787011 7f4c9f251700 5 -- op tracker -- , seq: 86240, time: 2014-09-18 18:08:35.786976, event: all_read, op: pg_notify(2.4f6(15) epoch 11713) -1965> 2014-09-18 18:08:35.787019 7f4c9f251700 5 -- op tracker -- , seq: 86240, time: 2014-09-18 18:08:35.786993, event: dispatched, op: pg_notify(2.4f6(15) epoch 11713) -1964> 2014-09-18 18:08:35.787027 7f4c9f251700 5 -- op tracker -- , seq: 86240, time: 2014-09-18 18:08:35.787026, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11713) -1963> 2014-09-18 18:08:35.787036 7f4c9f251700 5 -- op tracker -- , seq: 86240, time: 2014-09-18 18:08:35.787034, event: started, op: pg_notify(2.4f6(15) epoch 11713) -1962> 2014-09-18 18:08:35.787053 7f4c9f251700 5 -- op tracker -- , seq: 86240, time: 2014-09-18 18:08:35.787051, event: done, op: pg_notify(2.4f6(15) epoch 11713) -1961> 2014-09-18 18:08:35.789330 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 704 ==== pg_notify(2.799(10),2.4c6(15) epoch 11713) v5 ==== 2779+0+0 (2678574510 0 0) 0xe957680 con 0x1f103e40 -1960> 2014-09-18 18:08:35.789345 7f4c9f251700 5 -- op tracker -- , seq: 86241, time: 2014-09-18 18:08:35.789255, event: header_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11713) -1959> 2014-09-18 18:08:35.789356 7f4c9f251700 5 -- op tracker -- , seq: 86241, time: 2014-09-18 18:08:35.789256, event: throttled, op: pg_notify(2.799(10),2.4c6(15) epoch 11713) -1958> 2014-09-18 18:08:35.789362 7f4c9f251700 5 -- op tracker -- , seq: 86241, time: 2014-09-18 18:08:35.789317, event: all_read, op: pg_notify(2.799(10),2.4c6(15) epoch 11713) -1957> 2014-09-18 18:08:35.789368 7f4c9f251700 5 -- op tracker -- , seq: 86241, time: 2014-09-18 18:08:35.789341, event: dispatched, op: pg_notify(2.799(10),2.4c6(15) epoch 11713) -1956> 2014-09-18 18:08:35.789375 7f4c9f251700 5 -- op tracker -- , seq: 86241, time: 2014-09-18 18:08:35.789373, event: waiting_for_osdmap, op: pg_notify(2.799(10),2.4c6(15) epoch 11713) -1955> 2014-09-18 18:08:35.789382 7f4c9f251700 5 -- op tracker -- , seq: 86241, time: 2014-09-18 18:08:35.789381, event: started, op: pg_notify(2.799(10),2.4c6(15) epoch 11713) -1954> 2014-09-18 18:08:35.789407 7f4c9f251700 5 -- op tracker -- , seq: 86241, time: 2014-09-18 18:08:35.789405, event: done, op: pg_notify(2.799(10),2.4c6(15) epoch 11713) -1953> 2014-09-18 18:08:35.794661 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3243 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x15e30900 con 0x1395ec60 -1952> 2014-09-18 18:08:35.794675 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -1951> 2014-09-18 18:08:35.794730 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3244 ==== pg_notify(2.71(19) epoch 11714) v5 ==== 1795+0+0 (1555277645 0 0) 0x2b93f2c0 con 0x1395ec60 -1950> 2014-09-18 18:08:35.794741 7f4c9f251700 5 -- op tracker -- , seq: 86242, time: 2014-09-18 18:08:35.794681, event: header_read, op: pg_notify(2.71(19) epoch 11714) -1949> 2014-09-18 18:08:35.794750 7f4c9f251700 5 -- op tracker -- , seq: 86242, time: 2014-09-18 18:08:35.794684, event: throttled, op: pg_notify(2.71(19) epoch 11714) -1948> 2014-09-18 18:08:35.794756 7f4c9f251700 5 -- op tracker -- , seq: 86242, time: 2014-09-18 18:08:35.794720, event: all_read, op: pg_notify(2.71(19) epoch 11714) -1947> 2014-09-18 18:08:35.794762 7f4c9f251700 5 -- op tracker -- , seq: 86242, time: 2014-09-18 18:08:35.794738, event: dispatched, op: pg_notify(2.71(19) epoch 11714) -1946> 2014-09-18 18:08:35.794768 7f4c9f251700 5 -- op tracker -- , seq: 86242, time: 2014-09-18 18:08:35.794767, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11714) -1945> 2014-09-18 18:08:35.794776 7f4c9f251700 5 -- op tracker -- , seq: 86242, time: 2014-09-18 18:08:35.794774, event: started, op: pg_notify(2.71(19) epoch 11714) -1944> 2014-09-18 18:08:35.794792 7f4c9f251700 5 -- op tracker -- , seq: 86242, time: 2014-09-18 18:08:35.794791, event: done, op: pg_notify(2.71(19) epoch 11714) -1943> 2014-09-18 18:08:35.795121 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 705 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x11bb7d40 con 0x1f103e40 -1942> 2014-09-18 18:08:35.795134 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -1941> 2014-09-18 18:08:35.795203 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 706 ==== pg_notify(2.4f6(15) epoch 11714) v5 ==== 1543+0+0 (501847009 0 0) 0xe9565a0 con 0x1f103e40 -1940> 2014-09-18 18:08:35.795217 7f4c9f251700 5 -- op tracker -- , seq: 86243, time: 2014-09-18 18:08:35.795140, event: header_read, op: pg_notify(2.4f6(15) epoch 11714) -1939> 2014-09-18 18:08:35.795226 7f4c9f251700 5 -- op tracker -- , seq: 86243, time: 2014-09-18 18:08:35.795141, event: throttled, op: pg_notify(2.4f6(15) epoch 11714) -1938> 2014-09-18 18:08:35.795230 7f4c9f251700 5 -- op tracker -- , seq: 86243, time: 2014-09-18 18:08:35.795189, event: all_read, op: pg_notify(2.4f6(15) epoch 11714) -1937> 2014-09-18 18:08:35.795236 7f4c9f251700 5 -- op tracker -- , seq: 86243, time: 2014-09-18 18:08:35.795214, event: dispatched, op: pg_notify(2.4f6(15) epoch 11714) -1936> 2014-09-18 18:08:35.795247 7f4c9f251700 5 -- op tracker -- , seq: 86243, time: 2014-09-18 18:08:35.795244, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11714) -1935> 2014-09-18 18:08:35.795258 7f4c9f251700 5 -- op tracker -- , seq: 86243, time: 2014-09-18 18:08:35.795256, event: started, op: pg_notify(2.4f6(15) epoch 11714) -1934> 2014-09-18 18:08:35.795285 7f4c9f251700 5 -- op tracker -- , seq: 86243, time: 2014-09-18 18:08:35.795282, event: done, op: pg_notify(2.4f6(15) epoch 11714) -1933> 2014-09-18 18:08:35.796538 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 707 ==== pg_notify(2.799(10) epoch 11714) v5 ==== 1280+0+0 (3450943997 0 0) 0xe1ad0c0 con 0x1f103e40 -1932> 2014-09-18 18:08:35.796554 7f4c9f251700 5 -- op tracker -- , seq: 86244, time: 2014-09-18 18:08:35.796492, event: header_read, op: pg_notify(2.799(10) epoch 11714) -1931> 2014-09-18 18:08:35.796562 7f4c9f251700 5 -- op tracker -- , seq: 86244, time: 2014-09-18 18:08:35.796493, event: throttled, op: pg_notify(2.799(10) epoch 11714) -1930> 2014-09-18 18:08:35.796568 7f4c9f251700 5 -- op tracker -- , seq: 86244, time: 2014-09-18 18:08:35.796525, event: all_read, op: pg_notify(2.799(10) epoch 11714) -1929> 2014-09-18 18:08:35.796574 7f4c9f251700 5 -- op tracker -- , seq: 86244, time: 2014-09-18 18:08:35.796550, event: dispatched, op: pg_notify(2.799(10) epoch 11714) -1928> 2014-09-18 18:08:35.796581 7f4c9f251700 5 -- op tracker -- , seq: 86244, time: 2014-09-18 18:08:35.796579, event: waiting_for_osdmap, op: pg_notify(2.799(10) epoch 11714) -1927> 2014-09-18 18:08:35.796588 7f4c9f251700 5 -- op tracker -- , seq: 86244, time: 2014-09-18 18:08:35.796586, event: started, op: pg_notify(2.799(10) epoch 11714) -1926> 2014-09-18 18:08:35.796604 7f4c9f251700 5 -- op tracker -- , seq: 86244, time: 2014-09-18 18:08:35.796603, event: done, op: pg_notify(2.799(10) epoch 11714) -1925> 2014-09-18 18:08:35.797284 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 708 ==== pg_notify(2.4c6(15) epoch 11714) v5 ==== 1511+0+0 (874141727 0 0) 0xe1aba40 con 0x1f103e40 -1924> 2014-09-18 18:08:35.797303 7f4c9f251700 5 -- op tracker -- , seq: 86245, time: 2014-09-18 18:08:35.797216, event: header_read, op: pg_notify(2.4c6(15) epoch 11714) -1923> 2014-09-18 18:08:35.797314 7f4c9f251700 5 -- op tracker -- , seq: 86245, time: 2014-09-18 18:08:35.797217, event: throttled, op: pg_notify(2.4c6(15) epoch 11714) -1922> 2014-09-18 18:08:35.797320 7f4c9f251700 5 -- op tracker -- , seq: 86245, time: 2014-09-18 18:08:35.797270, event: all_read, op: pg_notify(2.4c6(15) epoch 11714) -1921> 2014-09-18 18:08:35.797326 7f4c9f251700 5 -- op tracker -- , seq: 86245, time: 2014-09-18 18:08:35.797299, event: dispatched, op: pg_notify(2.4c6(15) epoch 11714) -1920> 2014-09-18 18:08:35.797333 7f4c9f251700 5 -- op tracker -- , seq: 86245, time: 2014-09-18 18:08:35.797331, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11714) -1919> 2014-09-18 18:08:35.797341 7f4c9f251700 5 -- op tracker -- , seq: 86245, time: 2014-09-18 18:08:35.797339, event: started, op: pg_notify(2.4c6(15) epoch 11714) -1918> 2014-09-18 18:08:35.797359 7f4c9f251700 5 -- op tracker -- , seq: 86245, time: 2014-09-18 18:08:35.797357, event: done, op: pg_notify(2.4c6(15) epoch 11714) -1917> 2014-09-18 18:08:35.835752 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5116 ==== osd_map(11711..11711 src has 1..11711) v3 ==== 476+0+0 (251611104 0 0) 0x11bb7b00 con 0x1088aec0 -1916> 2014-09-18 18:08:35.835771 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11711,11711], i have 11714, src has [1,11711] -1915> 2014-09-18 18:08:35.835780 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5117 ==== pg_query(2.4e8 epoch 11711) v3 ==== 144+0+0 (1279071393 0 0) 0x21ced480 con 0x1088aec0 -1914> 2014-09-18 18:08:35.835795 7f4c9f251700 5 -- op tracker -- , seq: 86246, time: 2014-09-18 18:08:35.835658, event: header_read, op: pg_query(2.4e8 epoch 11711) -1913> 2014-09-18 18:08:35.835809 7f4c9f251700 5 -- op tracker -- , seq: 86246, time: 2014-09-18 18:08:35.835659, event: throttled, op: pg_query(2.4e8 epoch 11711) -1912> 2014-09-18 18:08:35.835819 7f4c9f251700 5 -- op tracker -- , seq: 86246, time: 2014-09-18 18:08:35.835697, event: all_read, op: pg_query(2.4e8 epoch 11711) -1911> 2014-09-18 18:08:35.835829 7f4c9f251700 5 -- op tracker -- , seq: 86246, time: 2014-09-18 18:08:35.835791, event: dispatched, op: pg_query(2.4e8 epoch 11711) -1910> 2014-09-18 18:08:35.835840 7f4c9f251700 5 -- op tracker -- , seq: 86246, time: 2014-09-18 18:08:35.835837, event: waiting_for_osdmap, op: pg_query(2.4e8 epoch 11711) -1909> 2014-09-18 18:08:35.835850 7f4c9f251700 5 -- op tracker -- , seq: 86246, time: 2014-09-18 18:08:35.835848, event: started, op: pg_query(2.4e8 epoch 11711) -1908> 2014-09-18 18:08:35.835890 7f4c9f251700 5 -- op tracker -- , seq: 86246, time: 2014-09-18 18:08:35.835888, event: done, op: pg_query(2.4e8 epoch 11711) -1907> 2014-09-18 18:08:35.835921 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.4e8(8) epoch 11714) v5 -- ?+0 0xf0ad2a0 con 0x1088aec0 -1906> 2014-09-18 18:08:35.836483 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5118 ==== pg_notify(2.62(44) epoch 11711) v5 ==== 3220+0+0 (3073724675 0 0) 0xf0ad2a0 con 0x1088aec0 -1905> 2014-09-18 18:08:35.836502 7f4c9f251700 5 -- op tracker -- , seq: 86247, time: 2014-09-18 18:08:35.836386, event: header_read, op: pg_notify(2.62(44) epoch 11711) -1904> 2014-09-18 18:08:35.836512 7f4c9f251700 5 -- op tracker -- , seq: 86247, time: 2014-09-18 18:08:35.836387, event: throttled, op: pg_notify(2.62(44) epoch 11711) -1903> 2014-09-18 18:08:35.836519 7f4c9f251700 5 -- op tracker -- , seq: 86247, time: 2014-09-18 18:08:35.836469, event: all_read, op: pg_notify(2.62(44) epoch 11711) -1902> 2014-09-18 18:08:35.836525 7f4c9f251700 5 -- op tracker -- , seq: 86247, time: 2014-09-18 18:08:35.836497, event: dispatched, op: pg_notify(2.62(44) epoch 11711) -1901> 2014-09-18 18:08:35.836530 7f4c9f251700 5 -- op tracker -- , seq: 86247, time: 2014-09-18 18:08:35.836529, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11711) -1900> 2014-09-18 18:08:35.836544 7f4c9f251700 5 -- op tracker -- , seq: 86247, time: 2014-09-18 18:08:35.836543, event: started, op: pg_notify(2.62(44) epoch 11711) -1899> 2014-09-18 18:08:35.836564 7f4c9f251700 5 -- op tracker -- , seq: 86247, time: 2014-09-18 18:08:35.836562, event: done, op: pg_notify(2.62(44) epoch 11711) -1898> 2014-09-18 18:08:35.839395 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5119 ==== pg_notify(2.3b6(10) epoch 11711) v5 ==== 1260+0+0 (4089318213 0 0) 0x21ceb0e0 con 0x1088aec0 -1897> 2014-09-18 18:08:35.839411 7f4c9f251700 5 -- op tracker -- , seq: 86248, time: 2014-09-18 18:08:35.839344, event: header_read, op: pg_notify(2.3b6(10) epoch 11711) -1896> 2014-09-18 18:08:35.839420 7f4c9f251700 5 -- op tracker -- , seq: 86248, time: 2014-09-18 18:08:35.839344, event: throttled, op: pg_notify(2.3b6(10) epoch 11711) -1895> 2014-09-18 18:08:35.839426 7f4c9f251700 5 -- op tracker -- , seq: 86248, time: 2014-09-18 18:08:35.839382, event: all_read, op: pg_notify(2.3b6(10) epoch 11711) -1894> 2014-09-18 18:08:35.839432 7f4c9f251700 5 -- op tracker -- , seq: 86248, time: 2014-09-18 18:08:35.839407, event: dispatched, op: pg_notify(2.3b6(10) epoch 11711) -1893> 2014-09-18 18:08:35.839439 7f4c9f251700 5 -- op tracker -- , seq: 86248, time: 2014-09-18 18:08:35.839437, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11711) -1892> 2014-09-18 18:08:35.839447 7f4c9f251700 5 -- op tracker -- , seq: 86248, time: 2014-09-18 18:08:35.839445, event: started, op: pg_notify(2.3b6(10) epoch 11711) -1891> 2014-09-18 18:08:35.839468 7f4c9f251700 5 -- op tracker -- , seq: 86248, time: 2014-09-18 18:08:35.839467, event: done, op: pg_notify(2.3b6(10) epoch 11711) -1890> 2014-09-18 18:08:35.840159 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5120 ==== pg_notify(2.6f8(10) epoch 11711) v5 ==== 1276+0+0 (2001591939 0 0) 0x21cec940 con 0x1088aec0 -1889> 2014-09-18 18:08:35.840187 7f4c9f251700 5 -- op tracker -- , seq: 86249, time: 2014-09-18 18:08:35.840098, event: header_read, op: pg_notify(2.6f8(10) epoch 11711) -1888> 2014-09-18 18:08:35.840195 7f4c9f251700 5 -- op tracker -- , seq: 86249, time: 2014-09-18 18:08:35.840099, event: throttled, op: pg_notify(2.6f8(10) epoch 11711) -1887> 2014-09-18 18:08:35.840210 7f4c9f251700 5 -- op tracker -- , seq: 86249, time: 2014-09-18 18:08:35.840144, event: all_read, op: pg_notify(2.6f8(10) epoch 11711) -1886> 2014-09-18 18:08:35.840216 7f4c9f251700 5 -- op tracker -- , seq: 86249, time: 2014-09-18 18:08:35.840182, event: dispatched, op: pg_notify(2.6f8(10) epoch 11711) -1885> 2014-09-18 18:08:35.840222 7f4c9f251700 5 -- op tracker -- , seq: 86249, time: 2014-09-18 18:08:35.840221, event: waiting_for_osdmap, op: pg_notify(2.6f8(10) epoch 11711) -1884> 2014-09-18 18:08:35.840230 7f4c9f251700 5 -- op tracker -- , seq: 86249, time: 2014-09-18 18:08:35.840228, event: started, op: pg_notify(2.6f8(10) epoch 11711) -1883> 2014-09-18 18:08:35.840247 7f4c9f251700 5 -- op tracker -- , seq: 86249, time: 2014-09-18 18:08:35.840246, event: done, op: pg_notify(2.6f8(10) epoch 11711) -1882> 2014-09-18 18:08:35.841239 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5121 ==== pg_notify(2.517(11),2.519(23) epoch 11711) v5 ==== 3274+0+0 (3418693015 0 0) 0x21cec580 con 0x1088aec0 -1881> 2014-09-18 18:08:35.841258 7f4c9f251700 5 -- op tracker -- , seq: 86250, time: 2014-09-18 18:08:35.841152, event: header_read, op: pg_notify(2.517(11),2.519(23) epoch 11711) -1880> 2014-09-18 18:08:35.841266 7f4c9f251700 5 -- op tracker -- , seq: 86250, time: 2014-09-18 18:08:35.841154, event: throttled, op: pg_notify(2.517(11),2.519(23) epoch 11711) -1879> 2014-09-18 18:08:35.841273 7f4c9f251700 5 -- op tracker -- , seq: 86250, time: 2014-09-18 18:08:35.841222, event: all_read, op: pg_notify(2.517(11),2.519(23) epoch 11711) -1878> 2014-09-18 18:08:35.841279 7f4c9f251700 5 -- op tracker -- , seq: 86250, time: 2014-09-18 18:08:35.841253, event: dispatched, op: pg_notify(2.517(11),2.519(23) epoch 11711) -1877> 2014-09-18 18:08:35.841286 7f4c9f251700 5 -- op tracker -- , seq: 86250, time: 2014-09-18 18:08:35.841284, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.519(23) epoch 11711) -1876> 2014-09-18 18:08:35.841293 7f4c9f251700 5 -- op tracker -- , seq: 86250, time: 2014-09-18 18:08:35.841292, event: started, op: pg_notify(2.517(11),2.519(23) epoch 11711) -1875> 2014-09-18 18:08:35.841317 7f4c9f251700 5 -- op tracker -- , seq: 86250, time: 2014-09-18 18:08:35.841316, event: done, op: pg_notify(2.517(11),2.519(23) epoch 11711) -1874> 2014-09-18 18:08:35.852163 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3693 ==== pg_notify(2.1bb(9) epoch 11714) v5 ==== 1189+0+0 (1726403941 0 0) 0xe146960 con 0xcbecec0 -1873> 2014-09-18 18:08:35.852182 7f4c9f251700 5 -- op tracker -- , seq: 86251, time: 2014-09-18 18:08:35.852081, event: header_read, op: pg_notify(2.1bb(9) epoch 11714) -1872> 2014-09-18 18:08:35.852193 7f4c9f251700 5 -- op tracker -- , seq: 86251, time: 2014-09-18 18:08:35.852083, event: throttled, op: pg_notify(2.1bb(9) epoch 11714) -1871> 2014-09-18 18:08:35.852199 7f4c9f251700 5 -- op tracker -- , seq: 86251, time: 2014-09-18 18:08:35.852127, event: all_read, op: pg_notify(2.1bb(9) epoch 11714) -1870> 2014-09-18 18:08:35.852205 7f4c9f251700 5 -- op tracker -- , seq: 86251, time: 2014-09-18 18:08:35.852178, event: dispatched, op: pg_notify(2.1bb(9) epoch 11714) -1869> 2014-09-18 18:08:35.852212 7f4c9f251700 5 -- op tracker -- , seq: 86251, time: 2014-09-18 18:08:35.852211, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11714) -1868> 2014-09-18 18:08:35.852229 7f4c9f251700 5 -- op tracker -- , seq: 86251, time: 2014-09-18 18:08:35.852227, event: started, op: pg_notify(2.1bb(9) epoch 11714) -1867> 2014-09-18 18:08:35.852248 7f4c9f251700 5 -- op tracker -- , seq: 86251, time: 2014-09-18 18:08:35.852246, event: done, op: pg_notify(2.1bb(9) epoch 11714) -1866> 2014-09-18 18:08:35.865436 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3694 ==== pg_notify(2.78f(58) epoch 11714) v5 ==== 3828+0+0 (2062822665 0 0) 0xe147c20 con 0xcbecec0 -1865> 2014-09-18 18:08:35.865463 7f4c9f251700 5 -- op tracker -- , seq: 86252, time: 2014-09-18 18:08:35.865359, event: header_read, op: pg_notify(2.78f(58) epoch 11714) -1864> 2014-09-18 18:08:35.865472 7f4c9f251700 5 -- op tracker -- , seq: 86252, time: 2014-09-18 18:08:35.865360, event: throttled, op: pg_notify(2.78f(58) epoch 11714) -1863> 2014-09-18 18:08:35.865476 7f4c9f251700 5 -- op tracker -- , seq: 86252, time: 2014-09-18 18:08:35.865422, event: all_read, op: pg_notify(2.78f(58) epoch 11714) -1862> 2014-09-18 18:08:35.865480 7f4c9f251700 5 -- op tracker -- , seq: 86252, time: 2014-09-18 18:08:35.865459, event: dispatched, op: pg_notify(2.78f(58) epoch 11714) -1861> 2014-09-18 18:08:35.865485 7f4c9f251700 5 -- op tracker -- , seq: 86252, time: 2014-09-18 18:08:35.865484, event: waiting_for_osdmap, op: pg_notify(2.78f(58) epoch 11714) -1860> 2014-09-18 18:08:35.865499 7f4c9f251700 5 -- op tracker -- , seq: 86252, time: 2014-09-18 18:08:35.865497, event: started, op: pg_notify(2.78f(58) epoch 11714) -1859> 2014-09-18 18:08:35.865517 7f4c9f251700 5 -- op tracker -- , seq: 86252, time: 2014-09-18 18:08:35.865516, event: done, op: pg_notify(2.78f(58) epoch 11714) -1858> 2014-09-18 18:08:35.865544 7f4c96a40700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.180902 6 0.000237 -1857> 2014-09-18 18:08:35.865697 7f4c96a40700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -1856> 2014-09-18 18:08:35.865964 7f4c96a40700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000268 0 0.000000 -1855> 2014-09-18 18:08:35.866065 7f4c96a40700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -1854> 2014-09-18 18:08:35.866114 7f4c96a40700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000048 0 0.000000 -1853> 2014-09-18 18:08:35.866125 7f4c96a40700 5 osd.42 pg_epoch: 11714 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -1852> 2014-09-18 18:08:35.866317 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3695 ==== pg_notify(2.78f(58) epoch 11714) v5 ==== 3828+0+0 (2062822665 0 0) 0xfc58580 con 0xcbecec0 -1851> 2014-09-18 18:08:35.866334 7f4c9f251700 5 -- op tracker -- , seq: 86253, time: 2014-09-18 18:08:35.866242, event: header_read, op: pg_notify(2.78f(58) epoch 11714) -1850> 2014-09-18 18:08:35.866344 7f4c9f251700 5 -- op tracker -- , seq: 86253, time: 2014-09-18 18:08:35.866243, event: throttled, op: pg_notify(2.78f(58) epoch 11714) -1849> 2014-09-18 18:08:35.866351 7f4c9f251700 5 -- op tracker -- , seq: 86253, time: 2014-09-18 18:08:35.866305, event: all_read, op: pg_notify(2.78f(58) epoch 11714) -1848> 2014-09-18 18:08:35.866357 7f4c9f251700 5 -- op tracker -- , seq: 86253, time: 2014-09-18 18:08:35.866330, event: dispatched, op: pg_notify(2.78f(58) epoch 11714) -1847> 2014-09-18 18:08:35.866363 7f4c9f251700 5 -- op tracker -- , seq: 86253, time: 2014-09-18 18:08:35.866362, event: waiting_for_osdmap, op: pg_notify(2.78f(58) epoch 11714) -1846> 2014-09-18 18:08:35.866371 7f4c9f251700 5 -- op tracker -- , seq: 86253, time: 2014-09-18 18:08:35.866369, event: started, op: pg_notify(2.78f(58) epoch 11714) -1845> 2014-09-18 18:08:35.866388 7f4c9f251700 5 -- op tracker -- , seq: 86253, time: 2014-09-18 18:08:35.866386, event: done, op: pg_notify(2.78f(58) epoch 11714) -1844> 2014-09-18 18:08:35.910339 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.27 10.242.42.174:0/21548 151 ==== osd_ping(ping e11714 stamp 2014-09-18 18:06:41.741223) v2 ==== 47+0+0 (2978866042 0 0) 0x146df2c0 con 0x107358c0 -1843> 2014-09-18 18:08:35.910381 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.174:0/21548 -- osd_ping(ping_reply e11714 stamp 2014-09-18 18:06:41.741223) v2 -- ?+0 0x11fd2ee0 con 0x107358c0 -1842> 2014-09-18 18:08:35.910596 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.27 10.242.42.174:0/21548 151 ==== osd_ping(ping e11714 stamp 2014-09-18 18:06:41.741223) v2 ==== 47+0+0 (2978866042 0 0) 0xf273c20 con 0x101cb4a0 -1841> 2014-09-18 18:08:35.910615 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.174:0/21548 -- osd_ping(ping_reply e11714 stamp 2014-09-18 18:06:41.741223) v2 -- ?+0 0x1421ed20 con 0x101cb4a0 -1840> 2014-09-18 18:08:35.948511 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3245 ==== pg_notify(2.519(4) epoch 11714) v5 ==== 962+0+0 (1536007413 0 0) 0x2b93e780 con 0x1395ec60 -1839> 2014-09-18 18:08:35.948536 7f4c9f251700 5 -- op tracker -- , seq: 86254, time: 2014-09-18 18:08:35.948430, event: header_read, op: pg_notify(2.519(4) epoch 11714) -1838> 2014-09-18 18:08:35.948552 7f4c9f251700 5 -- op tracker -- , seq: 86254, time: 2014-09-18 18:08:35.948431, event: throttled, op: pg_notify(2.519(4) epoch 11714) -1837> 2014-09-18 18:08:35.948562 7f4c9f251700 5 -- op tracker -- , seq: 86254, time: 2014-09-18 18:08:35.948491, event: all_read, op: pg_notify(2.519(4) epoch 11714) -1836> 2014-09-18 18:08:35.948573 7f4c9f251700 5 -- op tracker -- , seq: 86254, time: 2014-09-18 18:08:35.948530, event: dispatched, op: pg_notify(2.519(4) epoch 11714) -1835> 2014-09-18 18:08:35.948583 7f4c9f251700 5 -- op tracker -- , seq: 86254, time: 2014-09-18 18:08:35.948581, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11714) -1834> 2014-09-18 18:08:35.948596 7f4c9f251700 5 -- op tracker -- , seq: 86254, time: 2014-09-18 18:08:35.948593, event: started, op: pg_notify(2.519(4) epoch 11714) -1833> 2014-09-18 18:08:35.948637 7f4c9f251700 5 -- op tracker -- , seq: 86254, time: 2014-09-18 18:08:35.948634, event: done, op: pg_notify(2.519(4) epoch 11714) -1832> 2014-09-18 18:08:35.951527 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3246 ==== pg_notify(2.410(12) epoch 11714) v5 ==== 1376+0+0 (2325007120 0 0) 0x2b93ef00 con 0x1395ec60 -1831> 2014-09-18 18:08:35.951547 7f4c9f251700 5 -- op tracker -- , seq: 86255, time: 2014-09-18 18:08:35.951472, event: header_read, op: pg_notify(2.410(12) epoch 11714) -1830> 2014-09-18 18:08:35.951557 7f4c9f251700 5 -- op tracker -- , seq: 86255, time: 2014-09-18 18:08:35.951473, event: throttled, op: pg_notify(2.410(12) epoch 11714) -1829> 2014-09-18 18:08:35.951563 7f4c9f251700 5 -- op tracker -- , seq: 86255, time: 2014-09-18 18:08:35.951512, event: all_read, op: pg_notify(2.410(12) epoch 11714) -1828> 2014-09-18 18:08:35.951569 7f4c9f251700 5 -- op tracker -- , seq: 86255, time: 2014-09-18 18:08:35.951542, event: dispatched, op: pg_notify(2.410(12) epoch 11714) -1827> 2014-09-18 18:08:35.951576 7f4c9f251700 5 -- op tracker -- , seq: 86255, time: 2014-09-18 18:08:35.951574, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11714) -1826> 2014-09-18 18:08:35.951584 7f4c9f251700 5 -- op tracker -- , seq: 86255, time: 2014-09-18 18:08:35.951582, event: started, op: pg_notify(2.410(12) epoch 11714) -1825> 2014-09-18 18:08:35.951790 7f4c9f251700 5 -- op tracker -- , seq: 86255, time: 2014-09-18 18:08:35.951787, event: done, op: pg_notify(2.410(12) epoch 11714) -1824> 2014-09-18 18:08:35.952248 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3247 ==== pg_notify(2.186(12) epoch 11714) v5 ==== 1386+0+0 (2152571139 0 0) 0x2b93f0e0 con 0x1395ec60 -1823> 2014-09-18 18:08:35.952313 7f4c9f251700 5 -- op tracker -- , seq: 86256, time: 2014-09-18 18:08:35.952197, event: header_read, op: pg_notify(2.186(12) epoch 11714) -1822> 2014-09-18 18:08:35.952328 7f4c9f251700 5 -- op tracker -- , seq: 86256, time: 2014-09-18 18:08:35.952198, event: throttled, op: pg_notify(2.186(12) epoch 11714) -1821> 2014-09-18 18:08:35.952337 7f4c9f251700 5 -- op tracker -- , seq: 86256, time: 2014-09-18 18:08:35.952231, event: all_read, op: pg_notify(2.186(12) epoch 11714) -1820> 2014-09-18 18:08:35.952349 7f4c9f251700 5 -- op tracker -- , seq: 86256, time: 2014-09-18 18:08:35.952308, event: dispatched, op: pg_notify(2.186(12) epoch 11714) -1819> 2014-09-18 18:08:35.952361 7f4c9f251700 5 -- op tracker -- , seq: 86256, time: 2014-09-18 18:08:35.952358, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11714) -1818> 2014-09-18 18:08:35.952376 7f4c9f251700 5 -- op tracker -- , seq: 86256, time: 2014-09-18 18:08:35.952373, event: started, op: pg_notify(2.186(12) epoch 11714) -1817> 2014-09-18 18:08:35.952410 7f4c9f251700 5 -- op tracker -- , seq: 86256, time: 2014-09-18 18:08:35.952407, event: done, op: pg_notify(2.186(12) epoch 11714) -1816> 2014-09-18 18:08:35.953234 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3248 ==== pg_notify(2.74e(11) epoch 11714) v5 ==== 1303+0+0 (4147478532 0 0) 0x2b940760 con 0x1395ec60 -1815> 2014-09-18 18:08:35.953254 7f4c9f251700 5 -- op tracker -- , seq: 86257, time: 2014-09-18 18:08:35.953188, event: header_read, op: pg_notify(2.74e(11) epoch 11714) -1814> 2014-09-18 18:08:35.953266 7f4c9f251700 5 -- op tracker -- , seq: 86257, time: 2014-09-18 18:08:35.953189, event: throttled, op: pg_notify(2.74e(11) epoch 11714) -1813> 2014-09-18 18:08:35.953278 7f4c9f251700 5 -- op tracker -- , seq: 86257, time: 2014-09-18 18:08:35.953219, event: all_read, op: pg_notify(2.74e(11) epoch 11714) -1812> 2014-09-18 18:08:35.953288 7f4c9f251700 5 -- op tracker -- , seq: 86257, time: 2014-09-18 18:08:35.953249, event: dispatched, op: pg_notify(2.74e(11) epoch 11714) -1811> 2014-09-18 18:08:35.953298 7f4c9f251700 5 -- op tracker -- , seq: 86257, time: 2014-09-18 18:08:35.953296, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11714) -1810> 2014-09-18 18:08:35.953313 7f4c9f251700 5 -- op tracker -- , seq: 86257, time: 2014-09-18 18:08:35.953310, event: started, op: pg_notify(2.74e(11) epoch 11714) -1809> 2014-09-18 18:08:35.953342 7f4c9f251700 5 -- op tracker -- , seq: 86257, time: 2014-09-18 18:08:35.953340, event: done, op: pg_notify(2.74e(11) epoch 11714) -1808> 2014-09-18 18:08:36.023752 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3249 ==== pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11714) v5 ==== 8086+0+0 (2576952662 0 0) 0x2b9410c0 con 0x1395ec60 -1807> 2014-09-18 18:08:36.023934 7f4c9f251700 5 -- op tracker -- , seq: 86258, time: 2014-09-18 18:08:36.023489, event: header_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11714) -1806> 2014-09-18 18:08:36.023948 7f4c9f251700 5 -- op tracker -- , seq: 86258, time: 2014-09-18 18:08:36.023492, event: throttled, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11714) -1805> 2014-09-18 18:08:36.023956 7f4c9f251700 5 -- op tracker -- , seq: 86258, time: 2014-09-18 18:08:36.023641, event: all_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11714) -1804> 2014-09-18 18:08:36.023963 7f4c9f251700 5 -- op tracker -- , seq: 86258, time: 2014-09-18 18:08:36.023927, event: dispatched, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11714) -1803> 2014-09-18 18:08:36.023970 7f4c9f251700 5 -- op tracker -- , seq: 86258, time: 2014-09-18 18:08:36.023968, event: waiting_for_osdmap, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11714) -1802> 2014-09-18 18:08:36.023980 7f4c9f251700 5 -- op tracker -- , seq: 86258, time: 2014-09-18 18:08:36.023978, event: started, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11714) -1801> 2014-09-18 18:08:36.024076 7f4c9f251700 5 -- op tracker -- , seq: 86258, time: 2014-09-18 18:08:36.024074, event: done, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11714) -1800> 2014-09-18 18:08:36.024097 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3250 ==== pg_notify(2.799(9),2.4d0(10) epoch 11714) v5 ==== 2509+0+0 (840256725 0 0) 0x2b9412a0 con 0x1395ec60 -1799> 2014-09-18 18:08:36.024109 7f4c9f251700 5 -- op tracker -- , seq: 86259, time: 2014-09-18 18:08:36.023672, event: header_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11714) -1798> 2014-09-18 18:08:36.024116 7f4c9f251700 5 -- op tracker -- , seq: 86259, time: 2014-09-18 18:08:36.023673, event: throttled, op: pg_notify(2.799(9),2.4d0(10) epoch 11714) -1797> 2014-09-18 18:08:36.024122 7f4c9f251700 5 -- op tracker -- , seq: 86259, time: 2014-09-18 18:08:36.023735, event: all_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11714) -1796> 2014-09-18 18:08:36.024128 7f4c9f251700 5 -- op tracker -- , seq: 86259, time: 2014-09-18 18:08:36.024106, event: dispatched, op: pg_notify(2.799(9),2.4d0(10) epoch 11714) -1795> 2014-09-18 18:08:36.024135 7f4c9f251700 5 -- op tracker -- , seq: 86259, time: 2014-09-18 18:08:36.024133, event: waiting_for_osdmap, op: pg_notify(2.799(9),2.4d0(10) epoch 11714) -1794> 2014-09-18 18:08:36.024142 7f4c9f251700 5 -- op tracker -- , seq: 86259, time: 2014-09-18 18:08:36.024140, event: started, op: pg_notify(2.799(9),2.4d0(10) epoch 11714) -1793> 2014-09-18 18:08:36.024166 7f4c9f251700 5 -- op tracker -- , seq: 86259, time: 2014-09-18 18:08:36.024164, event: done, op: pg_notify(2.799(9),2.4d0(10) epoch 11714) -1792> 2014-09-18 18:08:36.094512 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.13 10.242.42.168:0/10879 696 ==== osd_ping(ping e11714 stamp 2014-09-18 18:08:35.961083) v2 ==== 47+0+0 (2855358655 0 0) 0x24f79e00 con 0x109f9de0 -1791> 2014-09-18 18:08:36.094551 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11714 stamp 2014-09-18 18:08:35.961083) v2 -- ?+0 0x146df2c0 con 0x109f9de0 -1790> 2014-09-18 18:08:36.094550 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.13 10.242.42.168:0/10879 696 ==== osd_ping(ping e11714 stamp 2014-09-18 18:08:35.961083) v2 ==== 47+0+0 (2855358655 0 0) 0x2b926780 con 0x140f7b80 -1789> 2014-09-18 18:08:36.094583 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11714 stamp 2014-09-18 18:08:35.961083) v2 -- ?+0 0xf273c20 con 0x140f7b80 -1788> 2014-09-18 18:08:36.139720 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2368 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0xe2678c0 con 0xdd8c940 -1787> 2014-09-18 18:08:36.139749 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -1786> 2014-09-18 18:08:36.139910 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2369 ==== pg_notify(2.2c7(139) epoch 11714) v5 ==== 8651+0+0 (3206652430 0 0) 0x19f37a40 con 0xdd8c940 -1785> 2014-09-18 18:08:36.139928 7f4c9f251700 5 -- op tracker -- , seq: 86260, time: 2014-09-18 18:08:36.139674, event: header_read, op: pg_notify(2.2c7(139) epoch 11714) -1784> 2014-09-18 18:08:36.139942 7f4c9f251700 5 -- op tracker -- , seq: 86260, time: 2014-09-18 18:08:36.139675, event: throttled, op: pg_notify(2.2c7(139) epoch 11714) -1783> 2014-09-18 18:08:36.139954 7f4c9f251700 5 -- op tracker -- , seq: 86260, time: 2014-09-18 18:08:36.139894, event: all_read, op: pg_notify(2.2c7(139) epoch 11714) -1782> 2014-09-18 18:08:36.139963 7f4c9f251700 5 -- op tracker -- , seq: 86260, time: 2014-09-18 18:08:36.139923, event: dispatched, op: pg_notify(2.2c7(139) epoch 11714) -1781> 2014-09-18 18:08:36.139973 7f4c9f251700 5 -- op tracker -- , seq: 86260, time: 2014-09-18 18:08:36.139971, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11714) -1780> 2014-09-18 18:08:36.139986 7f4c9f251700 5 -- op tracker -- , seq: 86260, time: 2014-09-18 18:08:36.139984, event: started, op: pg_notify(2.2c7(139) epoch 11714) -1779> 2014-09-18 18:08:36.140014 7f4c9f251700 5 -- op tracker -- , seq: 86260, time: 2014-09-18 18:08:36.140012, event: done, op: pg_notify(2.2c7(139) epoch 11714) -1778> 2014-09-18 18:08:36.140039 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2370 ==== pg_notify(2.457(21) epoch 11714) v5 ==== 1875+0+0 (4068168627 0 0) 0x19f36960 con 0xdd8c940 -1777> 2014-09-18 18:08:36.140055 7f4c9f251700 5 -- op tracker -- , seq: 86261, time: 2014-09-18 18:08:36.139930, event: header_read, op: pg_notify(2.457(21) epoch 11714) -1776> 2014-09-18 18:08:36.140066 7f4c9f251700 5 -- op tracker -- , seq: 86261, time: 2014-09-18 18:08:36.139931, event: throttled, op: pg_notify(2.457(21) epoch 11714) -1775> 2014-09-18 18:08:36.140074 7f4c9f251700 5 -- op tracker -- , seq: 86261, time: 2014-09-18 18:08:36.139978, event: all_read, op: pg_notify(2.457(21) epoch 11714) -1774> 2014-09-18 18:08:36.140084 7f4c9f251700 5 -- op tracker -- , seq: 86261, time: 2014-09-18 18:08:36.140051, event: dispatched, op: pg_notify(2.457(21) epoch 11714) -1773> 2014-09-18 18:08:36.140093 7f4c9f251700 5 -- op tracker -- , seq: 86261, time: 2014-09-18 18:08:36.140091, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11714) -1772> 2014-09-18 18:08:36.140105 7f4c9f251700 5 -- op tracker -- , seq: 86261, time: 2014-09-18 18:08:36.140102, event: started, op: pg_notify(2.457(21) epoch 11714) -1771> 2014-09-18 18:08:36.140124 7f4c9f251700 5 -- op tracker -- , seq: 86261, time: 2014-09-18 18:08:36.140123, event: done, op: pg_notify(2.457(21) epoch 11714) -1770> 2014-09-18 18:08:36.174614 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 974 ==== osd_map(11713..11713 src has 1..11714) v3 ==== 235+0+0 (1403665526 0 0) 0x12bf06c0 con 0x10a878c0 -1769> 2014-09-18 18:08:36.174642 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11713,11713], i have 11714, src has [1,11714] -1768> 2014-09-18 18:08:36.174679 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 975 ==== pg_notify(2.517(11),2.4c6(15) epoch 11713) v5 ==== 2834+0+0 (2216731667 0 0) 0x162501c0 con 0x10a878c0 -1767> 2014-09-18 18:08:36.174697 7f4c9f251700 5 -- op tracker -- , seq: 86262, time: 2014-09-18 18:08:36.174614, event: header_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11713) -1766> 2014-09-18 18:08:36.174714 7f4c9f251700 5 -- op tracker -- , seq: 86262, time: 2014-09-18 18:08:36.174614, event: throttled, op: pg_notify(2.517(11),2.4c6(15) epoch 11713) -1765> 2014-09-18 18:08:36.174725 7f4c9f251700 5 -- op tracker -- , seq: 86262, time: 2014-09-18 18:08:36.174665, event: all_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11713) -1764> 2014-09-18 18:08:36.174735 7f4c9f251700 5 -- op tracker -- , seq: 86262, time: 2014-09-18 18:08:36.174692, event: dispatched, op: pg_notify(2.517(11),2.4c6(15) epoch 11713) -1763> 2014-09-18 18:08:36.174745 7f4c9f251700 5 -- op tracker -- , seq: 86262, time: 2014-09-18 18:08:36.174742, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.4c6(15) epoch 11713) -1762> 2014-09-18 18:08:36.174758 7f4c9f251700 5 -- op tracker -- , seq: 86262, time: 2014-09-18 18:08:36.174754, event: started, op: pg_notify(2.517(11),2.4c6(15) epoch 11713) -1761> 2014-09-18 18:08:36.174789 7f4c9f251700 5 -- op tracker -- , seq: 86262, time: 2014-09-18 18:08:36.174787, event: done, op: pg_notify(2.517(11),2.4c6(15) epoch 11713) -1760> 2014-09-18 18:08:36.184003 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 976 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x12bf0000 con 0x10a878c0 -1759> 2014-09-18 18:08:36.184022 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -1758> 2014-09-18 18:08:36.184033 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 977 ==== pg_notify(2.517(11) epoch 11714) v5 ==== 1335+0+0 (83566388 0 0) 0x16251de0 con 0x10a878c0 -1757> 2014-09-18 18:08:36.184048 7f4c9f251700 5 -- op tracker -- , seq: 86263, time: 2014-09-18 18:08:36.183980, event: header_read, op: pg_notify(2.517(11) epoch 11714) -1756> 2014-09-18 18:08:36.184062 7f4c9f251700 5 -- op tracker -- , seq: 86263, time: 2014-09-18 18:08:36.183980, event: throttled, op: pg_notify(2.517(11) epoch 11714) -1755> 2014-09-18 18:08:36.184072 7f4c9f251700 5 -- op tracker -- , seq: 86263, time: 2014-09-18 18:08:36.184012, event: all_read, op: pg_notify(2.517(11) epoch 11714) -1754> 2014-09-18 18:08:36.184082 7f4c9f251700 5 -- op tracker -- , seq: 86263, time: 2014-09-18 18:08:36.184043, event: dispatched, op: pg_notify(2.517(11) epoch 11714) -1753> 2014-09-18 18:08:36.184093 7f4c9f251700 5 -- op tracker -- , seq: 86263, time: 2014-09-18 18:08:36.184090, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11714) -1752> 2014-09-18 18:08:36.184105 7f4c9f251700 5 -- op tracker -- , seq: 86263, time: 2014-09-18 18:08:36.184102, event: started, op: pg_notify(2.517(11) epoch 11714) -1751> 2014-09-18 18:08:36.184130 7f4c9f251700 5 -- op tracker -- , seq: 86263, time: 2014-09-18 18:08:36.184128, event: done, op: pg_notify(2.517(11) epoch 11714) -1750> 2014-09-18 18:08:36.184224 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 978 ==== pg_notify(2.4c6(15) epoch 11714) v5 ==== 1511+0+0 (2917107688 0 0) 0x1624fa40 con 0x10a878c0 -1749> 2014-09-18 18:08:36.184238 7f4c9f251700 5 -- op tracker -- , seq: 86264, time: 2014-09-18 18:08:36.184175, event: header_read, op: pg_notify(2.4c6(15) epoch 11714) -1748> 2014-09-18 18:08:36.184246 7f4c9f251700 5 -- op tracker -- , seq: 86264, time: 2014-09-18 18:08:36.184176, event: throttled, op: pg_notify(2.4c6(15) epoch 11714) -1747> 2014-09-18 18:08:36.184252 7f4c9f251700 5 -- op tracker -- , seq: 86264, time: 2014-09-18 18:08:36.184212, event: all_read, op: pg_notify(2.4c6(15) epoch 11714) -1746> 2014-09-18 18:08:36.184258 7f4c9f251700 5 -- op tracker -- , seq: 86264, time: 2014-09-18 18:08:36.184235, event: dispatched, op: pg_notify(2.4c6(15) epoch 11714) -1745> 2014-09-18 18:08:36.184264 7f4c9f251700 5 -- op tracker -- , seq: 86264, time: 2014-09-18 18:08:36.184263, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11714) -1744> 2014-09-18 18:08:36.184272 7f4c9f251700 5 -- op tracker -- , seq: 86264, time: 2014-09-18 18:08:36.184270, event: started, op: pg_notify(2.4c6(15) epoch 11714) -1743> 2014-09-18 18:08:36.184289 7f4c9f251700 5 -- op tracker -- , seq: 86264, time: 2014-09-18 18:08:36.184287, event: done, op: pg_notify(2.4c6(15) epoch 11714) -1742> 2014-09-18 18:08:36.206299 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5122 ==== pg_notify(2.11b(8),2.120(11) epoch 11711) v5 ==== 2507+0+0 (318614613 0 0) 0xe76dc20 con 0x1088aec0 -1741> 2014-09-18 18:08:36.206319 7f4c9f251700 5 -- op tracker -- , seq: 86265, time: 2014-09-18 18:08:36.206163, event: header_read, op: pg_notify(2.11b(8),2.120(11) epoch 11711) -1740> 2014-09-18 18:08:36.206331 7f4c9f251700 5 -- op tracker -- , seq: 86265, time: 2014-09-18 18:08:36.206164, event: throttled, op: pg_notify(2.11b(8),2.120(11) epoch 11711) -1739> 2014-09-18 18:08:36.206342 7f4c9f251700 5 -- op tracker -- , seq: 86265, time: 2014-09-18 18:08:36.206260, event: all_read, op: pg_notify(2.11b(8),2.120(11) epoch 11711) -1738> 2014-09-18 18:08:36.206353 7f4c9f251700 5 -- op tracker -- , seq: 86265, time: 2014-09-18 18:08:36.206315, event: dispatched, op: pg_notify(2.11b(8),2.120(11) epoch 11711) -1737> 2014-09-18 18:08:36.206364 7f4c9f251700 5 -- op tracker -- , seq: 86265, time: 2014-09-18 18:08:36.206362, event: waiting_for_osdmap, op: pg_notify(2.11b(8),2.120(11) epoch 11711) -1736> 2014-09-18 18:08:36.206377 7f4c9f251700 5 -- op tracker -- , seq: 86265, time: 2014-09-18 18:08:36.206375, event: started, op: pg_notify(2.11b(8),2.120(11) epoch 11711) -1735> 2014-09-18 18:08:36.206404 7f4c9f251700 5 -- op tracker -- , seq: 86265, time: 2014-09-18 18:08:36.206402, event: done, op: pg_notify(2.11b(8),2.120(11) epoch 11711) -1734> 2014-09-18 18:08:36.206808 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5123 ==== pg_notify(2.40a(162),2.410(12) epoch 11711) v5 ==== 11062+0+0 (2240113914 0 0) 0xe76c780 con 0x1088aec0 -1733> 2014-09-18 18:08:36.206823 7f4c9f251700 5 -- op tracker -- , seq: 86266, time: 2014-09-18 18:08:36.206650, event: header_read, op: pg_notify(2.40a(162),2.410(12) epoch 11711) -1732> 2014-09-18 18:08:36.206832 7f4c9f251700 5 -- op tracker -- , seq: 86266, time: 2014-09-18 18:08:36.206651, event: throttled, op: pg_notify(2.40a(162),2.410(12) epoch 11711) -1731> 2014-09-18 18:08:36.206838 7f4c9f251700 5 -- op tracker -- , seq: 86266, time: 2014-09-18 18:08:36.206796, event: all_read, op: pg_notify(2.40a(162),2.410(12) epoch 11711) -1730> 2014-09-18 18:08:36.206844 7f4c9f251700 5 -- op tracker -- , seq: 86266, time: 2014-09-18 18:08:36.206819, event: dispatched, op: pg_notify(2.40a(162),2.410(12) epoch 11711) -1729> 2014-09-18 18:08:36.206851 7f4c9f251700 5 -- op tracker -- , seq: 86266, time: 2014-09-18 18:08:36.206849, event: waiting_for_osdmap, op: pg_notify(2.40a(162),2.410(12) epoch 11711) -1728> 2014-09-18 18:08:36.206858 7f4c9f251700 5 -- op tracker -- , seq: 86266, time: 2014-09-18 18:08:36.206857, event: started, op: pg_notify(2.40a(162),2.410(12) epoch 11711) -1727> 2014-09-18 18:08:36.207053 7f4c9f251700 5 -- op tracker -- , seq: 86266, time: 2014-09-18 18:08:36.207051, event: done, op: pg_notify(2.40a(162),2.410(12) epoch 11711) -1726> 2014-09-18 18:08:36.207443 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5124 ==== pg_notify(2.179(10) epoch 11711) v5 ==== 1276+0+0 (2553177481 0 0) 0xe76d4a0 con 0x1088aec0 -1725> 2014-09-18 18:08:36.207457 7f4c9f251700 5 -- op tracker -- , seq: 86267, time: 2014-09-18 18:08:36.207402, event: header_read, op: pg_notify(2.179(10) epoch 11711) -1724> 2014-09-18 18:08:36.207465 7f4c9f251700 5 -- op tracker -- , seq: 86267, time: 2014-09-18 18:08:36.207403, event: throttled, op: pg_notify(2.179(10) epoch 11711) -1723> 2014-09-18 18:08:36.207471 7f4c9f251700 5 -- op tracker -- , seq: 86267, time: 2014-09-18 18:08:36.207433, event: all_read, op: pg_notify(2.179(10) epoch 11711) -1722> 2014-09-18 18:08:36.207477 7f4c9f251700 5 -- op tracker -- , seq: 86267, time: 2014-09-18 18:08:36.207454, event: dispatched, op: pg_notify(2.179(10) epoch 11711) -1721> 2014-09-18 18:08:36.207483 7f4c9f251700 5 -- op tracker -- , seq: 86267, time: 2014-09-18 18:08:36.207482, event: waiting_for_osdmap, op: pg_notify(2.179(10) epoch 11711) -1720> 2014-09-18 18:08:36.207490 7f4c9f251700 5 -- op tracker -- , seq: 86267, time: 2014-09-18 18:08:36.207489, event: started, op: pg_notify(2.179(10) epoch 11711) -1719> 2014-09-18 18:08:36.207507 7f4c9f251700 5 -- op tracker -- , seq: 86267, time: 2014-09-18 18:08:36.207506, event: done, op: pg_notify(2.179(10) epoch 11711) -1718> 2014-09-18 18:08:36.208547 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5125 ==== pg_notify(2.1a2(20) epoch 11711) v5 ==== 1798+0+0 (349747401 0 0) 0xe76c3c0 con 0x1088aec0 -1717> 2014-09-18 18:08:36.208559 7f4c9f251700 5 -- op tracker -- , seq: 86268, time: 2014-09-18 18:08:36.208501, event: header_read, op: pg_notify(2.1a2(20) epoch 11711) -1716> 2014-09-18 18:08:36.208567 7f4c9f251700 5 -- op tracker -- , seq: 86268, time: 2014-09-18 18:08:36.208502, event: throttled, op: pg_notify(2.1a2(20) epoch 11711) -1715> 2014-09-18 18:08:36.208573 7f4c9f251700 5 -- op tracker -- , seq: 86268, time: 2014-09-18 18:08:36.208536, event: all_read, op: pg_notify(2.1a2(20) epoch 11711) -1714> 2014-09-18 18:08:36.208579 7f4c9f251700 5 -- op tracker -- , seq: 86268, time: 2014-09-18 18:08:36.208556, event: dispatched, op: pg_notify(2.1a2(20) epoch 11711) -1713> 2014-09-18 18:08:36.208585 7f4c9f251700 5 -- op tracker -- , seq: 86268, time: 2014-09-18 18:08:36.208584, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11711) -1712> 2014-09-18 18:08:36.208592 7f4c9f251700 5 -- op tracker -- , seq: 86268, time: 2014-09-18 18:08:36.208591, event: started, op: pg_notify(2.1a2(20) epoch 11711) -1711> 2014-09-18 18:08:36.208607 7f4c9f251700 5 -- op tracker -- , seq: 86268, time: 2014-09-18 18:08:36.208606, event: done, op: pg_notify(2.1a2(20) epoch 11711) -1710> 2014-09-18 18:08:36.209893 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5126 ==== pg_notify(2.4c1(15) epoch 11711) v5 ==== 1507+0+0 (4026697715 0 0) 0xe76c5a0 con 0x1088aec0 -1709> 2014-09-18 18:08:36.209908 7f4c9f251700 5 -- op tracker -- , seq: 86269, time: 2014-09-18 18:08:36.209849, event: header_read, op: pg_notify(2.4c1(15) epoch 11711) -1708> 2014-09-18 18:08:36.209917 7f4c9f251700 5 -- op tracker -- , seq: 86269, time: 2014-09-18 18:08:36.209849, event: throttled, op: pg_notify(2.4c1(15) epoch 11711) -1707> 2014-09-18 18:08:36.209923 7f4c9f251700 5 -- op tracker -- , seq: 86269, time: 2014-09-18 18:08:36.209883, event: all_read, op: pg_notify(2.4c1(15) epoch 11711) -1706> 2014-09-18 18:08:36.209929 7f4c9f251700 5 -- op tracker -- , seq: 86269, time: 2014-09-18 18:08:36.209905, event: dispatched, op: pg_notify(2.4c1(15) epoch 11711) -1705> 2014-09-18 18:08:36.209935 7f4c9f251700 5 -- op tracker -- , seq: 86269, time: 2014-09-18 18:08:36.209934, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11711) -1704> 2014-09-18 18:08:36.209943 7f4c9f251700 5 -- op tracker -- , seq: 86269, time: 2014-09-18 18:08:36.209941, event: started, op: pg_notify(2.4c1(15) epoch 11711) -1703> 2014-09-18 18:08:36.209960 7f4c9f251700 5 -- op tracker -- , seq: 86269, time: 2014-09-18 18:08:36.209958, event: done, op: pg_notify(2.4c1(15) epoch 11711) -1702> 2014-09-18 18:08:36.225890 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1736 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0x14409b00 con 0x4900000 -1701> 2014-09-18 18:08:36.225926 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -1700> 2014-09-18 18:08:36.226040 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1737 ==== pg_notify(2.e0(79) epoch 11714) v5 ==== 5119+0+0 (1745445274 0 0) 0xf1ecd00 con 0x4900000 -1699> 2014-09-18 18:08:36.226052 7f4c9f251700 5 -- op tracker -- , seq: 86270, time: 2014-09-18 18:08:36.225919, event: header_read, op: pg_notify(2.e0(79) epoch 11714) -1698> 2014-09-18 18:08:36.226062 7f4c9f251700 5 -- op tracker -- , seq: 86270, time: 2014-09-18 18:08:36.225920, event: throttled, op: pg_notify(2.e0(79) epoch 11714) -1697> 2014-09-18 18:08:36.226070 7f4c9f251700 5 -- op tracker -- , seq: 86270, time: 2014-09-18 18:08:36.226028, event: all_read, op: pg_notify(2.e0(79) epoch 11714) -1696> 2014-09-18 18:08:36.226078 7f4c9f251700 5 -- op tracker -- , seq: 86270, time: 2014-09-18 18:08:36.226049, event: dispatched, op: pg_notify(2.e0(79) epoch 11714) -1695> 2014-09-18 18:08:36.226086 7f4c9f251700 5 -- op tracker -- , seq: 86270, time: 2014-09-18 18:08:36.226085, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11714) -1694> 2014-09-18 18:08:36.226094 7f4c9f251700 5 -- op tracker -- , seq: 86270, time: 2014-09-18 18:08:36.226093, event: started, op: pg_notify(2.e0(79) epoch 11714) -1693> 2014-09-18 18:08:36.226113 7f4c9f251700 5 -- op tracker -- , seq: 86270, time: 2014-09-18 18:08:36.226112, event: done, op: pg_notify(2.e0(79) epoch 11714) -1692> 2014-09-18 18:08:36.244899 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.24 10.242.42.172:0/5612 114 ==== osd_ping(ping e11714 stamp 2014-09-18 23:37:25.348513) v2 ==== 47+0+0 (2413294720 0 0) 0xf973680 con 0x110bb020 -1691> 2014-09-18 18:08:36.244896 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.24 10.242.42.172:0/5612 114 ==== osd_ping(ping e11714 stamp 2014-09-18 23:37:25.348513) v2 ==== 47+0+0 (2413294720 0 0) 0xe9ffe00 con 0x10a5bb80 -1690> 2014-09-18 18:08:36.244937 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11714 stamp 2014-09-18 23:37:25.348513) v2 -- ?+0 0x24f79e00 con 0x110bb020 -1689> 2014-09-18 18:08:36.244973 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/5612 -- osd_ping(ping_reply e11714 stamp 2014-09-18 23:37:25.348513) v2 -- ?+0 0x2b926780 con 0x10a5bb80 -1688> 2014-09-18 18:08:36.259299 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.45 10.242.42.178:0/7086 108 ==== osd_ping(ping e11714 stamp 2014-09-18 18:08:36.258672) v2 ==== 47+0+0 (198136198 0 0) 0xf1e74a0 con 0x10a5d9c0 -1687> 2014-09-18 18:08:36.259303 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.45 10.242.42.178:0/7086 108 ==== osd_ping(ping e11714 stamp 2014-09-18 18:08:36.258672) v2 ==== 47+0+0 (198136198 0 0) 0xf80f0e0 con 0x110bb700 -1686> 2014-09-18 18:08:36.259379 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11714 stamp 2014-09-18 18:08:36.258672) v2 -- ?+0 0xe9ffe00 con 0x10a5d9c0 -1685> 2014-09-18 18:08:36.259433 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/7086 -- osd_ping(ping_reply e11714 stamp 2014-09-18 18:08:36.258672) v2 -- ?+0 0xf973680 con 0x110bb700 -1684> 2014-09-18 18:08:36.290743 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.6 10.242.42.166:0/22481 442 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:36.292857) v2 ==== 47+0+0 (539218883 0 0) 0x12c10000 con 0x10201700 -1683> 2014-09-18 18:08:36.290763 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.166:0/22481 -- osd_ping(ping_reply e11714 stamp 2014-09-18 18:08:36.292857) v2 -- ?+0 0xf80f0e0 con 0x10201700 -1682> 2014-09-18 18:08:36.291088 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.6 10.242.42.166:0/22481 442 ==== osd_ping(ping e11711 stamp 2014-09-18 18:08:36.292857) v2 ==== 47+0+0 (539218883 0 0) 0x100ef860 con 0x14393080 -1681> 2014-09-18 18:08:36.291108 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.166:0/22481 -- osd_ping(ping_reply e11714 stamp 2014-09-18 18:08:36.292857) v2 -- ?+0 0xf1e74a0 con 0x14393080 -1680> 2014-09-18 18:08:36.379736 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1291 ==== pg_notify(2.4f6(15) epoch 11714) v5 ==== 1543+0+0 (3917634359 0 0) 0xe1a50c0 con 0x1f102840 -1679> 2014-09-18 18:08:36.379763 7f4c9f251700 5 -- op tracker -- , seq: 86271, time: 2014-09-18 18:08:36.379643, event: header_read, op: pg_notify(2.4f6(15) epoch 11714) -1678> 2014-09-18 18:08:36.379773 7f4c9f251700 5 -- op tracker -- , seq: 86271, time: 2014-09-18 18:08:36.379646, event: throttled, op: pg_notify(2.4f6(15) epoch 11714) -1677> 2014-09-18 18:08:36.379799 7f4c9f251700 5 -- op tracker -- , seq: 86271, time: 2014-09-18 18:08:36.379702, event: all_read, op: pg_notify(2.4f6(15) epoch 11714) -1676> 2014-09-18 18:08:36.379803 7f4c9f251700 5 -- op tracker -- , seq: 86271, time: 2014-09-18 18:08:36.379756, event: dispatched, op: pg_notify(2.4f6(15) epoch 11714) -1675> 2014-09-18 18:08:36.379808 7f4c9f251700 5 -- op tracker -- , seq: 86271, time: 2014-09-18 18:08:36.379806, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11714) -1674> 2014-09-18 18:08:36.379818 7f4c9f251700 5 -- op tracker -- , seq: 86271, time: 2014-09-18 18:08:36.379817, event: started, op: pg_notify(2.4f6(15) epoch 11714) -1673> 2014-09-18 18:08:36.379842 7f4c9f251700 5 -- op tracker -- , seq: 86271, time: 2014-09-18 18:08:36.379841, event: done, op: pg_notify(2.4f6(15) epoch 11714) -1672> 2014-09-18 18:08:36.391100 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.22 10.242.42.172:0/37668 444 ==== osd_ping(ping e11714 stamp 2014-09-18 23:37:25.495265) v2 ==== 47+0+0 (2508318917 0 0) 0xfae9a20 con 0x109caf20 -1671> 2014-09-18 18:08:36.391130 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11714 stamp 2014-09-18 23:37:25.495265) v2 -- ?+0 0x12c10000 con 0x109caf20 -1670> 2014-09-18 18:08:36.391345 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.22 10.242.42.172:0/37668 444 ==== osd_ping(ping e11714 stamp 2014-09-18 23:37:25.495265) v2 ==== 47+0+0 (2508318917 0 0) 0x2b986d20 con 0x14393340 -1669> 2014-09-18 18:08:36.391366 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/37668 -- osd_ping(ping_reply e11714 stamp 2014-09-18 23:37:25.495265) v2 -- ?+0 0x100ef860 con 0x14393340 -1668> 2014-09-18 18:08:36.582026 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 350 ==== osd_map(11714..11714 src has 1..11714) v3 ==== 1397+0+0 (2883508531 0 0) 0xe9393c0 con 0x12562260 -1667> 2014-09-18 18:08:36.582065 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11714,11714], i have 11714, src has [1,11714] -1666> 2014-09-18 18:08:36.582095 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 351 ==== pg_notify(2.719(17) epoch 11714) v5 ==== 1601+0+0 (986720234 0 0) 0x11739fe0 con 0x12562260 -1665> 2014-09-18 18:08:36.582108 7f4c9f251700 5 -- op tracker -- , seq: 86272, time: 2014-09-18 18:08:36.581927, event: header_read, op: pg_notify(2.719(17) epoch 11714) -1664> 2014-09-18 18:08:36.582120 7f4c9f251700 5 -- op tracker -- , seq: 86272, time: 2014-09-18 18:08:36.581927, event: throttled, op: pg_notify(2.719(17) epoch 11714) -1663> 2014-09-18 18:08:36.582125 7f4c9f251700 5 -- op tracker -- , seq: 86272, time: 2014-09-18 18:08:36.581974, event: all_read, op: pg_notify(2.719(17) epoch 11714) -1662> 2014-09-18 18:08:36.582134 7f4c9f251700 5 -- op tracker -- , seq: 86272, time: 2014-09-18 18:08:36.582103, event: dispatched, op: pg_notify(2.719(17) epoch 11714) -1661> 2014-09-18 18:08:36.582144 7f4c9f251700 5 -- op tracker -- , seq: 86272, time: 2014-09-18 18:08:36.582143, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11714) -1660> 2014-09-18 18:08:36.582154 7f4c9f251700 5 -- op tracker -- , seq: 86272, time: 2014-09-18 18:08:36.582152, event: started, op: pg_notify(2.719(17) epoch 11714) -1659> 2014-09-18 18:08:36.582177 7f4c9f251700 5 -- op tracker -- , seq: 86272, time: 2014-09-18 18:08:36.582175, event: done, op: pg_notify(2.719(17) epoch 11714) -1658> 2014-09-18 18:08:36.682185 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 304 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xe338880 con 0x140f95a0 -1657> 2014-09-18 18:08:36.682357 7f4c9f251700 3 osd.42 11714 handle_osd_map epochs [11715,11715], i have 11714, src has [1,11715] -1656> 2014-09-18 18:08:36.682618 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 33 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x15e333c0 con 0x111771e0 -1655> 2014-09-18 18:08:36.685741 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 305 ==== pg_query(2.30f epoch 11715) v3 ==== 144+0+0 (1375762752 0 0) 0xe2870e0 con 0x140f95a0 -1654> 2014-09-18 18:08:36.685762 7f4c9f251700 5 -- op tracker -- , seq: 86273, time: 2014-09-18 18:08:36.682185, event: header_read, op: pg_query(2.30f epoch 11715) -1653> 2014-09-18 18:08:36.685775 7f4c9f251700 5 -- op tracker -- , seq: 86273, time: 2014-09-18 18:08:36.682186, event: throttled, op: pg_query(2.30f epoch 11715) -1652> 2014-09-18 18:08:36.685783 7f4c9f251700 5 -- op tracker -- , seq: 86273, time: 2014-09-18 18:08:36.682210, event: all_read, op: pg_query(2.30f epoch 11715) -1651> 2014-09-18 18:08:36.685790 7f4c9f251700 5 -- op tracker -- , seq: 86273, time: 2014-09-18 18:08:36.685757, event: dispatched, op: pg_query(2.30f epoch 11715) -1650> 2014-09-18 18:08:36.685797 7f4c9f251700 5 -- op tracker -- , seq: 86273, time: 2014-09-18 18:08:36.685795, event: waiting_for_osdmap, op: pg_query(2.30f epoch 11715) -1649> 2014-09-18 18:08:36.685867 7f4c9f251700 5 -- op tracker -- , seq: 86273, time: 2014-09-18 18:08:36.685864, event: started, op: pg_query(2.30f epoch 11715) -1648> 2014-09-18 18:08:36.685884 7f4c9f251700 5 -- op tracker -- , seq: 86273, time: 2014-09-18 18:08:36.685882, event: done, op: pg_query(2.30f epoch 11715) -1647> 2014-09-18 18:08:36.685897 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3696 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xef593c0 con 0xcbecec0 -1646> 2014-09-18 18:08:36.685911 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1645> 2014-09-18 18:08:36.685917 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 55 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xe337d40 con 0x1395f340 -1644> 2014-09-18 18:08:36.685926 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1643> 2014-09-18 18:08:36.685931 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3318 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x10d19840 con 0x4900580 -1642> 2014-09-18 18:08:36.685941 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1641> 2014-09-18 18:08:36.685946 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2608 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x1440b180 con 0x4900b00 -1640> 2014-09-18 18:08:36.685981 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1639> 2014-09-18 18:08:36.685986 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1276 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xeeeb840 con 0xe8db760 -1638> 2014-09-18 18:08:36.685996 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1637> 2014-09-18 18:08:36.686001 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 306 ==== pg_notify(2.763(56) epoch 11715) v5 ==== 3866+0+0 (3924089360 0 0) 0xe795680 con 0x140f95a0 -1636> 2014-09-18 18:08:36.686013 7f4c9f251700 5 -- op tracker -- , seq: 86274, time: 2014-09-18 18:08:36.684982, event: header_read, op: pg_notify(2.763(56) epoch 11715) -1635> 2014-09-18 18:08:36.686022 7f4c9f251700 5 -- op tracker -- , seq: 86274, time: 2014-09-18 18:08:36.684983, event: throttled, op: pg_notify(2.763(56) epoch 11715) -1634> 2014-09-18 18:08:36.686028 7f4c9f251700 5 -- op tracker -- , seq: 86274, time: 2014-09-18 18:08:36.685103, event: all_read, op: pg_notify(2.763(56) epoch 11715) -1633> 2014-09-18 18:08:36.686034 7f4c9f251700 5 -- op tracker -- , seq: 86274, time: 2014-09-18 18:08:36.686009, event: dispatched, op: pg_notify(2.763(56) epoch 11715) -1632> 2014-09-18 18:08:36.686041 7f4c9f251700 5 -- op tracker -- , seq: 86274, time: 2014-09-18 18:08:36.686039, event: waiting_for_osdmap, op: pg_notify(2.763(56) epoch 11715) -1631> 2014-09-18 18:08:36.686049 7f4c9f251700 5 -- op tracker -- , seq: 86274, time: 2014-09-18 18:08:36.686047, event: started, op: pg_notify(2.763(56) epoch 11715) -1630> 2014-09-18 18:08:36.686254 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763(unlocked)] enter Initial -1629> 2014-09-18 18:08:36.686377 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11504/11504/11493) [9,45,27]/[45] r=-1 lpr=11504 pi=9903-11503/23 crt=6840'27412 remapped NOTIFY] exit Started/Stray 371.049843 633 0.034482 -1628> 2014-09-18 18:08:36.686405 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11504/11504/11493) [9,45,27]/[45] r=-1 lpr=11504 pi=9903-11503/23 crt=6840'27412 remapped NOTIFY] exit Started 371.050028 0 0.000000 -1627> 2014-09-18 18:08:36.686420 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11504/11504/11493) [9,45,27]/[45] r=-1 lpr=11504 pi=9903-11503/23 crt=6840'27412 remapped NOTIFY] enter Reset -1626> 2014-09-18 18:08:36.686436 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=0 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] exit Initial 0.000181 0 0.000000 -1625> 2014-09-18 18:08:36.686457 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=0 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] enter Reset -1624> 2014-09-18 18:08:36.686469 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] exit Reset 0.000012 1 0.000033 -1623> 2014-09-18 18:08:36.686478 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] enter Started -1622> 2014-09-18 18:08:36.686486 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] enter Start -1621> 2014-09-18 18:08:36.686493 7f4c9f251700 1 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] state: transitioning to Primary -1620> 2014-09-18 18:08:36.686502 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] exit Start 0.000016 0 0.000000 -1619> 2014-09-18 18:08:36.686496 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11715/11715/11493) [42,32,2]/[45] r=-1 lpr=11715 pi=9903-11714/24 crt=6840'27412 remapped NOTIFY] exit Reset 0.000075 1 0.000083 -1618> 2014-09-18 18:08:36.686511 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] enter Started/Primary -1617> 2014-09-18 18:08:36.686513 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11715/11715/11493) [42,32,2]/[45] r=-1 lpr=11715 pi=9903-11714/24 crt=6840'27412 remapped NOTIFY] enter Started -1616> 2014-09-18 18:08:36.686519 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1615> 2014-09-18 18:08:36.686533 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11715/11715/11493) [42,32,2]/[45] r=-1 lpr=11715 pi=9903-11714/24 crt=6840'27412 remapped NOTIFY] enter Start -1614> 2014-09-18 18:08:36.686549 7f4c9f251700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1613> 2014-09-18 18:08:36.686604 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6817/19139 -- osd_map(11711..11715 src has 1..11715) v3 -- ?+0 0x1533a480 con 0x10f60520 -1612> 2014-09-18 18:08:36.686556 7f4c97241700 1 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11715/11715/11493) [42,32,2]/[45] r=-1 lpr=11715 pi=9903-11714/24 crt=6840'27412 remapped NOTIFY] state: transitioning to Stray -1611> 2014-09-18 18:08:36.686634 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11715/11715/11493) [42,32,2]/[45] r=-1 lpr=11715 pi=9903-11714/24 crt=6840'27412 remapped NOTIFY] exit Start 0.000102 0 0.000000 -1610> 2014-09-18 18:08:36.686654 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.3c6( empty lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11350/9947 11715/11715/11493) [42,32,2]/[45] r=-1 lpr=11715 pi=9903-11714/24 crt=6840'27412 remapped NOTIFY] enter Started/Stray -1609> 2014-09-18 18:08:36.686665 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11490/11490/11490) [9,42,27] r=1 lpr=11490 pi=7889-11489/22 crt=0'0 active] exit Started/ReplicaActive/RepNotRecovering 385.042142 673 0.000889 -1608> 2014-09-18 18:08:36.686679 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11490/11490/11490) [9,42,27] r=1 lpr=11490 pi=7889-11489/22 crt=0'0 active] exit Started/ReplicaActive 385.042163 0 0.000000 -1607> 2014-09-18 18:08:36.686689 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11490/11490/11490) [9,42,27] r=1 lpr=11490 pi=7889-11489/22 crt=0'0 active] exit Started 386.053562 0 0.000000 -1606> 2014-09-18 18:08:36.686697 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11490/11490/11490) [9,42,27] r=1 lpr=11490 pi=7889-11489/22 crt=0'0 active] enter Reset -1605> 2014-09-18 18:08:36.686833 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6817/19139 -- pg_query(2.763 epoch 11715) v3 -- ?+0 0xf2b8ee0 con 0x10f60520 -1604> 2014-09-18 18:08:36.687074 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] exit Reset 0.000377 1 0.000060 -1603> 2014-09-18 18:08:36.687089 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] enter Started -1602> 2014-09-18 18:08:36.687115 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] enter Start -1601> 2014-09-18 18:08:36.687122 7f4c96a40700 1 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] state: transitioning to Stray -1600> 2014-09-18 18:08:36.687143 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0x1533c640 con 0x140f95a0 -1599> 2014-09-18 18:08:36.687269 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_query(2.763 epoch 11715) v3 -- ?+0 0xf2b8b20 con 0x140f95a0 -1598> 2014-09-18 18:08:36.687378 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0x24dbc000 con 0xdd8c940 -1597> 2014-09-18 18:08:36.687447 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- pg_query(2.763 epoch 11715) v3 -- ?+0 0xf2b8760 con 0xdd8c940 -1596> 2014-09-18 18:08:36.687544 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6835/46756 -- osd_map(11711..11715 src has 1..11715) v3 -- ?+0 0x24dbed00 con 0x142f8260 -1595> 2014-09-18 18:08:36.687562 7f4c9f251700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6835/46756 -- pg_query(2.763 epoch 11715) v3 -- ?+0 0xf6dffe0 con 0x142f8260 -1594> 2014-09-18 18:08:36.687131 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] exit Start 0.000016 0 0.000000 -1593> 2014-09-18 18:08:36.687359 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11504/11516/11015) [6,42,27]/[6,42] r=1 lpr=11516 pi=9903-11515/13 luod=0'0 crt=6775'27542 lcod 0'0 active+remapped] exit Started/ReplicaActive/RepNotRecovering 355.114710 591 0.000992 -1592> 2014-09-18 18:08:36.687655 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11504/11516/11015) [6,42,27]/[6,42] r=1 lpr=11516 pi=9903-11515/13 luod=0'0 crt=6775'27542 lcod 0'0 active+remapped] exit Started/ReplicaActive 355.115018 0 0.000000 -1591> 2014-09-18 18:08:36.687611 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] enter Started/Stray -1590> 2014-09-18 18:08:36.687671 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11504/11516/11015) [6,42,27]/[6,42] r=1 lpr=11516 pi=9903-11515/13 luod=0'0 crt=6775'27542 lcod 0'0 active+remapped] exit Started 358.038625 0 0.000000 -1589> 2014-09-18 18:08:36.687683 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11504/11516/11015) [6,42,27]/[6,42] r=1 lpr=11516 pi=9903-11515/13 luod=0'0 crt=6775'27542 lcod 0'0 active+remapped] enter Reset -1588> 2014-09-18 18:08:36.688003 7f4c9f251700 5 -- op tracker -- , seq: 86274, time: 2014-09-18 18:08:36.687997, event: done, op: pg_notify(2.763 epoch 11715) -1587> 2014-09-18 18:08:36.688041 7f4ca0a54700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1586> 2014-09-18 18:08:36.688038 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3697 ==== pg_notify(2.4f6(12) epoch 11715) v5 ==== 1382+0+0 (2424649588 0 0) 0xfc58760 con 0xcbecec0 -1585> 2014-09-18 18:08:36.688057 7f4c9f251700 5 -- op tracker -- , seq: 86275, time: 2014-09-18 18:08:36.683928, event: header_read, op: pg_notify(2.4f6(12) epoch 11715) -1584> 2014-09-18 18:08:36.688074 7f4c9f251700 5 -- op tracker -- , seq: 86275, time: 2014-09-18 18:08:36.683929, event: throttled, op: pg_notify(2.4f6(12) epoch 11715) -1583> 2014-09-18 18:08:36.688086 7f4c9f251700 5 -- op tracker -- , seq: 86275, time: 2014-09-18 18:08:36.683966, event: all_read, op: pg_notify(2.4f6(12) epoch 11715) -1582> 2014-09-18 18:08:36.688097 7f4c9f251700 5 -- op tracker -- , seq: 86275, time: 2014-09-18 18:08:36.688052, event: dispatched, op: pg_notify(2.4f6(12) epoch 11715) -1581> 2014-09-18 18:08:36.688119 7f4c9f251700 5 -- op tracker -- , seq: 86275, time: 2014-09-18 18:08:36.688117, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11715) -1580> 2014-09-18 18:08:36.688133 7f4c9f251700 5 -- op tracker -- , seq: 86275, time: 2014-09-18 18:08:36.688131, event: started, op: pg_notify(2.4f6(12) epoch 11715) -1579> 2014-09-18 18:08:36.688296 7f4c9f251700 5 -- op tracker -- , seq: 86275, time: 2014-09-18 18:08:36.688292, event: done, op: pg_notify(2.4f6(12) epoch 11715) -1578> 2014-09-18 18:08:36.688316 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 44 ==== osd_map(11711..11715 src has 1..11715) v3 ==== 5778+0+0 (777227498 0 0) 0x12330000 con 0x142f8260 -1577> 2014-09-18 18:08:36.688328 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11711,11715], i have 11715, src has [1,11715] -1576> 2014-09-18 18:08:36.688337 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 948 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xef59a80 con 0x1395f8c0 -1575> 2014-09-18 18:08:36.688346 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1574> 2014-09-18 18:08:36.688351 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 56 ==== pg_notify(2.1d8(9) epoch 11715) v5 ==== 1261+0+0 (2562675737 0 0) 0xf1e9de0 con 0x1395f340 -1573> 2014-09-18 18:08:36.688361 7f4c9f251700 5 -- op tracker -- , seq: 86276, time: 2014-09-18 18:08:36.683919, event: header_read, op: pg_notify(2.1d8(9) epoch 11715) -1572> 2014-09-18 18:08:36.688368 7f4c9f251700 5 -- op tracker -- , seq: 86276, time: 2014-09-18 18:08:36.683919, event: throttled, op: pg_notify(2.1d8(9) epoch 11715) -1571> 2014-09-18 18:08:36.688374 7f4c9f251700 5 -- op tracker -- , seq: 86276, time: 2014-09-18 18:08:36.683964, event: all_read, op: pg_notify(2.1d8(9) epoch 11715) -1570> 2014-09-18 18:08:36.688380 7f4c9f251700 5 -- op tracker -- , seq: 86276, time: 2014-09-18 18:08:36.688358, event: dispatched, op: pg_notify(2.1d8(9) epoch 11715) -1569> 2014-09-18 18:08:36.688422 7f4c9f251700 5 -- op tracker -- , seq: 86276, time: 2014-09-18 18:08:36.688421, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11715) -1568> 2014-09-18 18:08:36.688428 7f4c9f251700 5 -- op tracker -- , seq: 86276, time: 2014-09-18 18:08:36.688427, event: started, op: pg_notify(2.1d8(9) epoch 11715) -1567> 2014-09-18 18:08:36.688442 7f4c9f251700 5 -- op tracker -- , seq: 86276, time: 2014-09-18 18:08:36.688440, event: done, op: pg_notify(2.1d8(9) epoch 11715) -1566> 2014-09-18 18:08:36.688453 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.45 10.242.42.178:6817/7086 1 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x2db45b00 con 0x1395f1e0 -1565> 2014-09-18 18:08:36.688461 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1564> 2014-09-18 18:08:36.688465 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 549 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x12330240 con 0x1395ef20 -1563> 2014-09-18 18:08:36.688471 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1562> 2014-09-18 18:08:36.688475 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 6 ==== pg_notify(2.120(15) epoch 11715) v5 ==== 1535+0+0 (962360391 0 0) 0xf2b23c0 con 0x12b3b2e0 -1561> 2014-09-18 18:08:36.688482 7f4c9f251700 5 -- op tracker -- , seq: 86277, time: 2014-09-18 18:08:36.687731, event: header_read, op: pg_notify(2.120(15) epoch 11715) -1560> 2014-09-18 18:08:36.688487 7f4c9f251700 5 -- op tracker -- , seq: 86277, time: 2014-09-18 18:08:36.687748, event: throttled, op: pg_notify(2.120(15) epoch 11715) -1559> 2014-09-18 18:08:36.688491 7f4c9f251700 5 -- op tracker -- , seq: 86277, time: 2014-09-18 18:08:36.687810, event: all_read, op: pg_notify(2.120(15) epoch 11715) -1558> 2014-09-18 18:08:36.688495 7f4c9f251700 5 -- op tracker -- , seq: 86277, time: 2014-09-18 18:08:36.688480, event: dispatched, op: pg_notify(2.120(15) epoch 11715) -1557> 2014-09-18 18:08:36.688499 7f4c9f251700 5 -- op tracker -- , seq: 86277, time: 2014-09-18 18:08:36.688498, event: waiting_for_osdmap, op: pg_notify(2.120(15) epoch 11715) -1556> 2014-09-18 18:08:36.688504 7f4c9f251700 5 -- op tracker -- , seq: 86277, time: 2014-09-18 18:08:36.688503, event: started, op: pg_notify(2.120(15) epoch 11715) -1555> 2014-09-18 18:08:36.688516 7f4c9f251700 5 -- op tracker -- , seq: 86277, time: 2014-09-18 18:08:36.688515, event: done, op: pg_notify(2.120(15) epoch 11715) -1554> 2014-09-18 18:08:36.688534 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2371 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x24dbc000 con 0xdd8c940 -1553> 2014-09-18 18:08:36.688550 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1552> 2014-09-18 18:08:36.688561 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3319 ==== pg_notify(2.4f6(184) epoch 11715) v5 ==== 11296+0+0 (4150763200 0 0) 0xf2b5a20 con 0x4900580 -1551> 2014-09-18 18:08:36.688575 7f4c9f251700 5 -- op tracker -- , seq: 86278, time: 2014-09-18 18:08:36.683782, event: header_read, op: pg_notify(2.4f6(184) epoch 11715) -1550> 2014-09-18 18:08:36.688584 7f4c9f251700 5 -- op tracker -- , seq: 86278, time: 2014-09-18 18:08:36.683783, event: throttled, op: pg_notify(2.4f6(184) epoch 11715) -1549> 2014-09-18 18:08:36.688590 7f4c9f251700 5 -- op tracker -- , seq: 86278, time: 2014-09-18 18:08:36.684006, event: all_read, op: pg_notify(2.4f6(184) epoch 11715) -1548> 2014-09-18 18:08:36.688596 7f4c9f251700 5 -- op tracker -- , seq: 86278, time: 2014-09-18 18:08:36.688572, event: dispatched, op: pg_notify(2.4f6(184) epoch 11715) -1547> 2014-09-18 18:08:36.688604 7f4c9f251700 5 -- op tracker -- , seq: 86278, time: 2014-09-18 18:08:36.688603, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11715) -1546> 2014-09-18 18:08:36.688612 7f4c9f251700 5 -- op tracker -- , seq: 86278, time: 2014-09-18 18:08:36.688611, event: started, op: pg_notify(2.4f6(184) epoch 11715) -1545> 2014-09-18 18:08:36.688626 7f4c9f251700 5 -- op tracker -- , seq: 86278, time: 2014-09-18 18:08:36.688624, event: done, op: pg_notify(2.4f6(184) epoch 11715) -1544> 2014-09-18 18:08:36.688774 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2609 ==== pg_notify(2.71c(12) epoch 11715) v5 ==== 1424+0+0 (3499071784 0 0) 0xf05b0e0 con 0x4900b00 -1543> 2014-09-18 18:08:36.688789 7f4c9f251700 5 -- op tracker -- , seq: 86279, time: 2014-09-18 18:08:36.682710, event: header_read, op: pg_notify(2.71c(12) epoch 11715) -1542> 2014-09-18 18:08:36.688798 7f4c9f251700 5 -- op tracker -- , seq: 86279, time: 2014-09-18 18:08:36.682711, event: throttled, op: pg_notify(2.71c(12) epoch 11715) -1541> 2014-09-18 18:08:36.688805 7f4c9f251700 5 -- op tracker -- , seq: 86279, time: 2014-09-18 18:08:36.682758, event: all_read, op: pg_notify(2.71c(12) epoch 11715) -1540> 2014-09-18 18:08:36.688813 7f4c9f251700 5 -- op tracker -- , seq: 86279, time: 2014-09-18 18:08:36.688786, event: dispatched, op: pg_notify(2.71c(12) epoch 11715) -1539> 2014-09-18 18:08:36.688821 7f4c9f251700 5 -- op tracker -- , seq: 86279, time: 2014-09-18 18:08:36.688820, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11715) -1538> 2014-09-18 18:08:36.688827 7f4c9f251700 5 -- op tracker -- , seq: 86279, time: 2014-09-18 18:08:36.688826, event: started, op: pg_notify(2.71c(12) epoch 11715) -1537> 2014-09-18 18:08:36.688842 7f4c9f251700 5 -- op tracker -- , seq: 86279, time: 2014-09-18 18:08:36.688841, event: done, op: pg_notify(2.71c(12) epoch 11715) -1536> 2014-09-18 18:08:36.688855 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2023 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xf954ac0 con 0xe2d9a20 -1535> 2014-09-18 18:08:36.688866 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1534> 2014-09-18 18:08:36.688872 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1277 ==== pg_notify(2.17e(17) epoch 11715) v5 ==== 1593+0+0 (3023352550 0 0) 0xf1e8580 con 0xe8db760 -1533> 2014-09-18 18:08:36.688882 7f4c9f251700 5 -- op tracker -- , seq: 86280, time: 2014-09-18 18:08:36.684399, event: header_read, op: pg_notify(2.17e(17) epoch 11715) -1532> 2014-09-18 18:08:36.688888 7f4c9f251700 5 -- op tracker -- , seq: 86280, time: 2014-09-18 18:08:36.684400, event: throttled, op: pg_notify(2.17e(17) epoch 11715) -1531> 2014-09-18 18:08:36.688891 7f4c9f251700 5 -- op tracker -- , seq: 86280, time: 2014-09-18 18:08:36.684452, event: all_read, op: pg_notify(2.17e(17) epoch 11715) -1530> 2014-09-18 18:08:36.688895 7f4c9f251700 5 -- op tracker -- , seq: 86280, time: 2014-09-18 18:08:36.688880, event: dispatched, op: pg_notify(2.17e(17) epoch 11715) -1529> 2014-09-18 18:08:36.688900 7f4c9f251700 5 -- op tracker -- , seq: 86280, time: 2014-09-18 18:08:36.688898, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11715) -1528> 2014-09-18 18:08:36.688905 7f4c9f251700 5 -- op tracker -- , seq: 86280, time: 2014-09-18 18:08:36.688903, event: started, op: pg_notify(2.17e(17) epoch 11715) -1527> 2014-09-18 18:08:36.688917 7f4c9f251700 5 -- op tracker -- , seq: 86280, time: 2014-09-18 18:08:36.688916, event: done, op: pg_notify(2.17e(17) epoch 11715) -1526> 2014-09-18 18:08:36.688930 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 798 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x2d7fd3c0 con 0xe9a27e0 -1525> 2014-09-18 18:08:36.688938 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1524> 2014-09-18 18:08:36.688942 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 307 ==== pg_query(2.756 epoch 11715) v3 ==== 144+0+0 (3982760494 0 0) 0xf4883a0 con 0x140f95a0 -1523> 2014-09-18 18:08:36.688949 7f4c9f251700 5 -- op tracker -- , seq: 86281, time: 2014-09-18 18:08:36.685468, event: header_read, op: pg_query(2.756 epoch 11715) -1522> 2014-09-18 18:08:36.688954 7f4c9f251700 5 -- op tracker -- , seq: 86281, time: 2014-09-18 18:08:36.685469, event: throttled, op: pg_query(2.756 epoch 11715) -1521> 2014-09-18 18:08:36.688958 7f4c9f251700 5 -- op tracker -- , seq: 86281, time: 2014-09-18 18:08:36.685496, event: all_read, op: pg_query(2.756 epoch 11715) -1520> 2014-09-18 18:08:36.688962 7f4c9f251700 5 -- op tracker -- , seq: 86281, time: 2014-09-18 18:08:36.688947, event: dispatched, op: pg_query(2.756 epoch 11715) -1519> 2014-09-18 18:08:36.688966 7f4c9f251700 5 -- op tracker -- , seq: 86281, time: 2014-09-18 18:08:36.688965, event: waiting_for_osdmap, op: pg_query(2.756 epoch 11715) -1518> 2014-09-18 18:08:36.688971 7f4c9f251700 5 -- op tracker -- , seq: 86281, time: 2014-09-18 18:08:36.688970, event: started, op: pg_query(2.756 epoch 11715) -1517> 2014-09-18 18:08:36.689021 7f4c9f251700 5 -- op tracker -- , seq: 86281, time: 2014-09-18 18:08:36.689019, event: done, op: pg_query(2.756 epoch 11715) -1516> 2014-09-18 18:08:36.689034 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.10 10.242.42.168:6845/1010695 502 ==== osd_map(11713..11715 src has 1..11715) v3 ==== 5046+0+0 (3853047371 0 0) 0x2fba1840 con 0x12b53020 -1515> 2014-09-18 18:08:36.689045 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11713,11715], i have 11715, src has [1,11715] -1514> 2014-09-18 18:08:36.689052 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1052 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xcc02400 con 0x14358260 -1513> 2014-09-18 18:08:36.689060 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1512> 2014-09-18 18:08:36.689066 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3698 ==== pg_query(2.654 epoch 11715) v3 ==== 144+0+0 (2983861923 0 0) 0x2b92ef00 con 0xcbecec0 -1511> 2014-09-18 18:08:36.689075 7f4c9f251700 5 -- op tracker -- , seq: 86282, time: 2014-09-18 18:08:36.688966, event: header_read, op: pg_query(2.654 epoch 11715) -1510> 2014-09-18 18:08:36.689082 7f4c9f251700 5 -- op tracker -- , seq: 86282, time: 2014-09-18 18:08:36.688967, event: throttled, op: pg_query(2.654 epoch 11715) -1509> 2014-09-18 18:08:36.689088 7f4c9f251700 5 -- op tracker -- , seq: 86282, time: 2014-09-18 18:08:36.689019, event: all_read, op: pg_query(2.654 epoch 11715) -1508> 2014-09-18 18:08:36.689094 7f4c9f251700 5 -- op tracker -- , seq: 86282, time: 2014-09-18 18:08:36.689072, event: dispatched, op: pg_query(2.654 epoch 11715) -1507> 2014-09-18 18:08:36.689100 7f4c9f251700 5 -- op tracker -- , seq: 86282, time: 2014-09-18 18:08:36.689098, event: waiting_for_osdmap, op: pg_query(2.654 epoch 11715) -1506> 2014-09-18 18:08:36.689107 7f4c9f251700 5 -- op tracker -- , seq: 86282, time: 2014-09-18 18:08:36.689105, event: started, op: pg_query(2.654 epoch 11715) -1505> 2014-09-18 18:08:36.689118 7f4c9f251700 5 -- op tracker -- , seq: 86282, time: 2014-09-18 18:08:36.689117, event: done, op: pg_query(2.654 epoch 11715) -1504> 2014-09-18 18:08:36.689128 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 45 ==== pg_notify(2.376(58) epoch 11715) v5 ==== 3960+0+0 (507216016 0 0) 0xedfd660 con 0x142f8260 -1503> 2014-09-18 18:08:36.689137 7f4c9f251700 5 -- op tracker -- , seq: 86283, time: 2014-09-18 18:08:36.686314, event: header_read, op: pg_notify(2.376(58) epoch 11715) -1502> 2014-09-18 18:08:36.689144 7f4c9f251700 5 -- op tracker -- , seq: 86283, time: 2014-09-18 18:08:36.686315, event: throttled, op: pg_notify(2.376(58) epoch 11715) -1501> 2014-09-18 18:08:36.689150 7f4c9f251700 5 -- op tracker -- , seq: 86283, time: 2014-09-18 18:08:36.686431, event: all_read, op: pg_notify(2.376(58) epoch 11715) -1500> 2014-09-18 18:08:36.689156 7f4c9f251700 5 -- op tracker -- , seq: 86283, time: 2014-09-18 18:08:36.689135, event: dispatched, op: pg_notify(2.376(58) epoch 11715) -1499> 2014-09-18 18:08:36.689161 7f4c9f251700 5 -- op tracker -- , seq: 86283, time: 2014-09-18 18:08:36.689160, event: waiting_for_osdmap, op: pg_notify(2.376(58) epoch 11715) -1498> 2014-09-18 18:08:36.689169 7f4c9f251700 5 -- op tracker -- , seq: 86283, time: 2014-09-18 18:08:36.689167, event: started, op: pg_notify(2.376(58) epoch 11715) -1497> 2014-09-18 18:08:36.689183 7f4c9f251700 5 -- op tracker -- , seq: 86283, time: 2014-09-18 18:08:36.689181, event: done, op: pg_notify(2.376(58) epoch 11715) -1496> 2014-09-18 18:08:36.689198 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 949 ==== pg_notify(2.78f(157) epoch 11715) v5 ==== 9441+0+0 (43489991 0 0) 0x2b93c760 con 0x1395f8c0 -1495> 2014-09-18 18:08:36.689199 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11711..11715 src has 1..11715) v3 -- ?+0 0xe269600 con 0x1395ef20 -1494> 2014-09-18 18:08:36.689207 7f4c9f251700 5 -- op tracker -- , seq: 86284, time: 2014-09-18 18:08:36.688267, event: header_read, op: pg_notify(2.78f(157) epoch 11715) -1493> 2014-09-18 18:08:36.689216 7f4c9f251700 5 -- op tracker -- , seq: 86284, time: 2014-09-18 18:08:36.688268, event: throttled, op: pg_notify(2.78f(157) epoch 11715) -1492> 2014-09-18 18:08:36.689220 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(0.34(23) epoch 11715) v5 -- ?+0 0xe0c9480 con 0x1395ef20 -1491> 2014-09-18 18:08:36.689226 7f4c9f251700 5 -- op tracker -- , seq: 86284, time: 2014-09-18 18:08:36.688403, event: all_read, op: pg_notify(2.78f(157) epoch 11715) -1490> 2014-09-18 18:08:36.689236 7f4c9f251700 5 -- op tracker -- , seq: 86284, time: 2014-09-18 18:08:36.689205, event: dispatched, op: pg_notify(2.78f(157) epoch 11715) -1489> 2014-09-18 18:08:36.689246 7f4c9f251700 5 -- op tracker -- , seq: 86284, time: 2014-09-18 18:08:36.689244, event: waiting_for_osdmap, op: pg_notify(2.78f(157) epoch 11715) -1488> 2014-09-18 18:08:36.689253 7f4c9f251700 5 -- op tracker -- , seq: 86284, time: 2014-09-18 18:08:36.689252, event: started, op: pg_notify(2.78f(157) epoch 11715) -1487> 2014-09-18 18:08:36.689275 7f4c9f251700 5 -- op tracker -- , seq: 86284, time: 2014-09-18 18:08:36.689273, event: done, op: pg_notify(2.78f(157) epoch 11715) -1486> 2014-09-18 18:08:36.689367 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.45 10.242.42.178:6817/7086 2 ==== pg_query(2.3c6 epoch 11715) v3 ==== 144+0+0 (2452034050 0 0) 0xf1e6d20 con 0x1395f1e0 -1485> 2014-09-18 18:08:36.689380 7f4c9f251700 5 -- op tracker -- , seq: 86285, time: 2014-09-18 18:08:36.688409, event: header_read, op: pg_query(2.3c6 epoch 11715) -1484> 2014-09-18 18:08:36.689392 7f4c9f251700 5 -- op tracker -- , seq: 86285, time: 2014-09-18 18:08:36.688410, event: throttled, op: pg_query(2.3c6 epoch 11715) -1483> 2014-09-18 18:08:36.689399 7f4c9f251700 5 -- op tracker -- , seq: 86285, time: 2014-09-18 18:08:36.688432, event: all_read, op: pg_query(2.3c6 epoch 11715) -1482> 2014-09-18 18:08:36.689406 7f4c9f251700 5 -- op tracker -- , seq: 86285, time: 2014-09-18 18:08:36.689377, event: dispatched, op: pg_query(2.3c6 epoch 11715) -1481> 2014-09-18 18:08:36.689411 7f4c9f251700 5 -- op tracker -- , seq: 86285, time: 2014-09-18 18:08:36.689409, event: waiting_for_osdmap, op: pg_query(2.3c6 epoch 11715) -1480> 2014-09-18 18:08:36.689420 7f4c9f251700 5 -- op tracker -- , seq: 86285, time: 2014-09-18 18:08:36.689417, event: started, op: pg_query(2.3c6 epoch 11715) -1479> 2014-09-18 18:08:36.689435 7f4c9f251700 5 -- op tracker -- , seq: 86285, time: 2014-09-18 18:08:36.689433, event: done, op: pg_query(2.3c6 epoch 11715) -1478> 2014-09-18 18:08:36.689447 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 550 ==== pg_query(0.34 epoch 11715) v3 ==== 144+0+0 (3620420941 0 0) 0xf7a6b40 con 0x1395ef20 -1477> 2014-09-18 18:08:36.689456 7f4c9f251700 5 -- op tracker -- , seq: 86286, time: 2014-09-18 18:08:36.687867, event: header_read, op: pg_query(0.34 epoch 11715) -1476> 2014-09-18 18:08:36.689461 7f4c9f251700 5 -- op tracker -- , seq: 86286, time: 2014-09-18 18:08:36.687868, event: throttled, op: pg_query(0.34 epoch 11715) -1475> 2014-09-18 18:08:36.689465 7f4c9f251700 5 -- op tracker -- , seq: 86286, time: 2014-09-18 18:08:36.687886, event: all_read, op: pg_query(0.34 epoch 11715) -1474> 2014-09-18 18:08:36.689469 7f4c9f251700 5 -- op tracker -- , seq: 86286, time: 2014-09-18 18:08:36.689453, event: dispatched, op: pg_query(0.34 epoch 11715) -1473> 2014-09-18 18:08:36.689473 7f4c9f251700 5 -- op tracker -- , seq: 86286, time: 2014-09-18 18:08:36.689472, event: waiting_for_osdmap, op: pg_query(0.34 epoch 11715) -1472> 2014-09-18 18:08:36.689478 7f4c9f251700 5 -- op tracker -- , seq: 86286, time: 2014-09-18 18:08:36.689477, event: started, op: pg_query(0.34 epoch 11715) -1471> 2014-09-18 18:08:36.689487 7f4c9f251700 5 -- op tracker -- , seq: 86286, time: 2014-09-18 18:08:36.689486, event: done, op: pg_query(0.34 epoch 11715) -1470> 2014-09-18 18:08:36.689494 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 7 ==== pg_notify(2.40a(162) epoch 11715) v5 ==== 9736+0+0 (206334107 0 0) 0xec6a940 con 0x12b3b2e0 -1469> 2014-09-18 18:08:36.689501 7f4c9f251700 5 -- op tracker -- , seq: 86287, time: 2014-09-18 18:08:36.688558, event: header_read, op: pg_notify(2.40a(162) epoch 11715) -1468> 2014-09-18 18:08:36.689367 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11482/11482/10657) [6,13,57]/[13,16] r=-1 lpr=11482 pi=10657-11481/13 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started/Stray 393.724465 699 0.013870 -1467> 2014-09-18 18:08:36.689506 7f4c9f251700 5 -- op tracker -- , seq: 86287, time: 2014-09-18 18:08:36.688560, event: throttled, op: pg_notify(2.40a(162) epoch 11715) -1466> 2014-09-18 18:08:36.689510 7f4c9f251700 5 -- op tracker -- , seq: 86287, time: 2014-09-18 18:08:36.688791, event: all_read, op: pg_notify(2.40a(162) epoch 11715) -1465> 2014-09-18 18:08:36.689514 7f4c9f251700 5 -- op tracker -- , seq: 86287, time: 2014-09-18 18:08:36.689499, event: dispatched, op: pg_notify(2.40a(162) epoch 11715) -1464> 2014-09-18 18:08:36.689518 7f4c9f251700 5 -- op tracker -- , seq: 86287, time: 2014-09-18 18:08:36.689517, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11715) -1463> 2014-09-18 18:08:36.689523 7f4c9f251700 5 -- op tracker -- , seq: 86287, time: 2014-09-18 18:08:36.689522, event: started, op: pg_notify(2.40a(162) epoch 11715) -1462> 2014-09-18 18:08:36.689511 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11482/11482/10657) [6,13,57]/[13,16] r=-1 lpr=11482 pi=10657-11481/13 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started 393.724681 0 0.000000 -1461> 2014-09-18 18:08:36.689536 7f4c9f251700 5 -- op tracker -- , seq: 86287, time: 2014-09-18 18:08:36.689534, event: done, op: pg_notify(2.40a(162) epoch 11715) -1460> 2014-09-18 18:08:36.689530 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11482/11482/10657) [6,13,57]/[13,16] r=-1 lpr=11482 pi=10657-11481/13 crt=0'0 lcod 0'0 remapped NOTIFY] enter Reset -1459> 2014-09-18 18:08:36.689573 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2372 ==== pg_notify(2.457(21) epoch 11715) v5 ==== 1875+0+0 (2950121765 0 0) 0xf2b8760 con 0xdd8c940 -1458> 2014-09-18 18:08:36.689581 7f4c9f251700 5 -- op tracker -- , seq: 86288, time: 2014-09-18 18:08:36.687942, event: header_read, op: pg_notify(2.457(21) epoch 11715) -1457> 2014-09-18 18:08:36.689586 7f4c9f251700 5 -- op tracker -- , seq: 86288, time: 2014-09-18 18:08:36.687942, event: throttled, op: pg_notify(2.457(21) epoch 11715) -1456> 2014-09-18 18:08:36.689589 7f4c9f251700 5 -- op tracker -- , seq: 86288, time: 2014-09-18 18:08:36.687981, event: all_read, op: pg_notify(2.457(21) epoch 11715) -1455> 2014-09-18 18:08:36.689593 7f4c9f251700 5 -- op tracker -- , seq: 86288, time: 2014-09-18 18:08:36.689579, event: dispatched, op: pg_notify(2.457(21) epoch 11715) -1454> 2014-09-18 18:08:36.689598 7f4c9f251700 5 -- op tracker -- , seq: 86288, time: 2014-09-18 18:08:36.689596, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11715) -1453> 2014-09-18 18:08:36.689603 7f4c9f251700 5 -- op tracker -- , seq: 86288, time: 2014-09-18 18:08:36.689601, event: started, op: pg_notify(2.457(21) epoch 11715) -1452> 2014-09-18 18:08:36.689615 7f4c9f251700 5 -- op tracker -- , seq: 86288, time: 2014-09-18 18:08:36.689613, event: done, op: pg_notify(2.457(21) epoch 11715) -1451> 2014-09-18 18:08:36.689625 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3320 ==== pg_notify(2.17e(17) epoch 11715) v5 ==== 1593+0+0 (3023352550 0 0) 0x12112d20 con 0x4900580 -1450> 2014-09-18 18:08:36.689632 7f4c9f251700 5 -- op tracker -- , seq: 86289, time: 2014-09-18 18:08:36.685602, event: header_read, op: pg_notify(2.17e(17) epoch 11715) -1449> 2014-09-18 18:08:36.689637 7f4c9f251700 5 -- op tracker -- , seq: 86289, time: 2014-09-18 18:08:36.685603, event: throttled, op: pg_notify(2.17e(17) epoch 11715) -1448> 2014-09-18 18:08:36.689640 7f4c9f251700 5 -- op tracker -- , seq: 86289, time: 2014-09-18 18:08:36.685653, event: all_read, op: pg_notify(2.17e(17) epoch 11715) -1447> 2014-09-18 18:08:36.689644 7f4c9f251700 5 -- op tracker -- , seq: 86289, time: 2014-09-18 18:08:36.689630, event: dispatched, op: pg_notify(2.17e(17) epoch 11715) -1446> 2014-09-18 18:08:36.689649 7f4c9f251700 5 -- op tracker -- , seq: 86289, time: 2014-09-18 18:08:36.689647, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11715) -1445> 2014-09-18 18:08:36.689654 7f4c9f251700 5 -- op tracker -- , seq: 86289, time: 2014-09-18 18:08:36.689652, event: started, op: pg_notify(2.17e(17) epoch 11715) -1444> 2014-09-18 18:08:36.689664 7f4c9f251700 5 -- op tracker -- , seq: 86289, time: 2014-09-18 18:08:36.689663, event: done, op: pg_notify(2.17e(17) epoch 11715) -1443> 2014-09-18 18:08:36.689677 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2024 ==== pg_notify(2.26d(9) epoch 11715) v5 ==== 1257+0+0 (2228899848 0 0) 0xf1ee780 con 0xe2d9a20 -1442> 2014-09-18 18:08:36.689684 7f4c9f251700 5 -- op tracker -- , seq: 86290, time: 2014-09-18 18:08:36.687673, event: header_read, op: pg_notify(2.26d(9) epoch 11715) -1441> 2014-09-18 18:08:36.689689 7f4c9f251700 5 -- op tracker -- , seq: 86290, time: 2014-09-18 18:08:36.687674, event: throttled, op: pg_notify(2.26d(9) epoch 11715) -1440> 2014-09-18 18:08:36.689693 7f4c9f251700 5 -- op tracker -- , seq: 86290, time: 2014-09-18 18:08:36.687710, event: all_read, op: pg_notify(2.26d(9) epoch 11715) -1439> 2014-09-18 18:08:36.689697 7f4c9f251700 5 -- op tracker -- , seq: 86290, time: 2014-09-18 18:08:36.689682, event: dispatched, op: pg_notify(2.26d(9) epoch 11715) -1438> 2014-09-18 18:08:36.689701 7f4c9f251700 5 -- op tracker -- , seq: 86290, time: 2014-09-18 18:08:36.689700, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11715) -1437> 2014-09-18 18:08:36.689706 7f4c9f251700 5 -- op tracker -- , seq: 86290, time: 2014-09-18 18:08:36.689705, event: started, op: pg_notify(2.26d(9) epoch 11715) -1436> 2014-09-18 18:08:36.689717 7f4c9f251700 5 -- op tracker -- , seq: 86290, time: 2014-09-18 18:08:36.689716, event: done, op: pg_notify(2.26d(9) epoch 11715) -1435> 2014-09-18 18:08:36.689726 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 799 ==== pg_notify(2.2d8(174) epoch 11715) v5 ==== 10808+0+0 (2960755102 0 0) 0x1421f860 con 0xe9a27e0 -1434> 2014-09-18 18:08:36.689733 7f4c9f251700 5 -- op tracker -- , seq: 86291, time: 2014-09-18 18:08:36.686788, event: header_read, op: pg_notify(2.2d8(174) epoch 11715) -1433> 2014-09-18 18:08:36.689738 7f4c9f251700 5 -- op tracker -- , seq: 86291, time: 2014-09-18 18:08:36.686789, event: throttled, op: pg_notify(2.2d8(174) epoch 11715) -1432> 2014-09-18 18:08:36.689742 7f4c9f251700 5 -- op tracker -- , seq: 86291, time: 2014-09-18 18:08:36.686981, event: all_read, op: pg_notify(2.2d8(174) epoch 11715) -1431> 2014-09-18 18:08:36.689746 7f4c9f251700 5 -- op tracker -- , seq: 86291, time: 2014-09-18 18:08:36.689731, event: dispatched, op: pg_notify(2.2d8(174) epoch 11715) -1430> 2014-09-18 18:08:36.689750 7f4c9f251700 5 -- op tracker -- , seq: 86291, time: 2014-09-18 18:08:36.689748, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11715) -1429> 2014-09-18 18:08:36.689754 7f4c9f251700 5 -- op tracker -- , seq: 86291, time: 2014-09-18 18:08:36.689753, event: started, op: pg_notify(2.2d8(174) epoch 11715) -1428> 2014-09-18 18:08:36.689766 7f4c9f251700 5 -- op tracker -- , seq: 86291, time: 2014-09-18 18:08:36.689764, event: done, op: pg_notify(2.2d8(174) epoch 11715) -1427> 2014-09-18 18:08:36.689821 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 308 ==== pg_notify(2.763(56) epoch 11715) v5 ==== 3866+0+0 (3924089360 0 0) 0xf2b8b20 con 0x140f95a0 -1426> 2014-09-18 18:08:36.689829 7f4c9f251700 5 -- op tracker -- , seq: 86292, time: 2014-09-18 18:08:36.687890, event: header_read, op: pg_notify(2.763(56) epoch 11715) -1425> 2014-09-18 18:08:36.689834 7f4c9f251700 5 -- op tracker -- , seq: 86292, time: 2014-09-18 18:08:36.687890, event: throttled, op: pg_notify(2.763(56) epoch 11715) -1424> 2014-09-18 18:08:36.689838 7f4c9f251700 5 -- op tracker -- , seq: 86292, time: 2014-09-18 18:08:36.687945, event: all_read, op: pg_notify(2.763(56) epoch 11715) -1423> 2014-09-18 18:08:36.689842 7f4c9f251700 5 -- op tracker -- , seq: 86292, time: 2014-09-18 18:08:36.689827, event: dispatched, op: pg_notify(2.763(56) epoch 11715) -1422> 2014-09-18 18:08:36.689846 7f4c9f251700 5 -- op tracker -- , seq: 86292, time: 2014-09-18 18:08:36.689845, event: waiting_for_osdmap, op: pg_notify(2.763(56) epoch 11715) -1421> 2014-09-18 18:08:36.689851 7f4c9f251700 5 -- op tracker -- , seq: 86292, time: 2014-09-18 18:08:36.689850, event: started, op: pg_notify(2.763(56) epoch 11715) -1420> 2014-09-18 18:08:36.689863 7f4c9f251700 5 -- op tracker -- , seq: 86292, time: 2014-09-18 18:08:36.689861, event: done, op: pg_notify(2.763(56) epoch 11715) -1419> 2014-09-18 18:08:36.689879 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.10 10.242.42.168:6845/1010695 503 ==== pg_query(2.250 epoch 11715) v3 ==== 144+0+0 (1985393488 0 0) 0xf29b4a0 con 0x12b53020 -1418> 2014-09-18 18:08:36.689886 7f4c9f251700 5 -- op tracker -- , seq: 86293, time: 2014-09-18 18:08:36.688187, event: header_read, op: pg_query(2.250 epoch 11715) -1417> 2014-09-18 18:08:36.689891 7f4c9f251700 5 -- op tracker -- , seq: 86293, time: 2014-09-18 18:08:36.688188, event: throttled, op: pg_query(2.250 epoch 11715) -1416> 2014-09-18 18:08:36.689895 7f4c9f251700 5 -- op tracker -- , seq: 86293, time: 2014-09-18 18:08:36.688225, event: all_read, op: pg_query(2.250 epoch 11715) -1415> 2014-09-18 18:08:36.689899 7f4c9f251700 5 -- op tracker -- , seq: 86293, time: 2014-09-18 18:08:36.689884, event: dispatched, op: pg_query(2.250 epoch 11715) -1414> 2014-09-18 18:08:36.689903 7f4c9f251700 5 -- op tracker -- , seq: 86293, time: 2014-09-18 18:08:36.689902, event: waiting_for_osdmap, op: pg_query(2.250 epoch 11715) -1413> 2014-09-18 18:08:36.689909 7f4c9f251700 5 -- op tracker -- , seq: 86293, time: 2014-09-18 18:08:36.689907, event: started, op: pg_query(2.250 epoch 11715) -1412> 2014-09-18 18:08:36.689867 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11715/11715/10657) [9,13,57]/[13,16] r=-1 lpr=11715 pi=10657-11714/14 crt=0'0 lcod 0'0 remapped NOTIFY] exit Reset 0.000337 1 0.000215 -1411> 2014-09-18 18:08:36.689918 7f4c9f251700 5 -- op tracker -- , seq: 86293, time: 2014-09-18 18:08:36.689916, event: done, op: pg_query(2.250 epoch 11715) -1410> 2014-09-18 18:08:36.689913 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11715/11715/10657) [9,13,57]/[13,16] r=-1 lpr=11715 pi=10657-11714/14 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started -1409> 2014-09-18 18:08:36.689924 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1053 ==== pg_notify(2.719(17) epoch 11715) v5 ==== 1601+0+0 (2610697462 0 0) 0xf72fc20 con 0x14358260 -1408> 2014-09-18 18:08:36.689956 7f4c9f251700 5 -- op tracker -- , seq: 86294, time: 2014-09-18 18:08:36.687459, event: header_read, op: pg_notify(2.719(17) epoch 11715) -1407> 2014-09-18 18:08:36.689962 7f4c9f251700 5 -- op tracker -- , seq: 86294, time: 2014-09-18 18:08:36.687463, event: throttled, op: pg_notify(2.719(17) epoch 11715) -1406> 2014-09-18 18:08:36.689966 7f4c9f251700 5 -- op tracker -- , seq: 86294, time: 2014-09-18 18:08:36.687507, event: all_read, op: pg_notify(2.719(17) epoch 11715) -1405> 2014-09-18 18:08:36.689970 7f4c9f251700 5 -- op tracker -- , seq: 86294, time: 2014-09-18 18:08:36.689951, event: dispatched, op: pg_notify(2.719(17) epoch 11715) -1404> 2014-09-18 18:08:36.689974 7f4c9f251700 5 -- op tracker -- , seq: 86294, time: 2014-09-18 18:08:36.689973, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11715) -1403> 2014-09-18 18:08:36.689979 7f4c9f251700 5 -- op tracker -- , seq: 86294, time: 2014-09-18 18:08:36.689978, event: started, op: pg_notify(2.719(17) epoch 11715) -1402> 2014-09-18 18:08:36.690000 7f4c9f251700 5 -- op tracker -- , seq: 86294, time: 2014-09-18 18:08:36.689997, event: done, op: pg_notify(2.719(17) epoch 11715) -1401> 2014-09-18 18:08:36.690014 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1208 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x2db46880 con 0x1088b440 -1400> 2014-09-18 18:08:36.690022 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1399> 2014-09-18 18:08:36.690026 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 46 ==== pg_notify(2.763(68) epoch 11715) v5 ==== 4550+0+0 (3831805237 0 0) 0xf6dffe0 con 0x142f8260 -1398> 2014-09-18 18:08:36.690033 7f4c9f251700 5 -- op tracker -- , seq: 86295, time: 2014-09-18 18:08:36.688665, event: header_read, op: pg_notify(2.763(68) epoch 11715) -1397> 2014-09-18 18:08:36.690038 7f4c9f251700 5 -- op tracker -- , seq: 86295, time: 2014-09-18 18:08:36.688666, event: throttled, op: pg_notify(2.763(68) epoch 11715) -1396> 2014-09-18 18:08:36.690042 7f4c9f251700 5 -- op tracker -- , seq: 86295, time: 2014-09-18 18:08:36.688787, event: all_read, op: pg_notify(2.763(68) epoch 11715) -1395> 2014-09-18 18:08:36.690046 7f4c9f251700 5 -- op tracker -- , seq: 86295, time: 2014-09-18 18:08:36.690031, event: dispatched, op: pg_notify(2.763(68) epoch 11715) -1394> 2014-09-18 18:08:36.690050 7f4c9f251700 5 -- op tracker -- , seq: 86295, time: 2014-09-18 18:08:36.690049, event: waiting_for_osdmap, op: pg_notify(2.763(68) epoch 11715) -1393> 2014-09-18 18:08:36.690056 7f4c9f251700 5 -- op tracker -- , seq: 86295, time: 2014-09-18 18:08:36.690054, event: started, op: pg_notify(2.763(68) epoch 11715) -1392> 2014-09-18 18:08:36.690067 7f4c9f251700 5 -- op tracker -- , seq: 86295, time: 2014-09-18 18:08:36.690065, event: done, op: pg_notify(2.763(68) epoch 11715) -1391> 2014-09-18 18:08:36.690086 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3321 ==== pg_notify(2.1bb(9) epoch 11715) v5 ==== 1189+0+0 (449349703 0 0) 0xf731840 con 0x4900580 -1390> 2014-09-18 18:08:36.689950 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11715/11715/10657) [9,13,57]/[13,16] r=-1 lpr=11715 pi=10657-11714/14 crt=0'0 lcod 0'0 remapped NOTIFY] enter Start -1389> 2014-09-18 18:08:36.690093 7f4c9f251700 5 -- op tracker -- , seq: 86296, time: 2014-09-18 18:08:36.687031, event: header_read, op: pg_notify(2.1bb(9) epoch 11715) -1388> 2014-09-18 18:08:36.690098 7f4c9f251700 5 -- op tracker -- , seq: 86296, time: 2014-09-18 18:08:36.687032, event: throttled, op: pg_notify(2.1bb(9) epoch 11715) -1387> 2014-09-18 18:08:36.690093 7f4c96a40700 1 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11715/11715/10657) [9,13,57]/[13,16] r=-1 lpr=11715 pi=10657-11714/14 crt=0'0 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -1386> 2014-09-18 18:08:36.690102 7f4c9f251700 5 -- op tracker -- , seq: 86296, time: 2014-09-18 18:08:36.687086, event: all_read, op: pg_notify(2.1bb(9) epoch 11715) -1385> 2014-09-18 18:08:36.690106 7f4c9f251700 5 -- op tracker -- , seq: 86296, time: 2014-09-18 18:08:36.690091, event: dispatched, op: pg_notify(2.1bb(9) epoch 11715) -1384> 2014-09-18 18:08:36.690111 7f4c9f251700 5 -- op tracker -- , seq: 86296, time: 2014-09-18 18:08:36.690109, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11715) -1383> 2014-09-18 18:08:36.690105 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11715/11715/10657) [9,13,57]/[13,16] r=-1 lpr=11715 pi=10657-11714/14 crt=0'0 lcod 0'0 remapped NOTIFY] exit Start 0.000155 0 0.000000 -1382> 2014-09-18 18:08:36.690116 7f4c9f251700 5 -- op tracker -- , seq: 86296, time: 2014-09-18 18:08:36.690114, event: started, op: pg_notify(2.1bb(9) epoch 11715) -1381> 2014-09-18 18:08:36.690117 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.5a2( v 6810'33332 (4131'30045,6810'33332] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11272/10658 11715/11715/10657) [9,13,57]/[13,16] r=-1 lpr=11715 pi=10657-11714/14 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/Stray -1380> 2014-09-18 18:08:36.690127 7f4c9f251700 5 -- op tracker -- , seq: 86296, time: 2014-09-18 18:08:36.690126, event: done, op: pg_notify(2.1bb(9) epoch 11715) -1379> 2014-09-18 18:08:36.690137 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1209 ==== pg_notify(2.2d8(174) epoch 11715) v5 ==== 10808+0+0 (2960755102 0 0) 0xf1f12a0 con 0x1088b440 -1378> 2014-09-18 18:08:36.690144 7f4c9f251700 5 -- op tracker -- , seq: 86297, time: 2014-09-18 18:08:36.689520, event: header_read, op: pg_notify(2.2d8(174) epoch 11715) -1377> 2014-09-18 18:08:36.690148 7f4c9f251700 5 -- op tracker -- , seq: 86297, time: 2014-09-18 18:08:36.689520, event: throttled, op: pg_notify(2.2d8(174) epoch 11715) -1376> 2014-09-18 18:08:36.690152 7f4c9f251700 5 -- op tracker -- , seq: 86297, time: 2014-09-18 18:08:36.689833, event: all_read, op: pg_notify(2.2d8(174) epoch 11715) -1375> 2014-09-18 18:08:36.690156 7f4c9f251700 5 -- op tracker -- , seq: 86297, time: 2014-09-18 18:08:36.690142, event: dispatched, op: pg_notify(2.2d8(174) epoch 11715) -1374> 2014-09-18 18:08:36.690160 7f4c9f251700 5 -- op tracker -- , seq: 86297, time: 2014-09-18 18:08:36.690159, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11715) -1373> 2014-09-18 18:08:36.690165 7f4c9f251700 5 -- op tracker -- , seq: 86297, time: 2014-09-18 18:08:36.690164, event: started, op: pg_notify(2.2d8(174) epoch 11715) -1372> 2014-09-18 18:08:36.690175 7f4c9f251700 5 -- op tracker -- , seq: 86297, time: 2014-09-18 18:08:36.690174, event: done, op: pg_notify(2.2d8(174) epoch 11715) -1371> 2014-09-18 18:08:36.690234 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2373 ==== pg_notify(2.2c7(139) epoch 11715) v5 ==== 8651+0+0 (1820753373 0 0) 0x12ba3a20 con 0xdd8c940 -1370> 2014-09-18 18:08:36.690243 7f4c9f251700 5 -- op tracker -- , seq: 86298, time: 2014-09-18 18:08:36.689926, event: header_read, op: pg_notify(2.2c7(139) epoch 11715) -1369> 2014-09-18 18:08:36.690248 7f4c9f251700 5 -- op tracker -- , seq: 86298, time: 2014-09-18 18:08:36.689927, event: throttled, op: pg_notify(2.2c7(139) epoch 11715) -1368> 2014-09-18 18:08:36.690252 7f4c9f251700 5 -- op tracker -- , seq: 86298, time: 2014-09-18 18:08:36.690117, event: all_read, op: pg_notify(2.2c7(139) epoch 11715) -1367> 2014-09-18 18:08:36.690256 7f4c9f251700 5 -- op tracker -- , seq: 86298, time: 2014-09-18 18:08:36.690240, event: dispatched, op: pg_notify(2.2c7(139) epoch 11715) -1366> 2014-09-18 18:08:36.690260 7f4c9f251700 5 -- op tracker -- , seq: 86298, time: 2014-09-18 18:08:36.690258, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11715) -1365> 2014-09-18 18:08:36.690267 7f4c9f251700 5 -- op tracker -- , seq: 86298, time: 2014-09-18 18:08:36.690265, event: started, op: pg_notify(2.2c7(139) epoch 11715) -1364> 2014-09-18 18:08:36.690302 7f4c9f251700 5 -- op tracker -- , seq: 86298, time: 2014-09-18 18:08:36.690299, event: done, op: pg_notify(2.2c7(139) epoch 11715) -1363> 2014-09-18 18:08:36.690380 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2374 ==== pg_notify(2.763(1) epoch 11715) v5 ==== 805+0+0 (2832021918 0 0) 0x12ba2940 con 0xdd8c940 -1362> 2014-09-18 18:08:36.690390 7f4c9f251700 5 -- op tracker -- , seq: 86299, time: 2014-09-18 18:08:36.690131, event: header_read, op: pg_notify(2.763(1) epoch 11715) -1361> 2014-09-18 18:08:36.690396 7f4c9f251700 5 -- op tracker -- , seq: 86299, time: 2014-09-18 18:08:36.690132, event: throttled, op: pg_notify(2.763(1) epoch 11715) -1360> 2014-09-18 18:08:36.690400 7f4c9f251700 5 -- op tracker -- , seq: 86299, time: 2014-09-18 18:08:36.690157, event: all_read, op: pg_notify(2.763(1) epoch 11715) -1359> 2014-09-18 18:08:36.690404 7f4c9f251700 5 -- op tracker -- , seq: 86299, time: 2014-09-18 18:08:36.690387, event: dispatched, op: pg_notify(2.763(1) epoch 11715) -1358> 2014-09-18 18:08:36.690408 7f4c9f251700 5 -- op tracker -- , seq: 86299, time: 2014-09-18 18:08:36.690407, event: waiting_for_osdmap, op: pg_notify(2.763(1) epoch 11715) -1357> 2014-09-18 18:08:36.690413 7f4c9f251700 5 -- op tracker -- , seq: 86299, time: 2014-09-18 18:08:36.690412, event: started, op: pg_notify(2.763(1) epoch 11715) -1356> 2014-09-18 18:08:36.690442 7f4c9f251700 5 -- op tracker -- , seq: 86299, time: 2014-09-18 18:08:36.690440, event: done, op: pg_notify(2.763(1) epoch 11715) -1355> 2014-09-18 18:08:36.690469 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3699 ==== pg_notify(2.3b6(10) epoch 11715) v5 ==== 1260+0+0 (4042584236 0 0) 0xde61c00 con 0xcbecec0 -1354> 2014-09-18 18:08:36.690477 7f4c9f251700 5 -- op tracker -- , seq: 86300, time: 2014-09-18 18:08:36.690291, event: header_read, op: pg_notify(2.3b6(10) epoch 11715) -1353> 2014-09-18 18:08:36.690482 7f4c9f251700 5 -- op tracker -- , seq: 86300, time: 2014-09-18 18:08:36.690292, event: throttled, op: pg_notify(2.3b6(10) epoch 11715) -1352> 2014-09-18 18:08:36.690486 7f4c9f251700 5 -- op tracker -- , seq: 86300, time: 2014-09-18 18:08:36.690350, event: all_read, op: pg_notify(2.3b6(10) epoch 11715) -1351> 2014-09-18 18:08:36.690490 7f4c9f251700 5 -- op tracker -- , seq: 86300, time: 2014-09-18 18:08:36.690475, event: dispatched, op: pg_notify(2.3b6(10) epoch 11715) -1350> 2014-09-18 18:08:36.690494 7f4c9f251700 5 -- op tracker -- , seq: 86300, time: 2014-09-18 18:08:36.690493, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11715) -1349> 2014-09-18 18:08:36.690499 7f4c9f251700 5 -- op tracker -- , seq: 86300, time: 2014-09-18 18:08:36.690498, event: started, op: pg_notify(2.3b6(10) epoch 11715) -1348> 2014-09-18 18:08:36.690523 7f4c9f251700 5 -- op tracker -- , seq: 86300, time: 2014-09-18 18:08:36.690521, event: done, op: pg_notify(2.3b6(10) epoch 11715) -1347> 2014-09-18 18:08:36.690582 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 luod=0'0 crt=6789'26114 active+remapped] exit Started/ReplicaActive/RepNotRecovering 5.996748 18 0.000276 -1346> 2014-09-18 18:08:36.690602 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 luod=0'0 crt=6789'26114 active+remapped] exit Started/ReplicaActive 5.996780 0 0.000000 -1345> 2014-09-18 18:08:36.688087 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11715/11715/11015) [20,42,27]/[6,42] r=1 lpr=11715 pi=9903-11714/14 crt=6775'27542 lcod 0'0 remapped NOTIFY] exit Reset 0.000403 1 0.000384 -1344> 2014-09-18 18:08:36.690648 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11715/11715/11015) [20,42,27]/[6,42] r=1 lpr=11715 pi=9903-11714/14 crt=6775'27542 lcod 0'0 remapped NOTIFY] enter Started -1343> 2014-09-18 18:08:36.690664 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11715/11715/11015) [20,42,27]/[6,42] r=1 lpr=11715 pi=9903-11714/14 crt=6775'27542 lcod 0'0 remapped NOTIFY] enter Start -1342> 2014-09-18 18:08:36.690677 7f4c97241700 1 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11715/11715/11015) [20,42,27]/[6,42] r=1 lpr=11715 pi=9903-11714/14 crt=6775'27542 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -1341> 2014-09-18 18:08:36.690693 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11715/11715/11015) [20,42,27]/[6,42] r=1 lpr=11715 pi=9903-11714/14 crt=6775'27542 lcod 0'0 remapped NOTIFY] exit Start 0.000029 0 0.000000 -1340> 2014-09-18 18:08:36.690708 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.43f( v 6775'27542 (4108'23989,6775'27542] local-les=11518 n=526 ec=188 les/c 11518/11259 11715/11715/11015) [20,42,27]/[6,42] r=1 lpr=11715 pi=9903-11714/14 crt=6775'27542 lcod 0'0 remapped NOTIFY] enter Started/Stray -1339> 2014-09-18 18:08:36.690617 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 luod=0'0 crt=6789'26114 active+remapped] exit Started 6.971504 0 0.000000 -1338> 2014-09-18 18:08:36.690962 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11708/11708/10657) [9,30,51]/[13,42] r=1 lpr=11708 pi=9952-11707/27 luod=0'0 crt=6789'26114 active+remapped] enter Reset -1337> 2014-09-18 18:08:36.691115 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11504/11504/10657) [6,13,27]/[13,44] r=-1 lpr=11504 pi=10657-11503/10 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started/Stray 371.052401 633 0.011039 -1336> 2014-09-18 18:08:36.691145 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11504/11504/10657) [6,13,27]/[13,44] r=-1 lpr=11504 pi=10657-11503/10 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started 371.052466 0 0.000000 -1335> 2014-09-18 18:08:36.691161 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11504/11504/10657) [6,13,27]/[13,44] r=-1 lpr=11504 pi=10657-11503/10 crt=0'0 lcod 0'0 remapped NOTIFY] enter Reset -1334> 2014-09-18 18:08:36.691254 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] exit Reset 0.000093 1 0.000084 -1333> 2014-09-18 18:08:36.691270 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started -1332> 2014-09-18 18:08:36.691284 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Start -1331> 2014-09-18 18:08:36.691296 7f4c97241700 1 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -1330> 2014-09-18 18:08:36.691311 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] exit Start 0.000027 0 0.000000 -1329> 2014-09-18 18:08:36.691325 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11349/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/Stray -1328> 2014-09-18 18:08:36.691308 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] exit Reset 0.000345 1 0.000426 -1327> 2014-09-18 18:08:36.691363 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] enter Started -1326> 2014-09-18 18:08:36.691402 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] enter Start -1325> 2014-09-18 18:08:36.691443 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0x12bf1b00 con 0x1088aec0 -1324> 2014-09-18 18:08:36.691409 7f4c96a40700 1 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] state: transitioning to Stray -1323> 2014-09-18 18:08:36.691461 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] exit Start 0.000059 0 0.000000 -1322> 2014-09-18 18:08:36.691470 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] enter Started/Stray -1321> 2014-09-18 18:08:36.691504 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.43f(14) epoch 11715) v5 -- ?+0 0x21e10780 con 0x1088aec0 -1320> 2014-09-18 18:08:36.691589 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(11) epoch 11715) v5 -- ?+0 0x121134a0 con 0x140f95a0 -1319> 2014-09-18 18:08:36.691617 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0x11bb93c0 con 0x1395edc0 -1318> 2014-09-18 18:08:36.691637 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11715) v5 -- ?+0 0x12c114a0 con 0x1395edc0 -1317> 2014-09-18 18:08:36.691686 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0x11bb81c0 con 0x1395f1e0 -1316> 2014-09-18 18:08:36.691704 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(24) epoch 11715) v5 -- ?+0 0x12c130c0 con 0x1395f1e0 -1315> 2014-09-18 18:08:36.691793 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11490/11490/9945) [42,27,9]/[42,9] r=0 lpr=11490 pi=10726-11489/9 bft=27 crt=6779'26713 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 145.591870 328 0.001743 -1314> 2014-09-18 18:08:36.692007 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11490/11490/9945) [42,27,9]/[42,9] r=0 lpr=11490 pi=10726-11489/9 bft=27 crt=6779'26713 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active 385.053296 0 0.000000 -1313> 2014-09-18 18:08:36.692167 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11490/11490/9945) [42,27,9]/[42,9] r=0 lpr=11490 pi=10726-11489/9 bft=27 crt=6779'26713 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 386.060018 0 0.000000 -1312> 2014-09-18 18:08:36.692202 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11490/11490/9945) [42,27,9]/[42,9] r=0 lpr=11490 pi=10726-11489/9 bft=27 crt=6779'26713 lcod 0'0 mlcod 0'0 active+remapped] exit Started 386.060110 0 0.000000 -1311> 2014-09-18 18:08:36.692215 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11490/11490/9945) [42,27,9]/[42,9] r=0 lpr=11490 pi=10726-11489/9 bft=27 crt=6779'26713 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -1310> 2014-09-18 18:08:36.692130 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/NotBackfilling 5.201419 26 0.000108 -1309> 2014-09-18 18:08:36.692265 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000050 1 0.000470 -1308> 2014-09-18 18:08:36.692268 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active 370.028391 0 0.000000 -1307> 2014-09-18 18:08:36.692280 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped] enter Started -1306> 2014-09-18 18:08:36.692291 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped] enter Start -1305> 2014-09-18 18:08:36.692299 7f4c96a40700 1 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -1304> 2014-09-18 18:08:36.692283 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+remapped] exit Started/Primary 371.049210 0 0.000000 -1303> 2014-09-18 18:08:36.692309 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000019 0 0.000000 -1302> 2014-09-18 18:08:36.692313 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+remapped] exit Started 371.049306 0 0.000000 -1301> 2014-09-18 18:08:36.692321 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -1300> 2014-09-18 18:08:36.692325 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11504/11504/11491) [9,27,42]/[42] r=0 lpr=11504 pi=3414-11503/173 bft=9,27 crt=6745'27699 lcod 0'0 mlcod 0'0 active+remapped] enter Reset -1299> 2014-09-18 18:08:36.692331 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -1298> 2014-09-18 18:08:36.692375 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000050 1 0.000243 -1297> 2014-09-18 18:08:36.692471 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped] enter Started -1296> 2014-09-18 18:08:36.692504 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1054 ==== pg_query(2.b5 epoch 11715) v3 ==== 144+0+0 (2247502035 0 0) 0xf72f4a0 con 0x14358260 -1295> 2014-09-18 18:08:36.692482 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped] enter Start -1294> 2014-09-18 18:08:36.692522 7f4c9f251700 5 -- op tracker -- , seq: 86301, time: 2014-09-18 18:08:36.692463, event: header_read, op: pg_query(2.b5 epoch 11715) -1293> 2014-09-18 18:08:36.692535 7f4c9f251700 5 -- op tracker -- , seq: 86301, time: 2014-09-18 18:08:36.692464, event: throttled, op: pg_query(2.b5 epoch 11715) -1292> 2014-09-18 18:08:36.692532 7f4c97241700 1 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -1291> 2014-09-18 18:08:36.692471 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -1290> 2014-09-18 18:08:36.692549 7f4c9f251700 5 -- op tracker -- , seq: 86301, time: 2014-09-18 18:08:36.692485, event: all_read, op: pg_query(2.b5 epoch 11715) -1289> 2014-09-18 18:08:36.692553 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000071 0 0.000000 -1288> 2014-09-18 18:08:36.692563 7f4c9f251700 5 -- op tracker -- , seq: 86301, time: 2014-09-18 18:08:36.692516, event: dispatched, op: pg_query(2.b5 epoch 11715) -1287> 2014-09-18 18:08:36.692577 7f4c9f251700 5 -- op tracker -- , seq: 86301, time: 2014-09-18 18:08:36.692575, event: waiting_for_osdmap, op: pg_query(2.b5 epoch 11715) -1286> 2014-09-18 18:08:36.692572 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -1285> 2014-09-18 18:08:36.692587 7f4c9f251700 5 -- op tracker -- , seq: 86301, time: 2014-09-18 18:08:36.692585, event: started, op: pg_query(2.b5 epoch 11715) -1284> 2014-09-18 18:08:36.692586 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -1283> 2014-09-18 18:08:36.692601 7f4c9f251700 5 -- op tracker -- , seq: 86301, time: 2014-09-18 18:08:36.692600, event: done, op: pg_query(2.b5 epoch 11715) -1282> 2014-09-18 18:08:36.692613 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2375 ==== pg_notify(2.763(1) epoch 11715) v5 ==== 805+0+0 (2832021918 0 0) 0xf1eba40 con 0xdd8c940 -1281> 2014-09-18 18:08:36.692623 7f4c9f251700 5 -- op tracker -- , seq: 86302, time: 2014-09-18 18:08:36.692558, event: header_read, op: pg_notify(2.763(1) epoch 11715) -1280> 2014-09-18 18:08:36.692630 7f4c9f251700 5 -- op tracker -- , seq: 86302, time: 2014-09-18 18:08:36.692559, event: throttled, op: pg_notify(2.763(1) epoch 11715) -1279> 2014-09-18 18:08:36.692637 7f4c9f251700 5 -- op tracker -- , seq: 86302, time: 2014-09-18 18:08:36.692589, event: all_read, op: pg_notify(2.763(1) epoch 11715) -1278> 2014-09-18 18:08:36.692643 7f4c9f251700 5 -- op tracker -- , seq: 86302, time: 2014-09-18 18:08:36.692621, event: dispatched, op: pg_notify(2.763(1) epoch 11715) -1277> 2014-09-18 18:08:36.692650 7f4c9f251700 5 -- op tracker -- , seq: 86302, time: 2014-09-18 18:08:36.692648, event: waiting_for_osdmap, op: pg_notify(2.763(1) epoch 11715) -1276> 2014-09-18 18:08:36.692658 7f4c9f251700 5 -- op tracker -- , seq: 86302, time: 2014-09-18 18:08:36.692656, event: started, op: pg_notify(2.763(1) epoch 11715) -1275> 2014-09-18 18:08:36.692673 7f4c9f251700 5 -- op tracker -- , seq: 86302, time: 2014-09-18 18:08:36.692672, event: done, op: pg_notify(2.763(1) epoch 11715) -1274> 2014-09-18 18:08:36.692643 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -1273> 2014-09-18 18:08:36.692785 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3700 ==== pg_notify(2.517(11) epoch 11715) v5 ==== 1343+0+0 (554478604 0 0) 0xfccad00 con 0xcbecec0 -1272> 2014-09-18 18:08:36.692808 7f4c9f251700 5 -- op tracker -- , seq: 86303, time: 2014-09-18 18:08:36.692712, event: header_read, op: pg_notify(2.517(11) epoch 11715) -1271> 2014-09-18 18:08:36.692823 7f4c9f251700 5 -- op tracker -- , seq: 86303, time: 2014-09-18 18:08:36.692713, event: throttled, op: pg_notify(2.517(11) epoch 11715) -1270> 2014-09-18 18:08:36.692834 7f4c9f251700 5 -- op tracker -- , seq: 86303, time: 2014-09-18 18:08:36.692772, event: all_read, op: pg_notify(2.517(11) epoch 11715) -1269> 2014-09-18 18:08:36.692843 7f4c9f251700 5 -- op tracker -- , seq: 86303, time: 2014-09-18 18:08:36.692802, event: dispatched, op: pg_notify(2.517(11) epoch 11715) -1268> 2014-09-18 18:08:36.692853 7f4c9f251700 5 -- op tracker -- , seq: 86303, time: 2014-09-18 18:08:36.692850, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11715) -1267> 2014-09-18 18:08:36.692872 7f4c9f251700 5 -- op tracker -- , seq: 86303, time: 2014-09-18 18:08:36.692869, event: started, op: pg_notify(2.517(11) epoch 11715) -1266> 2014-09-18 18:08:36.692923 7f4c9f251700 5 -- op tracker -- , seq: 86303, time: 2014-09-18 18:08:36.692920, event: done, op: pg_notify(2.517(11) epoch 11715) -1265> 2014-09-18 18:08:36.693048 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -1264> 2014-09-18 18:08:36.693059 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_alive(want up_thru 11715 have 11715) v1 -- ?+0 0x21e11860 con 0x111771e0 -1263> 2014-09-18 18:08:36.693132 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(14),2.756(28) epoch 11715) v5 -- ?+0 0x21e110e0 con 0x140f95a0 -1262> 2014-09-18 18:08:36.693169 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 8 ==== pg_notify(2.1a2(20) epoch 11715) v5 ==== 1798+0+0 (3888766600 0 0) 0xf726b40 con 0x12b3b2e0 -1261> 2014-09-18 18:08:36.693190 7f4c9f251700 5 -- op tracker -- , seq: 86304, time: 2014-09-18 18:08:36.692944, event: header_read, op: pg_notify(2.1a2(20) epoch 11715) -1260> 2014-09-18 18:08:36.693204 7f4c9f251700 5 -- op tracker -- , seq: 86304, time: 2014-09-18 18:08:36.692945, event: throttled, op: pg_notify(2.1a2(20) epoch 11715) -1259> 2014-09-18 18:08:36.693214 7f4c9f251700 5 -- op tracker -- , seq: 86304, time: 2014-09-18 18:08:36.693001, event: all_read, op: pg_notify(2.1a2(20) epoch 11715) -1258> 2014-09-18 18:08:36.693224 7f4c9f251700 5 -- op tracker -- , seq: 86304, time: 2014-09-18 18:08:36.693185, event: dispatched, op: pg_notify(2.1a2(20) epoch 11715) -1257> 2014-09-18 18:08:36.693234 7f4c9f251700 5 -- op tracker -- , seq: 86304, time: 2014-09-18 18:08:36.693231, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11715) -1256> 2014-09-18 18:08:36.693247 7f4c9f251700 5 -- op tracker -- , seq: 86304, time: 2014-09-18 18:08:36.693244, event: started, op: pg_notify(2.1a2(20) epoch 11715) -1255> 2014-09-18 18:08:36.693271 7f4c9f251700 5 -- op tracker -- , seq: 86304, time: 2014-09-18 18:08:36.693268, event: done, op: pg_notify(2.1a2(20) epoch 11715) -1254> 2014-09-18 18:08:36.693288 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 551 ==== pg_notify(2.4d0(10) epoch 11715) v5 ==== 1292+0+0 (1130396050 0 0) 0xe0c9480 con 0x1395ef20 -1253> 2014-09-18 18:08:36.693306 7f4c9f251700 5 -- op tracker -- , seq: 86305, time: 2014-09-18 18:08:36.693010, event: header_read, op: pg_notify(2.4d0(10) epoch 11715) -1252> 2014-09-18 18:08:36.693318 7f4c9f251700 5 -- op tracker -- , seq: 86305, time: 2014-09-18 18:08:36.693011, event: throttled, op: pg_notify(2.4d0(10) epoch 11715) -1251> 2014-09-18 18:08:36.693325 7f4c9f251700 5 -- op tracker -- , seq: 86305, time: 2014-09-18 18:08:36.693051, event: all_read, op: pg_notify(2.4d0(10) epoch 11715) -1250> 2014-09-18 18:08:36.693332 7f4c9f251700 5 -- op tracker -- , seq: 86305, time: 2014-09-18 18:08:36.693301, event: dispatched, op: pg_notify(2.4d0(10) epoch 11715) -1249> 2014-09-18 18:08:36.693338 7f4c9f251700 5 -- op tracker -- , seq: 86305, time: 2014-09-18 18:08:36.693337, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11715) -1248> 2014-09-18 18:08:36.693347 7f4c9f251700 5 -- op tracker -- , seq: 86305, time: 2014-09-18 18:08:36.693345, event: started, op: pg_notify(2.4d0(10) epoch 11715) -1247> 2014-09-18 18:08:36.693344 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- osd_map(11712..11715 src has 1..11715) v3 -- ?+0 0x10d19180 con 0x10a878c0 -1246> 2014-09-18 18:08:36.693362 7f4c9f251700 5 -- op tracker -- , seq: 86305, time: 2014-09-18 18:08:36.693360, event: done, op: pg_notify(2.4d0(10) epoch 11715) -1245> 2014-09-18 18:08:36.693364 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- pg_query(2.204 epoch 11715) v3 -- ?+0 0x24fd2580 con 0x10a878c0 -1244> 2014-09-18 18:08:36.693406 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- osd_map(11713..11715 src has 1..11715) v3 -- ?+0 0x10d18640 con 0x1395ec60 -1243> 2014-09-18 18:08:36.693427 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_query(2.204 epoch 11715) v3 -- ?+0 0x21cfda20 con 0x1395ec60 -1242> 2014-09-18 18:08:36.693467 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0x10d16d80 con 0x1395f8c0 -1241> 2014-09-18 18:08:36.693484 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.204 epoch 11715) v3 -- ?+0 0x21cfc940 con 0x1395f8c0 -1240> 2014-09-18 18:08:36.693553 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11483/11489/11015) [6,30,42]/[6,42] r=1 lpr=11489 pi=2483-11488/130 luod=0'0 crt=6745'24067 lcod 0'0 active+remapped] exit Started/ReplicaActive/RepNotRecovering 381.608037 663 0.001473 -1239> 2014-09-18 18:08:36.693572 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11483/11489/11015) [6,30,42]/[6,42] r=1 lpr=11489 pi=2483-11488/130 luod=0'0 crt=6745'24067 lcod 0'0 active+remapped] exit Started/ReplicaActive 381.608083 0 0.000000 -1238> 2014-09-18 18:08:36.693587 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11483/11489/11015) [6,30,42]/[6,42] r=1 lpr=11489 pi=2483-11488/130 luod=0'0 crt=6745'24067 lcod 0'0 active+remapped] exit Started 386.969167 0 0.000000 -1237> 2014-09-18 18:08:36.693599 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11483/11489/11015) [6,30,42]/[6,42] r=1 lpr=11489 pi=2483-11488/130 luod=0'0 crt=6745'24067 lcod 0'0 active+remapped] enter Reset -1236> 2014-09-18 18:08:36.693608 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 luod=0'0 crt=6775'24651 lcod 0'0 active+remapped] exit Started/ReplicaActive/RepNotRecovering 2.584778 8 0.000487 -1235> 2014-09-18 18:08:36.693644 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 luod=0'0 crt=6775'24651 lcod 0'0 active+remapped] exit Started/ReplicaActive 2.584827 0 0.000000 -1234> 2014-09-18 18:08:36.693654 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 luod=0'0 crt=6775'24651 lcod 0'0 active+remapped] exit Started 6.970660 0 0.000000 -1233> 2014-09-18 18:08:36.693667 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11708/11708/11332) [60,27,6]/[35,42] r=1 lpr=11708 pi=10768-11707/11 luod=0'0 crt=6775'24651 lcod 0'0 active+remapped] enter Reset -1232> 2014-09-18 18:08:36.693792 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11715/11715/11015) [20,30,42]/[6,42] r=1 lpr=11715 pi=2483-11714/131 crt=6745'24067 lcod 0'0 remapped NOTIFY] exit Reset 0.000193 1 0.000086 -1231> 2014-09-18 18:08:36.693808 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11715/11715/11015) [20,30,42]/[6,42] r=1 lpr=11715 pi=2483-11714/131 crt=6745'24067 lcod 0'0 remapped NOTIFY] enter Started -1230> 2014-09-18 18:08:36.693821 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11715/11715/11015) [20,30,42]/[6,42] r=1 lpr=11715 pi=2483-11714/131 crt=6745'24067 lcod 0'0 remapped NOTIFY] enter Start -1229> 2014-09-18 18:08:36.693794 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] exit Reset 0.000127 1 0.000137 -1228> 2014-09-18 18:08:36.693832 7f4c97241700 1 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11715/11715/11015) [20,30,42]/[6,42] r=1 lpr=11715 pi=2483-11714/131 crt=6745'24067 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -1227> 2014-09-18 18:08:36.693844 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Started -1226> 2014-09-18 18:08:36.693845 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11715/11715/11015) [20,30,42]/[6,42] r=1 lpr=11715 pi=2483-11714/131 crt=6745'24067 lcod 0'0 remapped NOTIFY] exit Start 0.000025 0 0.000000 -1225> 2014-09-18 18:08:36.693853 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Start -1224> 2014-09-18 18:08:36.693861 7f4c96a40700 1 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -1223> 2014-09-18 18:08:36.693876 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] exit Start 0.000023 0 0.000000 -1222> 2014-09-18 18:08:36.693858 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.237( v 6819'24076 (4155'21085,6819'24076] local-les=11494 n=554 ec=188 les/c 11494/10726 11715/11715/11015) [20,30,42]/[6,42] r=1 lpr=11715 pi=2483-11714/131 crt=6745'24067 lcod 0'0 remapped NOTIFY] enter Started/Stray -1221> 2014-09-18 18:08:36.693887 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Started/Stray -1220> 2014-09-18 18:08:36.694006 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.827881 5 0.000651 -1219> 2014-09-18 18:08:36.694029 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.009396 0 0.000000 -1218> 2014-09-18 18:08:36.694048 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11709 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -1217> 2014-09-18 18:08:36.694071 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3701 ==== pg_notify(2.2c7(139) epoch 11715) v5 ==== 8651+0+0 (2905702966 0 0) 0xfcc85a0 con 0xcbecec0 -1216> 2014-09-18 18:08:36.694094 7f4c9f251700 5 -- op tracker -- , seq: 86306, time: 2014-09-18 18:08:36.693918, event: header_read, op: pg_notify(2.2c7(139) epoch 11715) -1215> 2014-09-18 18:08:36.694109 7f4c9f251700 5 -- op tracker -- , seq: 86306, time: 2014-09-18 18:08:36.693919, event: throttled, op: pg_notify(2.2c7(139) epoch 11715) -1214> 2014-09-18 18:08:36.694116 7f4c9f251700 5 -- op tracker -- , seq: 86306, time: 2014-09-18 18:08:36.694057, event: all_read, op: pg_notify(2.2c7(139) epoch 11715) -1213> 2014-09-18 18:08:36.694125 7f4c9f251700 5 -- op tracker -- , seq: 86306, time: 2014-09-18 18:08:36.694088, event: dispatched, op: pg_notify(2.2c7(139) epoch 11715) -1212> 2014-09-18 18:08:36.694136 7f4c9f251700 5 -- op tracker -- , seq: 86306, time: 2014-09-18 18:08:36.694133, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11715) -1211> 2014-09-18 18:08:36.694148 7f4c9f251700 5 -- op tracker -- , seq: 86306, time: 2014-09-18 18:08:36.694146, event: started, op: pg_notify(2.2c7(139) epoch 11715) -1210> 2014-09-18 18:08:36.694165 7f4c9f251700 5 -- op tracker -- , seq: 86306, time: 2014-09-18 18:08:36.694163, event: done, op: pg_notify(2.2c7(139) epoch 11715) -1209> 2014-09-18 18:08:36.694143 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11490/11490/9945) [42,9,58] r=0 lpr=11490 crt=6764'25987 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 385.004214 670 0.001844 -1208> 2014-09-18 18:08:36.694181 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11490/11490/9945) [42,9,58] r=0 lpr=11490 crt=6764'25987 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 385.053188 0 0.000000 -1207> 2014-09-18 18:08:36.694183 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 47 ==== pg_notify(2.763(68) epoch 11715) v5 ==== 4550+0+0 (3831805237 0 0) 0x21cea1e0 con 0x142f8260 -1206> 2014-09-18 18:08:36.694193 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11490/11490/9945) [42,9,58] r=0 lpr=11490 crt=6764'25987 lcod 0'0 mlcod 0'0 active] exit Started/Primary 386.057639 0 0.000000 -1205> 2014-09-18 18:08:36.694197 7f4c9f251700 5 -- op tracker -- , seq: 86307, time: 2014-09-18 18:08:36.694045, event: header_read, op: pg_notify(2.763(68) epoch 11715) -1204> 2014-09-18 18:08:36.694202 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11490/11490/9945) [42,9,58] r=0 lpr=11490 crt=6764'25987 lcod 0'0 mlcod 0'0 active] exit Started 386.057699 0 0.000000 -1203> 2014-09-18 18:08:36.694206 7f4c9f251700 5 -- op tracker -- , seq: 86307, time: 2014-09-18 18:08:36.694046, event: throttled, op: pg_notify(2.763(68) epoch 11715) -1202> 2014-09-18 18:08:36.694212 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11490/11490/9945) [42,9,58] r=0 lpr=11490 crt=6764'25987 lcod 0'0 mlcod 0'0 active] enter Reset -1201> 2014-09-18 18:08:36.694216 7f4c9f251700 5 -- op tracker -- , seq: 86307, time: 2014-09-18 18:08:36.694156, event: all_read, op: pg_notify(2.763(68) epoch 11715) -1200> 2014-09-18 18:08:36.694222 7f4c9f251700 5 -- op tracker -- , seq: 86307, time: 2014-09-18 18:08:36.694192, event: dispatched, op: pg_notify(2.763(68) epoch 11715) -1199> 2014-09-18 18:08:36.694229 7f4c9f251700 5 -- op tracker -- , seq: 86307, time: 2014-09-18 18:08:36.694228, event: waiting_for_osdmap, op: pg_notify(2.763(68) epoch 11715) -1198> 2014-09-18 18:08:36.694237 7f4c9f251700 5 -- op tracker -- , seq: 86307, time: 2014-09-18 18:08:36.694235, event: started, op: pg_notify(2.763(68) epoch 11715) -1197> 2014-09-18 18:08:36.694251 7f4c9f251700 5 -- op tracker -- , seq: 86307, time: 2014-09-18 18:08:36.694249, event: done, op: pg_notify(2.763(68) epoch 11715) -1196> 2014-09-18 18:08:36.694322 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000110 1 0.000176 -1195> 2014-09-18 18:08:36.694343 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] enter Started -1194> 2014-09-18 18:08:36.694356 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] enter Start -1193> 2014-09-18 18:08:36.694368 7f4c96a40700 1 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1192> 2014-09-18 18:08:36.694382 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000026 0 0.000000 -1191> 2014-09-18 18:08:36.694463 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1190> 2014-09-18 18:08:36.694479 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1189> 2014-09-18 18:08:36.694487 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11715 n=505 ec=188 les/c 11703/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -1188> 2014-09-18 18:08:36.694491 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1187> 2014-09-18 18:08:36.694687 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.237(131) epoch 11715) v5 -- ?+0 0xf5783a0 con 0x1088aec0 -1186> 2014-09-18 18:08:36.694704 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6817/19139 -- pg_query(2.2d8 epoch 11715) v3 -- ?+0 0xf5772c0 con 0x10f60520 -1185> 2014-09-18 18:08:36.694723 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_query(2.2d8 epoch 11715) v3 -- ?+0 0x2b93c760 con 0x1395f8c0 -1184> 2014-09-18 18:08:36.694771 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6857/39817 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0xf85cd00 con 0x12b3b2e0 -1183> 2014-09-18 18:08:36.694847 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6857/39817 -- pg_info(1 pgs e11715:2.78f(157)) v4 -- ?+0 0xe147c20 con 0x12b3b2e0 -1182> 2014-09-18 18:08:36.694962 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11715:2.78f(157)) v4 -- ?+0 0xe146960 con 0x1395f8c0 -1181> 2014-09-18 18:08:36.694942 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11490/11490/11490) [32,42,9]/[9,42] r=1 lpr=11490 pi=10726-11489/8 luod=0'0 crt=6835'27134 lcod 0'0 active+remapped] exit Started/ReplicaActive/RepNotRecovering 385.032747 672 0.001224 -1180> 2014-09-18 18:08:36.695082 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0x10d166c0 con 0xcbecec0 -1179> 2014-09-18 18:08:36.695070 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11490/11490/11490) [32,42,9]/[9,42] r=1 lpr=11490 pi=10726-11489/8 luod=0'0 crt=6835'27134 lcod 0'0 active+remapped] exit Started/ReplicaActive 385.032887 0 0.000000 -1178> 2014-09-18 18:08:36.695126 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11490/11490/11490) [32,42,9]/[9,42] r=1 lpr=11490 pi=10726-11489/8 luod=0'0 crt=6835'27134 lcod 0'0 active+remapped] exit Started 386.058710 0 0.000000 -1177> 2014-09-18 18:08:36.695115 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11715:2.78f(157)) v4 -- ?+0 0xf192f00 con 0xcbecec0 -1176> 2014-09-18 18:08:36.695138 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11490/11490/11490) [32,42,9]/[9,42] r=1 lpr=11490 pi=10726-11489/8 luod=0'0 crt=6835'27134 lcod 0'0 active+remapped] enter Reset -1175> 2014-09-18 18:08:36.695405 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] exit Reset 0.000268 1 0.000260 -1174> 2014-09-18 18:08:36.695433 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] enter Started -1173> 2014-09-18 18:08:36.695447 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] enter Start -1172> 2014-09-18 18:08:36.695459 7f4c96a40700 1 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -1171> 2014-09-18 18:08:36.695413 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 luod=0'0 crt=6745'24728 lcod 0'0 active+remapped] exit Started/ReplicaActive/RepNotRecovering 5.523492 18 0.000343 -1170> 2014-09-18 18:08:36.695511 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 luod=0'0 crt=6745'24728 lcod 0'0 active+remapped] exit Started/ReplicaActive 5.523646 0 0.000000 -1169> 2014-09-18 18:08:36.695473 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] exit Start 0.000026 0 0.000000 -1168> 2014-09-18 18:08:36.695526 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 luod=0'0 crt=6745'24728 lcod 0'0 active+remapped] exit Started 6.972767 0 0.000000 -1167> 2014-09-18 18:08:36.695527 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] enter Started/Stray -1166> 2014-09-18 18:08:36.695541 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11708/11708/11572) [9,27,10]/[9,42] r=1 lpr=11708 pi=11491-11707/4 luod=0'0 crt=6745'24728 lcod 0'0 active+remapped] enter Reset -1165> 2014-09-18 18:08:36.695581 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11715/11715/11572) [6,27,10]/[9,42] r=1 lpr=11715 pi=11491-11714/5 crt=6745'24728 lcod 0'0 remapped NOTIFY] exit Reset 0.000039 1 0.000161 -1164> 2014-09-18 18:08:36.695607 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11715/11715/11572) [6,27,10]/[9,42] r=1 lpr=11715 pi=11491-11714/5 crt=6745'24728 lcod 0'0 remapped NOTIFY] enter Started -1163> 2014-09-18 18:08:36.695617 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11715/11715/11572) [6,27,10]/[9,42] r=1 lpr=11715 pi=11491-11714/5 crt=6745'24728 lcod 0'0 remapped NOTIFY] enter Start -1162> 2014-09-18 18:08:36.695626 7f4c97241700 1 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11715/11715/11572) [6,27,10]/[9,42] r=1 lpr=11715 pi=11491-11714/5 crt=6745'24728 lcod 0'0 remapped NOTIFY] state: transitioning to Stray -1161> 2014-09-18 18:08:36.695634 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 9 ==== pg_notify(2.78f(157),2.4c6(15) epoch 11715) v5 ==== 10940+0+0 (620305890 0 0) 0xe147c20 con 0x12b3b2e0 -1160> 2014-09-18 18:08:36.695637 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11715/11715/11572) [6,27,10]/[9,42] r=1 lpr=11715 pi=11491-11714/5 crt=6745'24728 lcod 0'0 remapped NOTIFY] exit Start 0.000020 0 0.000000 -1159> 2014-09-18 18:08:36.695648 7f4c9f251700 5 -- op tracker -- , seq: 86308, time: 2014-09-18 18:08:36.695420, event: header_read, op: pg_notify(2.78f(157),2.4c6(15) epoch 11715) -1158> 2014-09-18 18:08:36.695649 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.67b( v 6745'24728 (4138'21305,6745'24728] local-les=11709 n=520 ec=188 les/c 11709/11573 11715/11715/11572) [6,27,10]/[9,42] r=1 lpr=11715 pi=11491-11714/5 crt=6745'24728 lcod 0'0 remapped NOTIFY] enter Started/Stray -1157> 2014-09-18 18:08:36.695655 7f4c9f251700 5 -- op tracker -- , seq: 86308, time: 2014-09-18 18:08:36.695421, event: throttled, op: pg_notify(2.78f(157),2.4c6(15) epoch 11715) -1156> 2014-09-18 18:08:36.695660 7f4c9f251700 5 -- op tracker -- , seq: 86308, time: 2014-09-18 18:08:36.695621, event: all_read, op: pg_notify(2.78f(157),2.4c6(15) epoch 11715) -1155> 2014-09-18 18:08:36.695667 7f4c9f251700 5 -- op tracker -- , seq: 86308, time: 2014-09-18 18:08:36.695644, event: dispatched, op: pg_notify(2.78f(157),2.4c6(15) epoch 11715) -1154> 2014-09-18 18:08:36.695674 7f4c9f251700 5 -- op tracker -- , seq: 86308, time: 2014-09-18 18:08:36.695672, event: waiting_for_osdmap, op: pg_notify(2.78f(157),2.4c6(15) epoch 11715) -1153> 2014-09-18 18:08:36.695682 7f4c9f251700 5 -- op tracker -- , seq: 86308, time: 2014-09-18 18:08:36.695680, event: started, op: pg_notify(2.78f(157),2.4c6(15) epoch 11715) -1152> 2014-09-18 18:08:36.695710 7f4c9f251700 5 -- op tracker -- , seq: 86308, time: 2014-09-18 18:08:36.695708, event: done, op: pg_notify(2.78f(157),2.4c6(15) epoch 11715) -1151> 2014-09-18 18:08:36.695826 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11504/11504/10662) [10,27,6]/[10,42] r=1 lpr=11504 pi=10123-11503/15 luod=0'0 crt=6704'25402 active+remapped] exit Started/ReplicaActive/RepNotRecovering 367.560098 624 0.001251 -1150> 2014-09-18 18:08:36.695845 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11504/11504/10662) [10,27,6]/[10,42] r=1 lpr=11504 pi=10123-11503/15 luod=0'0 crt=6704'25402 active+remapped] exit Started/ReplicaActive 367.560163 0 0.000000 -1149> 2014-09-18 18:08:36.695855 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11504/11504/10662) [10,27,6]/[10,42] r=1 lpr=11504 pi=10123-11503/15 luod=0'0 crt=6704'25402 active+remapped] exit Started 371.049244 0 0.000000 -1148> 2014-09-18 18:08:36.695864 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11504/11504/10662) [10,27,6]/[10,42] r=1 lpr=11504 pi=10123-11503/15 luod=0'0 crt=6704'25402 active+remapped] enter Reset -1147> 2014-09-18 18:08:36.695868 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11455/11455/11153) [11,56,27]/[56,42] r=1 lpr=11455 pi=10199-11454/20 luod=0'0 crt=6775'26858 active+remapped] exit Started/ReplicaActive/RepNotRecovering 426.699175 777 0.001379 -1146> 2014-09-18 18:08:36.695889 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11455/11455/11153) [11,56,27]/[56,42] r=1 lpr=11455 pi=10199-11454/20 luod=0'0 crt=6775'26858 active+remapped] exit Started/ReplicaActive 426.699249 0 0.000000 -1145> 2014-09-18 18:08:36.695903 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11455/11455/11153) [11,56,27]/[56,42] r=1 lpr=11455 pi=10199-11454/20 luod=0'0 crt=6775'26858 active+remapped] exit Started 427.863945 0 0.000000 -1144> 2014-09-18 18:08:36.695915 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11455/11455/11153) [11,56,27]/[56,42] r=1 lpr=11455 pi=10199-11454/20 luod=0'0 crt=6775'26858 active+remapped] enter Reset -1143> 2014-09-18 18:08:36.695935 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] exit Reset 0.000071 1 0.000066 -1142> 2014-09-18 18:08:36.696000 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] enter Started -1141> 2014-09-18 18:08:36.696009 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] enter Start -1140> 2014-09-18 18:08:36.696017 7f4c96a40700 1 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] state: transitioning to Stray -1139> 2014-09-18 18:08:36.696025 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] exit Start 0.000016 0 0.000000 -1138> 2014-09-18 18:08:36.696004 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] exit Reset 0.000089 1 0.000081 -1137> 2014-09-18 18:08:36.696037 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] enter Started/Stray -1136> 2014-09-18 18:08:36.696044 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] enter Started -1135> 2014-09-18 18:08:36.696056 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] enter Start -1134> 2014-09-18 18:08:36.696067 7f4c97241700 1 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] state: transitioning to Stray -1133> 2014-09-18 18:08:36.696080 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] exit Start 0.000024 0 0.000000 -1132> 2014-09-18 18:08:36.696093 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] enter Started/Stray -1131> 2014-09-18 18:08:36.696115 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.4c0(9) epoch 11715) v5 -- ?+0 0xf2b01c0 con 0x1395ec60 -1130> 2014-09-18 18:08:36.696133 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6845/1010695 -- osd_map(11713..11715 src has 1..11715) v3 -- ?+0 0xcc03cc0 con 0x12b53020 -1129> 2014-09-18 18:08:36.696143 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6845/1010695 -- pg_notify(2.250(16) epoch 11715) v5 -- ?+0 0x104130e0 con 0x12b53020 -1128> 2014-09-18 18:08:36.696182 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(12) epoch 11715) v5 -- ?+0 0x10413fe0 con 0xcbecec0 -1127> 2014-09-18 18:08:36.696208 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_query(2.376 epoch 11715) v3 -- ?+0 0x10413e00 con 0x1395ec60 -1126> 2014-09-18 18:08:36.696287 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- pg_query(2.376 epoch 11715) v3 -- ?+0 0x104150c0 con 0xdd8c940 -1125> 2014-09-18 18:08:36.696304 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6835/46756 -- pg_query(2.376 epoch 11715) v3 -- ?+0 0x104125a0 con 0x142f8260 -1124> 2014-09-18 18:08:36.696610 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.67b(5) epoch 11715) v5 -- ?+0 0xf972000 con 0x1395ec60 -1123> 2014-09-18 18:08:36.696674 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11715..11715 src has 1..11715) v3 -- ?+0 0x11bb9840 con 0x23a74f20 -1122> 2014-09-18 18:08:36.696687 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11715) v5 -- ?+0 0xe1acd00 con 0x23a74f20 -1121> 2014-09-18 18:08:36.696697 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2376 ==== pg_notify(2.376 epoch 11715) v5 ==== 722+0+0 (3597790752 0 0) 0x104150c0 con 0xdd8c940 -1120> 2014-09-18 18:08:36.696713 7f4c9f251700 5 -- op tracker -- , seq: 86309, time: 2014-09-18 18:08:36.696640, event: header_read, op: pg_notify(2.376 epoch 11715) -1119> 2014-09-18 18:08:36.696722 7f4c9f251700 5 -- op tracker -- , seq: 86309, time: 2014-09-18 18:08:36.696641, event: throttled, op: pg_notify(2.376 epoch 11715) -1118> 2014-09-18 18:08:36.696722 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- osd_map(11714..11715 src has 1..11715) v3 -- ?+0 0x11bb9600 con 0x14358260 -1117> 2014-09-18 18:08:36.696729 7f4c9f251700 5 -- op tracker -- , seq: 86309, time: 2014-09-18 18:08:36.696683, event: all_read, op: pg_notify(2.376 epoch 11715) -1116> 2014-09-18 18:08:36.696735 7f4c9f251700 5 -- op tracker -- , seq: 86309, time: 2014-09-18 18:08:36.696709, event: dispatched, op: pg_notify(2.376 epoch 11715) -1115> 2014-09-18 18:08:36.696740 7f4c9f251700 5 -- op tracker -- , seq: 86309, time: 2014-09-18 18:08:36.696739, event: waiting_for_osdmap, op: pg_notify(2.376 epoch 11715) -1114> 2014-09-18 18:08:36.696748 7f4c9f251700 5 -- op tracker -- , seq: 86309, time: 2014-09-18 18:08:36.696746, event: started, op: pg_notify(2.376 epoch 11715) -1113> 2014-09-18 18:08:36.696756 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- pg_notify(2.b5(21) epoch 11715) v5 -- ?+0 0xe1aab40 con 0x14358260 -1112> 2014-09-18 18:08:36.696761 7f4c9f251700 5 -- op tracker -- , seq: 86309, time: 2014-09-18 18:08:36.696760, event: done, op: pg_notify(2.376 epoch 11715) -1111> 2014-09-18 18:08:36.696912 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.4e8(8) epoch 11715) v5 -- ?+0 0x1213bde0 con 0x1088aec0 -1110> 2014-09-18 18:08:36.696934 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(11),2.756(28) epoch 11715) v5 -- ?+0 0x128cc940 con 0x140f95a0 -1109> 2014-09-18 18:08:36.696951 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(0.34(23) epoch 11715) v5 -- ?+0 0xedfd660 con 0x1395ef20 -1108> 2014-09-18 18:08:36.697020 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(12) epoch 11715) v5 -- ?+0 0x2b92ef00 con 0xcbecec0 -1107> 2014-09-18 18:08:36.697050 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(24) epoch 11715) v5 -- ?+0 0xf4883a0 con 0x1395f1e0 -1106> 2014-09-18 18:08:36.697135 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6845/1010695 -- pg_notify(2.250(16) epoch 11715) v5 -- ?+0 0xe958d00 con 0x12b53020 -1105> 2014-09-18 18:08:36.697151 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- pg_notify(2.b5(21) epoch 11715) v5 -- ?+0 0x24f6fde0 con 0x14358260 -1104> 2014-09-18 18:08:36.697172 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3702 ==== pg_notify(2.40a(162) epoch 11715) v5 ==== 9736+0+0 (4062722493 0 0) 0x10413fe0 con 0xcbecec0 -1103> 2014-09-18 18:08:36.697188 7f4c9f251700 5 -- op tracker -- , seq: 86310, time: 2014-09-18 18:08:36.697047, event: header_read, op: pg_notify(2.40a(162) epoch 11715) -1102> 2014-09-18 18:08:36.697198 7f4c9f251700 5 -- op tracker -- , seq: 86310, time: 2014-09-18 18:08:36.697048, event: throttled, op: pg_notify(2.40a(162) epoch 11715) -1101> 2014-09-18 18:08:36.697204 7f4c9f251700 5 -- op tracker -- , seq: 86310, time: 2014-09-18 18:08:36.697159, event: all_read, op: pg_notify(2.40a(162) epoch 11715) -1100> 2014-09-18 18:08:36.697210 7f4c9f251700 5 -- op tracker -- , seq: 86310, time: 2014-09-18 18:08:36.697185, event: dispatched, op: pg_notify(2.40a(162) epoch 11715) -1099> 2014-09-18 18:08:36.697217 7f4c9f251700 5 -- op tracker -- , seq: 86310, time: 2014-09-18 18:08:36.697215, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11715) -1098> 2014-09-18 18:08:36.697234 7f4c9f251700 5 -- op tracker -- , seq: 86310, time: 2014-09-18 18:08:36.697232, event: started, op: pg_notify(2.40a(162) epoch 11715) -1097> 2014-09-18 18:08:36.697252 7f4c9f251700 5 -- op tracker -- , seq: 86310, time: 2014-09-18 18:08:36.697251, event: done, op: pg_notify(2.40a(162) epoch 11715) -1096> 2014-09-18 18:08:36.697671 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 48 ==== pg_notify(2.376(58) epoch 11715) v5 ==== 3960+0+0 (507216016 0 0) 0x104125a0 con 0x142f8260 -1095> 2014-09-18 18:08:36.697703 7f4c9f251700 5 -- op tracker -- , seq: 86311, time: 2014-09-18 18:08:36.697608, event: header_read, op: pg_notify(2.376(58) epoch 11715) -1094> 2014-09-18 18:08:36.697712 7f4c9f251700 5 -- op tracker -- , seq: 86311, time: 2014-09-18 18:08:36.697609, event: throttled, op: pg_notify(2.376(58) epoch 11715) -1093> 2014-09-18 18:08:36.697720 7f4c9f251700 5 -- op tracker -- , seq: 86311, time: 2014-09-18 18:08:36.697660, event: all_read, op: pg_notify(2.376(58) epoch 11715) -1092> 2014-09-18 18:08:36.697726 7f4c9f251700 5 -- op tracker -- , seq: 86311, time: 2014-09-18 18:08:36.697699, event: dispatched, op: pg_notify(2.376(58) epoch 11715) -1091> 2014-09-18 18:08:36.697732 7f4c9f251700 5 -- op tracker -- , seq: 86311, time: 2014-09-18 18:08:36.697731, event: waiting_for_osdmap, op: pg_notify(2.376(58) epoch 11715) -1090> 2014-09-18 18:08:36.697740 7f4c9f251700 5 -- op tracker -- , seq: 86311, time: 2014-09-18 18:08:36.697739, event: started, op: pg_notify(2.376(58) epoch 11715) -1089> 2014-09-18 18:08:36.697756 7f4c9f251700 5 -- op tracker -- , seq: 86311, time: 2014-09-18 18:08:36.697755, event: done, op: pg_notify(2.376(58) epoch 11715) -1088> 2014-09-18 18:08:36.697969 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3703 ==== pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11715) v5 ==== 6174+0+0 (662014541 0 0) 0x2b92ef00 con 0xcbecec0 -1087> 2014-09-18 18:08:36.697991 7f4c9f251700 5 -- op tracker -- , seq: 86312, time: 2014-09-18 18:08:36.697907, event: header_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11715) -1086> 2014-09-18 18:08:36.698000 7f4c9f251700 5 -- op tracker -- , seq: 86312, time: 2014-09-18 18:08:36.697907, event: throttled, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11715) -1085> 2014-09-18 18:08:36.698006 7f4c9f251700 5 -- op tracker -- , seq: 86312, time: 2014-09-18 18:08:36.697959, event: all_read, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11715) -1084> 2014-09-18 18:08:36.698013 7f4c9f251700 5 -- op tracker -- , seq: 86312, time: 2014-09-18 18:08:36.697988, event: dispatched, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11715) -1083> 2014-09-18 18:08:36.698020 7f4c9f251700 5 -- op tracker -- , seq: 86312, time: 2014-09-18 18:08:36.698018, event: waiting_for_osdmap, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11715) -1082> 2014-09-18 18:08:36.698028 7f4c9f251700 5 -- op tracker -- , seq: 86312, time: 2014-09-18 18:08:36.698026, event: started, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11715) -1081> 2014-09-18 18:08:36.698081 7f4c9f251700 5 -- op tracker -- , seq: 86312, time: 2014-09-18 18:08:36.698079, event: done, op: pg_notify(2.186(12),2.719(17),2.71c(12),2.457(21) epoch 11715) -1080> 2014-09-18 18:08:36.698785 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3704 ==== pg_notify(2.1bb(9) epoch 11715) v5 ==== 1189+0+0 (3057728084 0 0) 0xf192f00 con 0xcbecec0 -1079> 2014-09-18 18:08:36.698800 7f4c9f251700 5 -- op tracker -- , seq: 86313, time: 2014-09-18 18:08:36.698747, event: header_read, op: pg_notify(2.1bb(9) epoch 11715) -1078> 2014-09-18 18:08:36.698818 7f4c9f251700 5 -- op tracker -- , seq: 86313, time: 2014-09-18 18:08:36.698748, event: throttled, op: pg_notify(2.1bb(9) epoch 11715) -1077> 2014-09-18 18:08:36.698824 7f4c9f251700 5 -- op tracker -- , seq: 86313, time: 2014-09-18 18:08:36.698773, event: all_read, op: pg_notify(2.1bb(9) epoch 11715) -1076> 2014-09-18 18:08:36.698830 7f4c9f251700 5 -- op tracker -- , seq: 86313, time: 2014-09-18 18:08:36.698796, event: dispatched, op: pg_notify(2.1bb(9) epoch 11715) -1075> 2014-09-18 18:08:36.698836 7f4c9f251700 5 -- op tracker -- , seq: 86313, time: 2014-09-18 18:08:36.698835, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11715) -1074> 2014-09-18 18:08:36.698842 7f4c9f251700 5 -- op tracker -- , seq: 86313, time: 2014-09-18 18:08:36.698840, event: started, op: pg_notify(2.1bb(9) epoch 11715) -1073> 2014-09-18 18:08:36.698903 7f4c9f251700 5 -- op tracker -- , seq: 86313, time: 2014-09-18 18:08:36.698901, event: done, op: pg_notify(2.1bb(9) epoch 11715) -1072> 2014-09-18 18:08:36.701105 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3705 ==== pg_notify(2.78f(58) epoch 11715) v5 ==== 3828+0+0 (2813574610 0 0) 0xfccaee0 con 0xcbecec0 -1071> 2014-09-18 18:08:36.701121 7f4c9f251700 5 -- op tracker -- , seq: 86314, time: 2014-09-18 18:08:36.701008, event: header_read, op: pg_notify(2.78f(58) epoch 11715) -1070> 2014-09-18 18:08:36.701130 7f4c9f251700 5 -- op tracker -- , seq: 86314, time: 2014-09-18 18:08:36.701009, event: throttled, op: pg_notify(2.78f(58) epoch 11715) -1069> 2014-09-18 18:08:36.701136 7f4c9f251700 5 -- op tracker -- , seq: 86314, time: 2014-09-18 18:08:36.701068, event: all_read, op: pg_notify(2.78f(58) epoch 11715) -1068> 2014-09-18 18:08:36.701143 7f4c9f251700 5 -- op tracker -- , seq: 86314, time: 2014-09-18 18:08:36.701117, event: dispatched, op: pg_notify(2.78f(58) epoch 11715) -1067> 2014-09-18 18:08:36.701149 7f4c9f251700 5 -- op tracker -- , seq: 86314, time: 2014-09-18 18:08:36.701148, event: waiting_for_osdmap, op: pg_notify(2.78f(58) epoch 11715) -1066> 2014-09-18 18:08:36.701157 7f4c9f251700 5 -- op tracker -- , seq: 86314, time: 2014-09-18 18:08:36.701155, event: started, op: pg_notify(2.78f(58) epoch 11715) -1065> 2014-09-18 18:08:36.701174 7f4c9f251700 5 -- op tracker -- , seq: 86314, time: 2014-09-18 18:08:36.701172, event: done, op: pg_notify(2.78f(58) epoch 11715) -1064> 2014-09-18 18:08:36.704164 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 950 ==== pg_notify(2.204(10) epoch 11715) v5 ==== 1276+0+0 (3082602235 0 0) 0xe146960 con 0x1395f8c0 -1063> 2014-09-18 18:08:36.704193 7f4c9f251700 5 -- op tracker -- , seq: 86315, time: 2014-09-18 18:08:36.704105, event: header_read, op: pg_notify(2.204(10) epoch 11715) -1062> 2014-09-18 18:08:36.704204 7f4c9f251700 5 -- op tracker -- , seq: 86315, time: 2014-09-18 18:08:36.704106, event: throttled, op: pg_notify(2.204(10) epoch 11715) -1061> 2014-09-18 18:08:36.704210 7f4c9f251700 5 -- op tracker -- , seq: 86315, time: 2014-09-18 18:08:36.704150, event: all_read, op: pg_notify(2.204(10) epoch 11715) -1060> 2014-09-18 18:08:36.704216 7f4c9f251700 5 -- op tracker -- , seq: 86315, time: 2014-09-18 18:08:36.704189, event: dispatched, op: pg_notify(2.204(10) epoch 11715) -1059> 2014-09-18 18:08:36.704223 7f4c9f251700 5 -- op tracker -- , seq: 86315, time: 2014-09-18 18:08:36.704221, event: waiting_for_osdmap, op: pg_notify(2.204(10) epoch 11715) -1058> 2014-09-18 18:08:36.704231 7f4c9f251700 5 -- op tracker -- , seq: 86315, time: 2014-09-18 18:08:36.704229, event: started, op: pg_notify(2.204(10) epoch 11715) -1057> 2014-09-18 18:08:36.704255 7f4c9f251700 5 -- op tracker -- , seq: 86315, time: 2014-09-18 18:08:36.704253, event: done, op: pg_notify(2.204(10) epoch 11715) -1056> 2014-09-18 18:08:36.706572 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 951 ==== pg_notify(2.2d8(174) epoch 11715) v5 ==== 10808+0+0 (2893517736 0 0) 0x2b93c760 con 0x1395f8c0 -1055> 2014-09-18 18:08:36.706588 7f4c9f251700 5 -- op tracker -- , seq: 86316, time: 2014-09-18 18:08:36.706445, event: header_read, op: pg_notify(2.2d8(174) epoch 11715) -1054> 2014-09-18 18:08:36.706596 7f4c9f251700 5 -- op tracker -- , seq: 86316, time: 2014-09-18 18:08:36.706447, event: throttled, op: pg_notify(2.2d8(174) epoch 11715) -1053> 2014-09-18 18:08:36.706602 7f4c9f251700 5 -- op tracker -- , seq: 86316, time: 2014-09-18 18:08:36.706559, event: all_read, op: pg_notify(2.2d8(174) epoch 11715) -1052> 2014-09-18 18:08:36.706608 7f4c9f251700 5 -- op tracker -- , seq: 86316, time: 2014-09-18 18:08:36.706584, event: dispatched, op: pg_notify(2.2d8(174) epoch 11715) -1051> 2014-09-18 18:08:36.706614 7f4c9f251700 5 -- op tracker -- , seq: 86316, time: 2014-09-18 18:08:36.706613, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11715) -1050> 2014-09-18 18:08:36.706622 7f4c9f251700 5 -- op tracker -- , seq: 86316, time: 2014-09-18 18:08:36.706621, event: started, op: pg_notify(2.2d8(174) epoch 11715) -1049> 2014-09-18 18:08:36.706638 7f4c9f251700 5 -- op tracker -- , seq: 86316, time: 2014-09-18 18:08:36.706637, event: done, op: pg_notify(2.2d8(174) epoch 11715) -1048> 2014-09-18 18:08:36.720410 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 952 ==== pg_notify(2.204(10) epoch 11715) v5 ==== 1276+0+0 (3082602235 0 0) 0x21cfc940 con 0x1395f8c0 -1047> 2014-09-18 18:08:36.720430 7f4c9f251700 5 -- op tracker -- , seq: 86317, time: 2014-09-18 18:08:36.720308, event: header_read, op: pg_notify(2.204(10) epoch 11715) -1046> 2014-09-18 18:08:36.720440 7f4c9f251700 5 -- op tracker -- , seq: 86317, time: 2014-09-18 18:08:36.720322, event: throttled, op: pg_notify(2.204(10) epoch 11715) -1045> 2014-09-18 18:08:36.720451 7f4c9f251700 5 -- op tracker -- , seq: 86317, time: 2014-09-18 18:08:36.720361, event: all_read, op: pg_notify(2.204(10) epoch 11715) -1044> 2014-09-18 18:08:36.720462 7f4c9f251700 5 -- op tracker -- , seq: 86317, time: 2014-09-18 18:08:36.720425, event: dispatched, op: pg_notify(2.204(10) epoch 11715) -1043> 2014-09-18 18:08:36.720468 7f4c9f251700 5 -- op tracker -- , seq: 86317, time: 2014-09-18 18:08:36.720467, event: waiting_for_osdmap, op: pg_notify(2.204(10) epoch 11715) -1042> 2014-09-18 18:08:36.720476 7f4c9f251700 5 -- op tracker -- , seq: 86317, time: 2014-09-18 18:08:36.720475, event: started, op: pg_notify(2.204(10) epoch 11715) -1041> 2014-09-18 18:08:36.720495 7f4c9f251700 5 -- op tracker -- , seq: 86317, time: 2014-09-18 18:08:36.720493, event: done, op: pg_notify(2.204(10) epoch 11715) -1040> 2014-09-18 18:08:36.721203 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 953 ==== pg_info(1 pgs e11715:2.78f) v4 ==== 738+0+0 (3876468942 0 0) 0x19f38d00 con 0x1395f8c0 -1039> 2014-09-18 18:08:36.721223 7f4c9f251700 5 -- op tracker -- , seq: 86318, time: 2014-09-18 18:08:36.721133, event: header_read, op: pg_info(1 pgs e11715:2.78f) -1038> 2014-09-18 18:08:36.721233 7f4c9f251700 5 -- op tracker -- , seq: 86318, time: 2014-09-18 18:08:36.721133, event: throttled, op: pg_info(1 pgs e11715:2.78f) -1037> 2014-09-18 18:08:36.721239 7f4c9f251700 5 -- op tracker -- , seq: 86318, time: 2014-09-18 18:08:36.721159, event: all_read, op: pg_info(1 pgs e11715:2.78f) -1036> 2014-09-18 18:08:36.721245 7f4c9f251700 5 -- op tracker -- , seq: 86318, time: 2014-09-18 18:08:36.721218, event: dispatched, op: pg_info(1 pgs e11715:2.78f) -1035> 2014-09-18 18:08:36.721252 7f4c9f251700 5 -- op tracker -- , seq: 86318, time: 2014-09-18 18:08:36.721250, event: waiting_for_osdmap, op: pg_info(1 pgs e11715:2.78f) -1034> 2014-09-18 18:08:36.721260 7f4c9f251700 5 -- op tracker -- , seq: 86318, time: 2014-09-18 18:08:36.721258, event: started, op: pg_info(1 pgs e11715:2.78f) -1033> 2014-09-18 18:08:36.721277 7f4c9f251700 5 -- op tracker -- , seq: 86318, time: 2014-09-18 18:08:36.721276, event: done, op: pg_info(1 pgs e11715:2.78f) -1032> 2014-09-18 18:08:36.721728 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 954 ==== pg_notify(2.2d8(174) epoch 11715) v5 ==== 10808+0+0 (2893517736 0 0) 0x104143a0 con 0x1395f8c0 -1031> 2014-09-18 18:08:36.721748 7f4c9f251700 5 -- op tracker -- , seq: 86319, time: 2014-09-18 18:08:36.721575, event: header_read, op: pg_notify(2.2d8(174) epoch 11715) -1030> 2014-09-18 18:08:36.721758 7f4c9f251700 5 -- op tracker -- , seq: 86319, time: 2014-09-18 18:08:36.721576, event: throttled, op: pg_notify(2.2d8(174) epoch 11715) -1029> 2014-09-18 18:08:36.721764 7f4c9f251700 5 -- op tracker -- , seq: 86319, time: 2014-09-18 18:08:36.721691, event: all_read, op: pg_notify(2.2d8(174) epoch 11715) -1028> 2014-09-18 18:08:36.721770 7f4c9f251700 5 -- op tracker -- , seq: 86319, time: 2014-09-18 18:08:36.721743, event: dispatched, op: pg_notify(2.2d8(174) epoch 11715) -1027> 2014-09-18 18:08:36.721777 7f4c9f251700 5 -- op tracker -- , seq: 86319, time: 2014-09-18 18:08:36.721775, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11715) -1026> 2014-09-18 18:08:36.721785 7f4c9f251700 5 -- op tracker -- , seq: 86319, time: 2014-09-18 18:08:36.721783, event: started, op: pg_notify(2.2d8(174) epoch 11715) -1025> 2014-09-18 18:08:36.721803 7f4c9f251700 5 -- op tracker -- , seq: 86319, time: 2014-09-18 18:08:36.721801, event: done, op: pg_notify(2.2d8(174) epoch 11715) -1024> 2014-09-18 18:08:36.735497 7f4caae41700 5 osd.42 11715 tick -1023> 2014-09-18 18:08:36.745642 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.37 10.242.42.168:0/18319 109 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:36.612236) v2 ==== 47+0+0 (1909511321 0 0) 0x117383c0 con 0x11ffdce0 -1022> 2014-09-18 18:08:36.745674 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:36.612236) v2 -- ?+0 0x2b986d20 con 0x11ffdce0 -1021> 2014-09-18 18:08:36.745711 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.37 10.242.42.168:0/18319 109 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:36.612236) v2 ==== 47+0+0 (1909511321 0 0) 0x11fd0b40 con 0x110bb860 -1020> 2014-09-18 18:08:36.745741 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/18319 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:36.612236) v2 -- ?+0 0xfae9a20 con 0x110bb860 -1019> 2014-09-18 18:08:36.751017 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3251 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x10d18640 con 0x1395ec60 -1018> 2014-09-18 18:08:36.751034 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -1017> 2014-09-18 18:08:36.751063 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3252 ==== pg_notify(2.71(19) epoch 11715) v5 ==== 1795+0+0 (2939585226 0 0) 0xf972000 con 0x1395ec60 -1016> 2014-09-18 18:08:36.751074 7f4c9f251700 5 -- op tracker -- , seq: 86320, time: 2014-09-18 18:08:36.751017, event: header_read, op: pg_notify(2.71(19) epoch 11715) -1015> 2014-09-18 18:08:36.751083 7f4c9f251700 5 -- op tracker -- , seq: 86320, time: 2014-09-18 18:08:36.751018, event: throttled, op: pg_notify(2.71(19) epoch 11715) -1014> 2014-09-18 18:08:36.751089 7f4c9f251700 5 -- op tracker -- , seq: 86320, time: 2014-09-18 18:08:36.751053, event: all_read, op: pg_notify(2.71(19) epoch 11715) -1013> 2014-09-18 18:08:36.751095 7f4c9f251700 5 -- op tracker -- , seq: 86320, time: 2014-09-18 18:08:36.751071, event: dispatched, op: pg_notify(2.71(19) epoch 11715) -1012> 2014-09-18 18:08:36.751101 7f4c9f251700 5 -- op tracker -- , seq: 86320, time: 2014-09-18 18:08:36.751100, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11715) -1011> 2014-09-18 18:08:36.751109 7f4c9f251700 5 -- op tracker -- , seq: 86320, time: 2014-09-18 18:08:36.751107, event: started, op: pg_notify(2.71(19) epoch 11715) -1010> 2014-09-18 18:08:36.751126 7f4c9f251700 5 -- op tracker -- , seq: 86320, time: 2014-09-18 18:08:36.751125, event: done, op: pg_notify(2.71(19) epoch 11715) -1009> 2014-09-18 18:08:36.753141 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3253 ==== pg_notify(2.376(10) epoch 11715) v5 ==== 1296+0+0 (1718467983 0 0) 0x10413e00 con 0x1395ec60 -1008> 2014-09-18 18:08:36.753155 7f4c9f251700 5 -- op tracker -- , seq: 86321, time: 2014-09-18 18:08:36.753100, event: header_read, op: pg_notify(2.376(10) epoch 11715) -1007> 2014-09-18 18:08:36.753163 7f4c9f251700 5 -- op tracker -- , seq: 86321, time: 2014-09-18 18:08:36.753100, event: throttled, op: pg_notify(2.376(10) epoch 11715) -1006> 2014-09-18 18:08:36.753169 7f4c9f251700 5 -- op tracker -- , seq: 86321, time: 2014-09-18 18:08:36.753130, event: all_read, op: pg_notify(2.376(10) epoch 11715) -1005> 2014-09-18 18:08:36.753175 7f4c9f251700 5 -- op tracker -- , seq: 86321, time: 2014-09-18 18:08:36.753152, event: dispatched, op: pg_notify(2.376(10) epoch 11715) -1004> 2014-09-18 18:08:36.753182 7f4c9f251700 5 -- op tracker -- , seq: 86321, time: 2014-09-18 18:08:36.753180, event: waiting_for_osdmap, op: pg_notify(2.376(10) epoch 11715) -1003> 2014-09-18 18:08:36.753189 7f4c9f251700 5 -- op tracker -- , seq: 86321, time: 2014-09-18 18:08:36.753188, event: started, op: pg_notify(2.376(10) epoch 11715) -1002> 2014-09-18 18:08:36.753205 7f4c9f251700 5 -- op tracker -- , seq: 86321, time: 2014-09-18 18:08:36.753204, event: done, op: pg_notify(2.376(10) epoch 11715) -1001> 2014-09-18 18:08:36.753221 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.058731 5 0.000270 -1000> 2014-09-18 18:08:36.753342 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -999> 2014-09-18 18:08:36.753435 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000093 0 0.000000 -998> 2014-09-18 18:08:36.753458 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.058979 0 0.000000 -997> 2014-09-18 18:08:36.753468 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] exit Started/Primary 0.059006 0 0.000000 -996> 2014-09-18 18:08:36.753478 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -995> 2014-09-18 18:08:36.753486 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering/WaitActingChange -994> 2014-09-18 18:08:36.753513 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -993> 2014-09-18 18:08:36.753519 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x143b5c00 con 0x111771e0 -992> 2014-09-18 18:08:36.756384 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3254 ==== pg_query(2.67b epoch 11715) v3 ==== 144+0+0 (831056390 0 0) 0xf2b01c0 con 0x1395ec60 -991> 2014-09-18 18:08:36.756396 7f4c9f251700 5 -- op tracker -- , seq: 86322, time: 2014-09-18 18:08:36.756352, event: header_read, op: pg_query(2.67b epoch 11715) -990> 2014-09-18 18:08:36.756402 7f4c9f251700 5 -- op tracker -- , seq: 86322, time: 2014-09-18 18:08:36.756353, event: throttled, op: pg_query(2.67b epoch 11715) -989> 2014-09-18 18:08:36.756406 7f4c9f251700 5 -- op tracker -- , seq: 86322, time: 2014-09-18 18:08:36.756373, event: all_read, op: pg_query(2.67b epoch 11715) -988> 2014-09-18 18:08:36.756410 7f4c9f251700 5 -- op tracker -- , seq: 86322, time: 2014-09-18 18:08:36.756393, event: dispatched, op: pg_query(2.67b epoch 11715) -987> 2014-09-18 18:08:36.756414 7f4c9f251700 5 -- op tracker -- , seq: 86322, time: 2014-09-18 18:08:36.756413, event: waiting_for_osdmap, op: pg_query(2.67b epoch 11715) -986> 2014-09-18 18:08:36.756420 7f4c9f251700 5 -- op tracker -- , seq: 86322, time: 2014-09-18 18:08:36.756418, event: started, op: pg_query(2.67b epoch 11715) -985> 2014-09-18 18:08:36.756432 7f4c9f251700 5 -- op tracker -- , seq: 86322, time: 2014-09-18 18:08:36.756430, event: done, op: pg_query(2.67b epoch 11715) -984> 2014-09-18 18:08:36.756450 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.67b(5) epoch 11715) v5 -- ?+0 0xf487c20 con 0x1395ec60 -983> 2014-09-18 18:08:36.756478 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -982> 2014-09-18 18:08:36.756482 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x1416d800 con 0x111771e0 -981> 2014-09-18 18:08:36.770240 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.1 10.242.42.164:6857/39817 10 ==== pg_info(1 pgs e11715:2.78f) v4 ==== 738+0+0 (3876468942 0 0) 0xf7283a0 con 0x12b3b2e0 -980> 2014-09-18 18:08:36.770261 7f4c9f251700 5 -- op tracker -- , seq: 86323, time: 2014-09-18 18:08:36.770192, event: header_read, op: pg_info(1 pgs e11715:2.78f) -979> 2014-09-18 18:08:36.770277 7f4c9f251700 5 -- op tracker -- , seq: 86323, time: 2014-09-18 18:08:36.770193, event: throttled, op: pg_info(1 pgs e11715:2.78f) -978> 2014-09-18 18:08:36.770286 7f4c9f251700 5 -- op tracker -- , seq: 86323, time: 2014-09-18 18:08:36.770222, event: all_read, op: pg_info(1 pgs e11715:2.78f) -977> 2014-09-18 18:08:36.770293 7f4c9f251700 5 -- op tracker -- , seq: 86323, time: 2014-09-18 18:08:36.770256, event: dispatched, op: pg_info(1 pgs e11715:2.78f) -976> 2014-09-18 18:08:36.770299 7f4c9f251700 5 -- op tracker -- , seq: 86323, time: 2014-09-18 18:08:36.770298, event: waiting_for_osdmap, op: pg_info(1 pgs e11715:2.78f) -975> 2014-09-18 18:08:36.770311 7f4c9f251700 5 -- op tracker -- , seq: 86323, time: 2014-09-18 18:08:36.770308, event: started, op: pg_info(1 pgs e11715:2.78f) -974> 2014-09-18 18:08:36.770337 7f4c9f251700 5 -- op tracker -- , seq: 86323, time: 2014-09-18 18:08:36.770334, event: done, op: pg_info(1 pgs e11715:2.78f) -973> 2014-09-18 18:08:36.770349 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -972> 2014-09-18 18:08:36.770358 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x143b4000 con 0x111771e0 -971> 2014-09-18 18:08:36.812361 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5127 ==== pg_notify(2.62(44) epoch 11713) v5 ==== 3220+0+0 (877466242 0 0) 0x1213bde0 con 0x1088aec0 -970> 2014-09-18 18:08:36.812396 7f4c9f251700 5 -- op tracker -- , seq: 86324, time: 2014-09-18 18:08:36.812208, event: header_read, op: pg_notify(2.62(44) epoch 11713) -969> 2014-09-18 18:08:36.812412 7f4c9f251700 5 -- op tracker -- , seq: 86324, time: 2014-09-18 18:08:36.812211, event: throttled, op: pg_notify(2.62(44) epoch 11713) -968> 2014-09-18 18:08:36.812419 7f4c9f251700 5 -- op tracker -- , seq: 86324, time: 2014-09-18 18:08:36.812330, event: all_read, op: pg_notify(2.62(44) epoch 11713) -967> 2014-09-18 18:08:36.812425 7f4c9f251700 5 -- op tracker -- , seq: 86324, time: 2014-09-18 18:08:36.812387, event: dispatched, op: pg_notify(2.62(44) epoch 11713) -966> 2014-09-18 18:08:36.812432 7f4c9f251700 5 -- op tracker -- , seq: 86324, time: 2014-09-18 18:08:36.812431, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11713) -965> 2014-09-18 18:08:36.812442 7f4c9f251700 5 -- op tracker -- , seq: 86324, time: 2014-09-18 18:08:36.812440, event: started, op: pg_notify(2.62(44) epoch 11713) -964> 2014-09-18 18:08:36.812489 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -963> 2014-09-18 18:08:36.812499 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x1416da00 con 0x111771e0 -962> 2014-09-18 18:08:36.812570 7f4c9f251700 5 -- op tracker -- , seq: 86324, time: 2014-09-18 18:08:36.812568, event: done, op: pg_notify(2.62(44) epoch 11713) -961> 2014-09-18 18:08:36.816099 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5128 ==== pg_notify(2.3b6(10) epoch 11713) v5 ==== 1260+0+0 (2893521525 0 0) 0xf5783a0 con 0x1088aec0 -960> 2014-09-18 18:08:36.816119 7f4c9f251700 5 -- op tracker -- , seq: 86325, time: 2014-09-18 18:08:36.816031, event: header_read, op: pg_notify(2.3b6(10) epoch 11713) -959> 2014-09-18 18:08:36.816130 7f4c9f251700 5 -- op tracker -- , seq: 86325, time: 2014-09-18 18:08:36.816032, event: throttled, op: pg_notify(2.3b6(10) epoch 11713) -958> 2014-09-18 18:08:36.816137 7f4c9f251700 5 -- op tracker -- , seq: 86325, time: 2014-09-18 18:08:36.816065, event: all_read, op: pg_notify(2.3b6(10) epoch 11713) -957> 2014-09-18 18:08:36.816143 7f4c9f251700 5 -- op tracker -- , seq: 86325, time: 2014-09-18 18:08:36.816115, event: dispatched, op: pg_notify(2.3b6(10) epoch 11713) -956> 2014-09-18 18:08:36.816150 7f4c9f251700 5 -- op tracker -- , seq: 86325, time: 2014-09-18 18:08:36.816148, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11713) -955> 2014-09-18 18:08:36.816158 7f4c9f251700 5 -- op tracker -- , seq: 86325, time: 2014-09-18 18:08:36.816156, event: started, op: pg_notify(2.3b6(10) epoch 11713) -954> 2014-09-18 18:08:36.816177 7f4c9f251700 5 -- op tracker -- , seq: 86325, time: 2014-09-18 18:08:36.816176, event: done, op: pg_notify(2.3b6(10) epoch 11713) -953> 2014-09-18 18:08:36.816196 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -952> 2014-09-18 18:08:36.816204 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x143b3000 con 0x111771e0 -951> 2014-09-18 18:08:36.816637 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1865 ==== osd_map(11713..11713 src has 1..11713) v3 ==== 235+0+0 (831042895 0 0) 0x1533a480 con 0x10f60520 -950> 2014-09-18 18:08:36.816663 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11713,11713], i have 11715, src has [1,11713] -949> 2014-09-18 18:08:36.816675 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1866 ==== pg_notify(2.4f6(15) epoch 11713) v5 ==== 1581+0+0 (323309909 0 0) 0xf5772c0 con 0x10f60520 -948> 2014-09-18 18:08:36.816686 7f4c9f251700 5 -- op tracker -- , seq: 86326, time: 2014-09-18 18:08:36.816606, event: header_read, op: pg_notify(2.4f6(15) epoch 11713) -947> 2014-09-18 18:08:36.816696 7f4c9f251700 5 -- op tracker -- , seq: 86326, time: 2014-09-18 18:08:36.816606, event: throttled, op: pg_notify(2.4f6(15) epoch 11713) -946> 2014-09-18 18:08:36.816703 7f4c9f251700 5 -- op tracker -- , seq: 86326, time: 2014-09-18 18:08:36.816644, event: all_read, op: pg_notify(2.4f6(15) epoch 11713) -945> 2014-09-18 18:08:36.816709 7f4c9f251700 5 -- op tracker -- , seq: 86326, time: 2014-09-18 18:08:36.816683, event: dispatched, op: pg_notify(2.4f6(15) epoch 11713) -944> 2014-09-18 18:08:36.816716 7f4c9f251700 5 -- op tracker -- , seq: 86326, time: 2014-09-18 18:08:36.816714, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11713) -943> 2014-09-18 18:08:36.816724 7f4c9f251700 5 -- op tracker -- , seq: 86326, time: 2014-09-18 18:08:36.816722, event: started, op: pg_notify(2.4f6(15) epoch 11713) -942> 2014-09-18 18:08:36.816759 7f4c9f251700 5 -- op tracker -- , seq: 86326, time: 2014-09-18 18:08:36.816758, event: done, op: pg_notify(2.4f6(15) epoch 11713) -941> 2014-09-18 18:08:36.816774 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -940> 2014-09-18 18:08:36.816812 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x1416a400 con 0x111771e0 -939> 2014-09-18 18:08:36.819475 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5129 ==== pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11713) v5 ==== 4538+0+0 (862895070 0 0) 0x21e10780 con 0x1088aec0 -938> 2014-09-18 18:08:36.819493 7f4c9f251700 5 -- op tracker -- , seq: 86327, time: 2014-09-18 18:08:36.819398, event: header_read, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11713) -937> 2014-09-18 18:08:36.819504 7f4c9f251700 5 -- op tracker -- , seq: 86327, time: 2014-09-18 18:08:36.819399, event: throttled, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11713) -936> 2014-09-18 18:08:36.819511 7f4c9f251700 5 -- op tracker -- , seq: 86327, time: 2014-09-18 18:08:36.819461, event: all_read, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11713) -935> 2014-09-18 18:08:36.819517 7f4c9f251700 5 -- op tracker -- , seq: 86327, time: 2014-09-18 18:08:36.819489, event: dispatched, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11713) -934> 2014-09-18 18:08:36.819524 7f4c9f251700 5 -- op tracker -- , seq: 86327, time: 2014-09-18 18:08:36.819522, event: waiting_for_osdmap, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11713) -933> 2014-09-18 18:08:36.819532 7f4c9f251700 5 -- op tracker -- , seq: 86327, time: 2014-09-18 18:08:36.819530, event: started, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11713) -932> 2014-09-18 18:08:36.819563 7f4c9f251700 5 -- op tracker -- , seq: 86327, time: 2014-09-18 18:08:36.819561, event: done, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11713) -931> 2014-09-18 18:08:36.819591 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -930> 2014-09-18 18:08:36.819599 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x14f12000 con 0x111771e0 -929> 2014-09-18 18:08:36.819622 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -928> 2014-09-18 18:08:36.819635 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x143b3800 con 0x111771e0 -927> 2014-09-18 18:08:36.819710 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1867 ==== pg_notify(2.517(11) epoch 11713) v5 ==== 1335+0+0 (1301671767 0 0) 0xf2b8ee0 con 0x10f60520 -926> 2014-09-18 18:08:36.819728 7f4c9f251700 5 -- op tracker -- , seq: 86328, time: 2014-09-18 18:08:36.819636, event: header_read, op: pg_notify(2.517(11) epoch 11713) -925> 2014-09-18 18:08:36.819738 7f4c9f251700 5 -- op tracker -- , seq: 86328, time: 2014-09-18 18:08:36.819637, event: throttled, op: pg_notify(2.517(11) epoch 11713) -924> 2014-09-18 18:08:36.819748 7f4c9f251700 5 -- op tracker -- , seq: 86328, time: 2014-09-18 18:08:36.819697, event: all_read, op: pg_notify(2.517(11) epoch 11713) -923> 2014-09-18 18:08:36.819758 7f4c9f251700 5 -- op tracker -- , seq: 86328, time: 2014-09-18 18:08:36.819723, event: dispatched, op: pg_notify(2.517(11) epoch 11713) -922> 2014-09-18 18:08:36.819768 7f4c9f251700 5 -- op tracker -- , seq: 86328, time: 2014-09-18 18:08:36.819766, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11713) -921> 2014-09-18 18:08:36.819780 7f4c9f251700 5 -- op tracker -- , seq: 86328, time: 2014-09-18 18:08:36.819777, event: started, op: pg_notify(2.517(11) epoch 11713) -920> 2014-09-18 18:08:36.819801 7f4c9f251700 5 -- op tracker -- , seq: 86328, time: 2014-09-18 18:08:36.819799, event: done, op: pg_notify(2.517(11) epoch 11713) -919> 2014-09-18 18:08:36.819810 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -918> 2014-09-18 18:08:36.819815 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x14f14e00 con 0x111771e0 -917> 2014-09-18 18:08:36.820216 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1868 ==== pg_notify(2.1d8(9) epoch 11713) v5 ==== 1261+0+0 (561800178 0 0) 0xec69a40 con 0x10f60520 -916> 2014-09-18 18:08:36.820232 7f4c9f251700 5 -- op tracker -- , seq: 86329, time: 2014-09-18 18:08:36.820163, event: header_read, op: pg_notify(2.1d8(9) epoch 11713) -915> 2014-09-18 18:08:36.820243 7f4c9f251700 5 -- op tracker -- , seq: 86329, time: 2014-09-18 18:08:36.820164, event: throttled, op: pg_notify(2.1d8(9) epoch 11713) -914> 2014-09-18 18:08:36.820253 7f4c9f251700 5 -- op tracker -- , seq: 86329, time: 2014-09-18 18:08:36.820203, event: all_read, op: pg_notify(2.1d8(9) epoch 11713) -913> 2014-09-18 18:08:36.820263 7f4c9f251700 5 -- op tracker -- , seq: 86329, time: 2014-09-18 18:08:36.820228, event: dispatched, op: pg_notify(2.1d8(9) epoch 11713) -912> 2014-09-18 18:08:36.820273 7f4c9f251700 5 -- op tracker -- , seq: 86329, time: 2014-09-18 18:08:36.820270, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11713) -911> 2014-09-18 18:08:36.820284 7f4c9f251700 5 -- op tracker -- , seq: 86329, time: 2014-09-18 18:08:36.820281, event: started, op: pg_notify(2.1d8(9) epoch 11713) -910> 2014-09-18 18:08:36.820306 7f4c9f251700 5 -- op tracker -- , seq: 86329, time: 2014-09-18 18:08:36.820304, event: done, op: pg_notify(2.1d8(9) epoch 11713) -909> 2014-09-18 18:08:36.820353 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -908> 2014-09-18 18:08:36.820362 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9]} v11715) v1 -- ?+0 0x143b5400 con 0x111771e0 -907> 2014-09-18 18:08:36.844685 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1869 ==== osd_map(11714..11715 src has 1..11715) v3 ==== 4843+0+0 (2326239219 0 0) 0x1440a1c0 con 0x10f60520 -906> 2014-09-18 18:08:36.844717 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11714,11715], i have 11715, src has [1,11715] -905> 2014-09-18 18:08:36.844728 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1870 ==== pg_notify(2.763 epoch 11715) v5 ==== 722+0+0 (2192209722 0 0) 0xf09b0e0 con 0x10f60520 -904> 2014-09-18 18:08:36.844744 7f4c9f251700 5 -- op tracker -- , seq: 86330, time: 2014-09-18 18:08:36.844687, event: header_read, op: pg_notify(2.763 epoch 11715) -903> 2014-09-18 18:08:36.844760 7f4c9f251700 5 -- op tracker -- , seq: 86330, time: 2014-09-18 18:08:36.844687, event: throttled, op: pg_notify(2.763 epoch 11715) -902> 2014-09-18 18:08:36.844779 7f4c9f251700 5 -- op tracker -- , seq: 86330, time: 2014-09-18 18:08:36.844715, event: all_read, op: pg_notify(2.763 epoch 11715) -901> 2014-09-18 18:08:36.844789 7f4c9f251700 5 -- op tracker -- , seq: 86330, time: 2014-09-18 18:08:36.844738, event: dispatched, op: pg_notify(2.763 epoch 11715) -900> 2014-09-18 18:08:36.844799 7f4c9f251700 5 -- op tracker -- , seq: 86330, time: 2014-09-18 18:08:36.844797, event: waiting_for_osdmap, op: pg_notify(2.763 epoch 11715) -899> 2014-09-18 18:08:36.844811 7f4c9f251700 5 -- op tracker -- , seq: 86330, time: 2014-09-18 18:08:36.844809, event: started, op: pg_notify(2.763 epoch 11715) -898> 2014-09-18 18:08:36.844836 7f4c9f251700 5 -- op tracker -- , seq: 86330, time: 2014-09-18 18:08:36.844834, event: done, op: pg_notify(2.763 epoch 11715) -897> 2014-09-18 18:08:36.844864 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.158315 7 0.000182 -896> 2014-09-18 18:08:36.844948 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1871 ==== pg_notify(2.4f6(15) epoch 11715) v5 ==== 1581+0+0 (3163922072 0 0) 0xf09cb20 con 0x10f60520 -895> 2014-09-18 18:08:36.844961 7f4c9f251700 5 -- op tracker -- , seq: 86331, time: 2014-09-18 18:08:36.844905, event: header_read, op: pg_notify(2.4f6(15) epoch 11715) -894> 2014-09-18 18:08:36.844972 7f4c9f251700 5 -- op tracker -- , seq: 86331, time: 2014-09-18 18:08:36.844906, event: throttled, op: pg_notify(2.4f6(15) epoch 11715) -893> 2014-09-18 18:08:36.844932 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -892> 2014-09-18 18:08:36.844982 7f4c9f251700 5 -- op tracker -- , seq: 86331, time: 2014-09-18 18:08:36.844937, event: all_read, op: pg_notify(2.4f6(15) epoch 11715) -891> 2014-09-18 18:08:36.845003 7f4c9f251700 5 -- op tracker -- , seq: 86331, time: 2014-09-18 18:08:36.844957, event: dispatched, op: pg_notify(2.4f6(15) epoch 11715) -890> 2014-09-18 18:08:36.845013 7f4c9f251700 5 -- op tracker -- , seq: 86331, time: 2014-09-18 18:08:36.845011, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11715) -889> 2014-09-18 18:08:36.845025 7f4c9f251700 5 -- op tracker -- , seq: 86331, time: 2014-09-18 18:08:36.845022, event: started, op: pg_notify(2.4f6(15) epoch 11715) -888> 2014-09-18 18:08:36.845057 7f4c9f251700 5 -- op tracker -- , seq: 86331, time: 2014-09-18 18:08:36.845055, event: done, op: pg_notify(2.4f6(15) epoch 11715) -887> 2014-09-18 18:08:36.845061 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000129 0 0.000000 -886> 2014-09-18 18:08:36.845070 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1872 ==== pg_notify(2.2d8 epoch 11715) v5 ==== 722+0+0 (2127269869 0 0) 0xf09b2c0 con 0x10f60520 -885> 2014-09-18 18:08:36.845085 7f4c9f251700 5 -- op tracker -- , seq: 86332, time: 2014-09-18 18:08:36.844957, event: header_read, op: pg_notify(2.2d8 epoch 11715) -884> 2014-09-18 18:08:36.845090 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -883> 2014-09-18 18:08:36.845095 7f4c9f251700 5 -- op tracker -- , seq: 86332, time: 2014-09-18 18:08:36.844957, event: throttled, op: pg_notify(2.2d8 epoch 11715) -882> 2014-09-18 18:08:36.845101 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x143b3a00 con 0x111771e0 -881> 2014-09-18 18:08:36.845105 7f4c9f251700 5 -- op tracker -- , seq: 86332, time: 2014-09-18 18:08:36.844975, event: all_read, op: pg_notify(2.2d8 epoch 11715) -880> 2014-09-18 18:08:36.845126 7f4c9f251700 5 -- op tracker -- , seq: 86332, time: 2014-09-18 18:08:36.845081, event: dispatched, op: pg_notify(2.2d8 epoch 11715) -879> 2014-09-18 18:08:36.845136 7f4c9f251700 5 -- op tracker -- , seq: 86332, time: 2014-09-18 18:08:36.845134, event: waiting_for_osdmap, op: pg_notify(2.2d8 epoch 11715) -878> 2014-09-18 18:08:36.845076 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.158557 0 0.000000 -877> 2014-09-18 18:08:36.845148 7f4c9f251700 5 -- op tracker -- , seq: 86332, time: 2014-09-18 18:08:36.845146, event: started, op: pg_notify(2.2d8 epoch 11715) -876> 2014-09-18 18:08:36.845148 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] exit Started/Primary 0.158637 0 0.000000 -875> 2014-09-18 18:08:36.845157 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] enter Started/Primary -874> 2014-09-18 18:08:36.845165 7f4c96a40700 5 osd.42 pg_epoch: 11715 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] enter Started/Primary/Peering/WaitActingChange -873> 2014-09-18 18:08:36.845171 7f4c9f251700 5 -- op tracker -- , seq: 86332, time: 2014-09-18 18:08:36.845168, event: done, op: pg_notify(2.2d8 epoch 11715) -872> 2014-09-18 18:08:36.845183 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -871> 2014-09-18 18:08:36.845188 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0xfcf3a00 con 0x111771e0 -870> 2014-09-18 18:08:36.845183 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.152540 6 0.000535 -869> 2014-09-18 18:08:36.845219 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -868> 2014-09-18 18:08:36.845257 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 bft=2,27 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000037 0 0.000000 -867> 2014-09-18 18:08:36.845350 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 bft=2,27 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -866> 2014-09-18 18:08:36.845423 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 bft=2,27 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000072 0 0.000000 -865> 2014-09-18 18:08:36.845439 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 bft=2,27 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -864> 2014-09-18 18:08:36.845538 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -863> 2014-09-18 18:08:36.845558 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x12599600 con 0x111771e0 -862> 2014-09-18 18:08:36.845852 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -861> 2014-09-18 18:08:36.845865 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0xfcf1a00 con 0x111771e0 -860> 2014-09-18 18:08:36.846991 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5130 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x12bf1b00 con 0x1088aec0 -859> 2014-09-18 18:08:36.847012 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -858> 2014-09-18 18:08:36.847074 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5131 ==== pg_notify(2.11b(8),2.120(11) epoch 11715) v5 ==== 2507+0+0 (3825173911 0 0) 0xe76cb40 con 0x1088aec0 -857> 2014-09-18 18:08:36.847096 7f4c9f251700 5 -- op tracker -- , seq: 86333, time: 2014-09-18 18:08:36.846994, event: header_read, op: pg_notify(2.11b(8),2.120(11) epoch 11715) -856> 2014-09-18 18:08:36.847109 7f4c9f251700 5 -- op tracker -- , seq: 86333, time: 2014-09-18 18:08:36.846994, event: throttled, op: pg_notify(2.11b(8),2.120(11) epoch 11715) -855> 2014-09-18 18:08:36.847113 7f4c9f251700 5 -- op tracker -- , seq: 86333, time: 2014-09-18 18:08:36.847057, event: all_read, op: pg_notify(2.11b(8),2.120(11) epoch 11715) -854> 2014-09-18 18:08:36.847122 7f4c9f251700 5 -- op tracker -- , seq: 86333, time: 2014-09-18 18:08:36.847093, event: dispatched, op: pg_notify(2.11b(8),2.120(11) epoch 11715) -853> 2014-09-18 18:08:36.847127 7f4c9f251700 5 -- op tracker -- , seq: 86333, time: 2014-09-18 18:08:36.847125, event: waiting_for_osdmap, op: pg_notify(2.11b(8),2.120(11) epoch 11715) -852> 2014-09-18 18:08:36.847138 7f4c9f251700 5 -- op tracker -- , seq: 86333, time: 2014-09-18 18:08:36.847137, event: started, op: pg_notify(2.11b(8),2.120(11) epoch 11715) -851> 2014-09-18 18:08:36.847166 7f4c9f251700 5 -- op tracker -- , seq: 86333, time: 2014-09-18 18:08:36.847164, event: done, op: pg_notify(2.11b(8),2.120(11) epoch 11715) -850> 2014-09-18 18:08:36.847186 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -849> 2014-09-18 18:08:36.847197 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1259b400 con 0x111771e0 -848> 2014-09-18 18:08:36.847221 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -847> 2014-09-18 18:08:36.847232 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0xfcf1000 con 0x111771e0 -846> 2014-09-18 18:08:36.848444 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1873 ==== pg_notify(2.517(11) epoch 11715) v5 ==== 1335+0+0 (1336319703 0 0) 0xf09d660 con 0x10f60520 -845> 2014-09-18 18:08:36.848460 7f4c9f251700 5 -- op tracker -- , seq: 86334, time: 2014-09-18 18:08:36.848371, event: header_read, op: pg_notify(2.517(11) epoch 11715) -844> 2014-09-18 18:08:36.848478 7f4c9f251700 5 -- op tracker -- , seq: 86334, time: 2014-09-18 18:08:36.848372, event: throttled, op: pg_notify(2.517(11) epoch 11715) -843> 2014-09-18 18:08:36.848484 7f4c9f251700 5 -- op tracker -- , seq: 86334, time: 2014-09-18 18:08:36.848430, event: all_read, op: pg_notify(2.517(11) epoch 11715) -842> 2014-09-18 18:08:36.848490 7f4c9f251700 5 -- op tracker -- , seq: 86334, time: 2014-09-18 18:08:36.848456, event: dispatched, op: pg_notify(2.517(11) epoch 11715) -841> 2014-09-18 18:08:36.848497 7f4c9f251700 5 -- op tracker -- , seq: 86334, time: 2014-09-18 18:08:36.848495, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11715) -840> 2014-09-18 18:08:36.848504 7f4c9f251700 5 -- op tracker -- , seq: 86334, time: 2014-09-18 18:08:36.848503, event: started, op: pg_notify(2.517(11) epoch 11715) -839> 2014-09-18 18:08:36.848522 7f4c9f251700 5 -- op tracker -- , seq: 86334, time: 2014-09-18 18:08:36.848520, event: done, op: pg_notify(2.517(11) epoch 11715) -838> 2014-09-18 18:08:36.848540 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -837> 2014-09-18 18:08:36.848548 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1259bc00 con 0x111771e0 -836> 2014-09-18 18:08:36.849214 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.2 10.242.42.164:6817/19139 1874 ==== pg_notify(2.1d8(9) epoch 11715) v5 ==== 1261+0+0 (2562675737 0 0) 0x19f22d20 con 0x10f60520 -835> 2014-09-18 18:08:36.849230 7f4c9f251700 5 -- op tracker -- , seq: 86335, time: 2014-09-18 18:08:36.849150, event: header_read, op: pg_notify(2.1d8(9) epoch 11715) -834> 2014-09-18 18:08:36.849241 7f4c9f251700 5 -- op tracker -- , seq: 86335, time: 2014-09-18 18:08:36.849150, event: throttled, op: pg_notify(2.1d8(9) epoch 11715) -833> 2014-09-18 18:08:36.849251 7f4c9f251700 5 -- op tracker -- , seq: 86335, time: 2014-09-18 18:08:36.849199, event: all_read, op: pg_notify(2.1d8(9) epoch 11715) -832> 2014-09-18 18:08:36.849261 7f4c9f251700 5 -- op tracker -- , seq: 86335, time: 2014-09-18 18:08:36.849225, event: dispatched, op: pg_notify(2.1d8(9) epoch 11715) -831> 2014-09-18 18:08:36.849271 7f4c9f251700 5 -- op tracker -- , seq: 86335, time: 2014-09-18 18:08:36.849269, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11715) -830> 2014-09-18 18:08:36.849283 7f4c9f251700 5 -- op tracker -- , seq: 86335, time: 2014-09-18 18:08:36.849280, event: started, op: pg_notify(2.1d8(9) epoch 11715) -829> 2014-09-18 18:08:36.849307 7f4c9f251700 5 -- op tracker -- , seq: 86335, time: 2014-09-18 18:08:36.849304, event: done, op: pg_notify(2.1d8(9) epoch 11715) -828> 2014-09-18 18:08:36.849360 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -827> 2014-09-18 18:08:36.849384 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0xfcf3200 con 0x111771e0 -826> 2014-09-18 18:08:36.914988 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3255 ==== pg_notify(2.204(64) epoch 11715) v5 ==== 4270+0+0 (1224785509 0 0) 0xf487c20 con 0x1395ec60 -825> 2014-09-18 18:08:36.915028 7f4c9f251700 5 -- op tracker -- , seq: 86336, time: 2014-09-18 18:08:36.914841, event: header_read, op: pg_notify(2.204(64) epoch 11715) -824> 2014-09-18 18:08:36.915040 7f4c9f251700 5 -- op tracker -- , seq: 86336, time: 2014-09-18 18:08:36.914843, event: throttled, op: pg_notify(2.204(64) epoch 11715) -823> 2014-09-18 18:08:36.915046 7f4c9f251700 5 -- op tracker -- , seq: 86336, time: 2014-09-18 18:08:36.914945, event: all_read, op: pg_notify(2.204(64) epoch 11715) -822> 2014-09-18 18:08:36.915052 7f4c9f251700 5 -- op tracker -- , seq: 86336, time: 2014-09-18 18:08:36.915021, event: dispatched, op: pg_notify(2.204(64) epoch 11715) -821> 2014-09-18 18:08:36.915060 7f4c9f251700 5 -- op tracker -- , seq: 86336, time: 2014-09-18 18:08:36.915058, event: waiting_for_osdmap, op: pg_notify(2.204(64) epoch 11715) -820> 2014-09-18 18:08:36.915070 7f4c9f251700 5 -- op tracker -- , seq: 86336, time: 2014-09-18 18:08:36.915068, event: started, op: pg_notify(2.204(64) epoch 11715) -819> 2014-09-18 18:08:36.915091 7f4c9f251700 5 -- op tracker -- , seq: 86336, time: 2014-09-18 18:08:36.915089, event: done, op: pg_notify(2.204(64) epoch 11715) -818> 2014-09-18 18:08:36.915121 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -817> 2014-09-18 18:08:36.915131 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1259ac00 con 0x111771e0 -816> 2014-09-18 18:08:36.917201 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3256 ==== pg_notify(2.519(4) epoch 11715) v5 ==== 962+0+0 (2923169691 0 0) 0x21cfda20 con 0x1395ec60 -815> 2014-09-18 18:08:36.917226 7f4c9f251700 5 -- op tracker -- , seq: 86337, time: 2014-09-18 18:08:36.917150, event: header_read, op: pg_notify(2.519(4) epoch 11715) -814> 2014-09-18 18:08:36.917242 7f4c9f251700 5 -- op tracker -- , seq: 86337, time: 2014-09-18 18:08:36.917152, event: throttled, op: pg_notify(2.519(4) epoch 11715) -813> 2014-09-18 18:08:36.917253 7f4c9f251700 5 -- op tracker -- , seq: 86337, time: 2014-09-18 18:08:36.917182, event: all_read, op: pg_notify(2.519(4) epoch 11715) -812> 2014-09-18 18:08:36.917263 7f4c9f251700 5 -- op tracker -- , seq: 86337, time: 2014-09-18 18:08:36.917219, event: dispatched, op: pg_notify(2.519(4) epoch 11715) -811> 2014-09-18 18:08:36.917274 7f4c9f251700 5 -- op tracker -- , seq: 86337, time: 2014-09-18 18:08:36.917272, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11715) -810> 2014-09-18 18:08:36.917286 7f4c9f251700 5 -- op tracker -- , seq: 86337, time: 2014-09-18 18:08:36.917283, event: started, op: pg_notify(2.519(4) epoch 11715) -809> 2014-09-18 18:08:36.917315 7f4c9f251700 5 -- op tracker -- , seq: 86337, time: 2014-09-18 18:08:36.917313, event: done, op: pg_notify(2.519(4) epoch 11715) -808> 2014-09-18 18:08:36.917382 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -807> 2014-09-18 18:08:36.917398 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0xfcf3600 con 0x111771e0 -806> 2014-09-18 18:08:36.919750 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3257 ==== pg_notify(2.2d8(174) epoch 11715) v5 ==== 10808+0+0 (1672781233 0 0) 0x2b93ffe0 con 0x1395ec60 -805> 2014-09-18 18:08:36.919770 7f4c9f251700 5 -- op tracker -- , seq: 86338, time: 2014-09-18 18:08:36.919559, event: header_read, op: pg_notify(2.2d8(174) epoch 11715) -804> 2014-09-18 18:08:36.919780 7f4c9f251700 5 -- op tracker -- , seq: 86338, time: 2014-09-18 18:08:36.919560, event: throttled, op: pg_notify(2.2d8(174) epoch 11715) -803> 2014-09-18 18:08:36.919786 7f4c9f251700 5 -- op tracker -- , seq: 86338, time: 2014-09-18 18:08:36.919699, event: all_read, op: pg_notify(2.2d8(174) epoch 11715) -802> 2014-09-18 18:08:36.919792 7f4c9f251700 5 -- op tracker -- , seq: 86338, time: 2014-09-18 18:08:36.919765, event: dispatched, op: pg_notify(2.2d8(174) epoch 11715) -801> 2014-09-18 18:08:36.919799 7f4c9f251700 5 -- op tracker -- , seq: 86338, time: 2014-09-18 18:08:36.919797, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11715) -800> 2014-09-18 18:08:36.919807 7f4c9f251700 5 -- op tracker -- , seq: 86338, time: 2014-09-18 18:08:36.919805, event: started, op: pg_notify(2.2d8(174) epoch 11715) -799> 2014-09-18 18:08:36.919826 7f4c9f251700 5 -- op tracker -- , seq: 86338, time: 2014-09-18 18:08:36.919824, event: done, op: pg_notify(2.2d8(174) epoch 11715) -798> 2014-09-18 18:08:36.919851 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -797> 2014-09-18 18:08:36.919860 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1259a800 con 0x111771e0 -796> 2014-09-18 18:08:36.921439 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3258 ==== pg_notify(2.410(12) epoch 11715) v5 ==== 1376+0+0 (800186017 0 0) 0x2b941a20 con 0x1395ec60 -795> 2014-09-18 18:08:36.921458 7f4c9f251700 5 -- op tracker -- , seq: 86339, time: 2014-09-18 18:08:36.921367, event: header_read, op: pg_notify(2.410(12) epoch 11715) -794> 2014-09-18 18:08:36.921469 7f4c9f251700 5 -- op tracker -- , seq: 86339, time: 2014-09-18 18:08:36.921368, event: throttled, op: pg_notify(2.410(12) epoch 11715) -793> 2014-09-18 18:08:36.921475 7f4c9f251700 5 -- op tracker -- , seq: 86339, time: 2014-09-18 18:08:36.921402, event: all_read, op: pg_notify(2.410(12) epoch 11715) -792> 2014-09-18 18:08:36.921481 7f4c9f251700 5 -- op tracker -- , seq: 86339, time: 2014-09-18 18:08:36.921454, event: dispatched, op: pg_notify(2.410(12) epoch 11715) -791> 2014-09-18 18:08:36.921488 7f4c9f251700 5 -- op tracker -- , seq: 86339, time: 2014-09-18 18:08:36.921486, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11715) -790> 2014-09-18 18:08:36.921496 7f4c9f251700 5 -- op tracker -- , seq: 86339, time: 2014-09-18 18:08:36.921494, event: started, op: pg_notify(2.410(12) epoch 11715) -789> 2014-09-18 18:08:36.921514 7f4c9f251700 5 -- op tracker -- , seq: 86339, time: 2014-09-18 18:08:36.921513, event: done, op: pg_notify(2.410(12) epoch 11715) -788> 2014-09-18 18:08:36.921570 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -787> 2014-09-18 18:08:36.921581 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0xfcf2600 con 0x111771e0 -786> 2014-09-18 18:08:36.922332 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3259 ==== pg_notify(2.186(12) epoch 11715) v5 ==== 1386+0+0 (628231526 0 0) 0x2b93fc20 con 0x1395ec60 -785> 2014-09-18 18:08:36.922357 7f4c9f251700 5 -- op tracker -- , seq: 86340, time: 2014-09-18 18:08:36.922281, event: header_read, op: pg_notify(2.186(12) epoch 11715) -784> 2014-09-18 18:08:36.922371 7f4c9f251700 5 -- op tracker -- , seq: 86340, time: 2014-09-18 18:08:36.922282, event: throttled, op: pg_notify(2.186(12) epoch 11715) -783> 2014-09-18 18:08:36.922381 7f4c9f251700 5 -- op tracker -- , seq: 86340, time: 2014-09-18 18:08:36.922316, event: all_read, op: pg_notify(2.186(12) epoch 11715) -782> 2014-09-18 18:08:36.922391 7f4c9f251700 5 -- op tracker -- , seq: 86340, time: 2014-09-18 18:08:36.922351, event: dispatched, op: pg_notify(2.186(12) epoch 11715) -781> 2014-09-18 18:08:36.922402 7f4c9f251700 5 -- op tracker -- , seq: 86340, time: 2014-09-18 18:08:36.922400, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11715) -780> 2014-09-18 18:08:36.922413 7f4c9f251700 5 -- op tracker -- , seq: 86340, time: 2014-09-18 18:08:36.922411, event: started, op: pg_notify(2.186(12) epoch 11715) -779> 2014-09-18 18:08:36.922434 7f4c9f251700 5 -- op tracker -- , seq: 86340, time: 2014-09-18 18:08:36.922432, event: done, op: pg_notify(2.186(12) epoch 11715) -778> 2014-09-18 18:08:36.922453 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -777> 2014-09-18 18:08:36.922462 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x5143000 con 0x111771e0 -776> 2014-09-18 18:08:36.929308 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5132 ==== pg_notify(2.40a(162),2.410(12) epoch 11715) v5 ==== 11062+0+0 (785501698 0 0) 0xe76c000 con 0x1088aec0 -775> 2014-09-18 18:08:36.929331 7f4c9f251700 5 -- op tracker -- , seq: 86341, time: 2014-09-18 18:08:36.929142, event: header_read, op: pg_notify(2.40a(162),2.410(12) epoch 11715) -774> 2014-09-18 18:08:36.929341 7f4c9f251700 5 -- op tracker -- , seq: 86341, time: 2014-09-18 18:08:36.929146, event: throttled, op: pg_notify(2.40a(162),2.410(12) epoch 11715) -773> 2014-09-18 18:08:36.929349 7f4c9f251700 5 -- op tracker -- , seq: 86341, time: 2014-09-18 18:08:36.929288, event: all_read, op: pg_notify(2.40a(162),2.410(12) epoch 11715) -772> 2014-09-18 18:08:36.929356 7f4c9f251700 5 -- op tracker -- , seq: 86341, time: 2014-09-18 18:08:36.929326, event: dispatched, op: pg_notify(2.40a(162),2.410(12) epoch 11715) -771> 2014-09-18 18:08:36.929364 7f4c9f251700 5 -- op tracker -- , seq: 86341, time: 2014-09-18 18:08:36.929362, event: waiting_for_osdmap, op: pg_notify(2.40a(162),2.410(12) epoch 11715) -770> 2014-09-18 18:08:36.929373 7f4c9f251700 5 -- op tracker -- , seq: 86341, time: 2014-09-18 18:08:36.929371, event: started, op: pg_notify(2.40a(162),2.410(12) epoch 11715) -769> 2014-09-18 18:08:36.929408 7f4c9f251700 5 -- op tracker -- , seq: 86341, time: 2014-09-18 18:08:36.929406, event: done, op: pg_notify(2.40a(162),2.410(12) epoch 11715) -768> 2014-09-18 18:08:36.929432 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -767> 2014-09-18 18:08:36.929442 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x119f6400 con 0x111771e0 -766> 2014-09-18 18:08:36.931176 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5133 ==== pg_query(2.43f epoch 11715) v3 ==== 144+0+0 (408489788 0 0) 0xe76c960 con 0x1088aec0 -765> 2014-09-18 18:08:36.931197 7f4c9f251700 5 -- op tracker -- , seq: 86342, time: 2014-09-18 18:08:36.931133, event: header_read, op: pg_query(2.43f epoch 11715) -764> 2014-09-18 18:08:36.931211 7f4c9f251700 5 -- op tracker -- , seq: 86342, time: 2014-09-18 18:08:36.931137, event: throttled, op: pg_query(2.43f epoch 11715) -763> 2014-09-18 18:08:36.931221 7f4c9f251700 5 -- op tracker -- , seq: 86342, time: 2014-09-18 18:08:36.931160, event: all_read, op: pg_query(2.43f epoch 11715) -762> 2014-09-18 18:08:36.931230 7f4c9f251700 5 -- op tracker -- , seq: 86342, time: 2014-09-18 18:08:36.931192, event: dispatched, op: pg_query(2.43f epoch 11715) -761> 2014-09-18 18:08:36.931240 7f4c9f251700 5 -- op tracker -- , seq: 86342, time: 2014-09-18 18:08:36.931238, event: waiting_for_osdmap, op: pg_query(2.43f epoch 11715) -760> 2014-09-18 18:08:36.931252 7f4c9f251700 5 -- op tracker -- , seq: 86342, time: 2014-09-18 18:08:36.931250, event: started, op: pg_query(2.43f epoch 11715) -759> 2014-09-18 18:08:36.931275 7f4c9f251700 5 -- op tracker -- , seq: 86342, time: 2014-09-18 18:08:36.931273, event: done, op: pg_query(2.43f epoch 11715) -758> 2014-09-18 18:08:36.931536 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.43f(14) epoch 11715) v5 -- ?+0 0xe1aa1e0 con 0x1088aec0 -757> 2014-09-18 18:08:36.931556 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -756> 2014-09-18 18:08:36.931561 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x143b2c00 con 0x111771e0 -755> 2014-09-18 18:08:36.931978 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5134 ==== pg_notify(2.179(10) epoch 11715) v5 ==== 1276+0+0 (3191187933 0 0) 0xe1aa1e0 con 0x1088aec0 -754> 2014-09-18 18:08:36.931998 7f4c9f251700 5 -- op tracker -- , seq: 86343, time: 2014-09-18 18:08:36.931894, event: header_read, op: pg_notify(2.179(10) epoch 11715) -753> 2014-09-18 18:08:36.932009 7f4c9f251700 5 -- op tracker -- , seq: 86343, time: 2014-09-18 18:08:36.931895, event: throttled, op: pg_notify(2.179(10) epoch 11715) -752> 2014-09-18 18:08:36.932017 7f4c9f251700 5 -- op tracker -- , seq: 86343, time: 2014-09-18 18:08:36.931937, event: all_read, op: pg_notify(2.179(10) epoch 11715) -751> 2014-09-18 18:08:36.932026 7f4c9f251700 5 -- op tracker -- , seq: 86343, time: 2014-09-18 18:08:36.931994, event: dispatched, op: pg_notify(2.179(10) epoch 11715) -750> 2014-09-18 18:08:36.932032 7f4c9f251700 5 -- op tracker -- , seq: 86343, time: 2014-09-18 18:08:36.932031, event: waiting_for_osdmap, op: pg_notify(2.179(10) epoch 11715) -749> 2014-09-18 18:08:36.932041 7f4c9f251700 5 -- op tracker -- , seq: 86343, time: 2014-09-18 18:08:36.932039, event: started, op: pg_notify(2.179(10) epoch 11715) -748> 2014-09-18 18:08:36.932060 7f4c9f251700 5 -- op tracker -- , seq: 86343, time: 2014-09-18 18:08:36.932058, event: done, op: pg_notify(2.179(10) epoch 11715) -747> 2014-09-18 18:08:36.932081 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -746> 2014-09-18 18:08:36.932091 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x119f7800 con 0x111771e0 -745> 2014-09-18 18:08:36.941818 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.14 10.242.42.168:0/16237 143 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:36.808387) v2 ==== 47+0+0 (4208886028 0 0) 0xe0bd840 con 0x101cc3c0 -744> 2014-09-18 18:08:36.941849 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:36.808387) v2 -- ?+0 0x117383c0 con 0x101cc3c0 -743> 2014-09-18 18:08:36.941861 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.14 10.242.42.168:0/16237 143 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:36.808387) v2 ==== 47+0+0 (4208886028 0 0) 0xf7a8ee0 con 0xcc49a20 -742> 2014-09-18 18:08:36.941889 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/16237 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:36.808387) v2 -- ?+0 0x11fd0b40 con 0xcc49a20 -741> 2014-09-18 18:08:36.959454 7f4c8e22f700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3b2e0).reader couldn't read tag, (0) Success -740> 2014-09-18 18:08:36.959484 7f4c8e22f700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3b2e0).fault (0) Success -739> 2014-09-18 18:08:36.959506 7f4c8e22f700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=2 pgs=32 cs=1 l=0 c=0x12b3b2e0).fault with nothing to send, going to standby -738> 2014-09-18 18:08:36.959595 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50680 sd=70 :36903 s=2 pgs=7 cs=1 l=1 c=0x2a7f5860).reader couldn't read tag, (0) Success -737> 2014-09-18 18:08:36.959600 7f4c885e8700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4edc0 sd=54 :46648 s=2 pgs=7 cs=1 l=1 c=0x1d7c8ec0).reader couldn't read tag, (0) Success -736> 2014-09-18 18:08:36.959654 7f4c885e8700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4edc0 sd=54 :46648 s=2 pgs=7 cs=1 l=1 c=0x1d7c8ec0).fault (0) Success -735> 2014-09-18 18:08:36.959648 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50680 sd=70 :36903 s=2 pgs=7 cs=1 l=1 c=0x2a7f5860).fault (0) Success -734> 2014-09-18 18:08:36.959749 7f4c9da4e700 1 -- 10.242.42.178:0/2483 mark_down 0x2a7f5860 -- pipe dne -733> 2014-09-18 18:08:36.959848 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).connect error 10.242.42.164:6858/39817, (111) Connection refused -732> 2014-09-18 18:08:36.959860 7f4c773de700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).connect error 10.242.42.164:6859/39817, (111) Connection refused -731> 2014-09-18 18:08:36.959885 7f4c773de700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).fault (111) Connection refused -730> 2014-09-18 18:08:36.959883 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).fault (111) Connection refused -729> 2014-09-18 18:08:36.959913 7f4c773de700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).fault -728> 2014-09-18 18:08:36.959917 7f4c71984700 0 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).fault -727> 2014-09-18 18:08:36.959966 7f4c773de700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).connect error 10.242.42.164:6859/39817, (111) Connection refused -726> 2014-09-18 18:08:36.959971 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).connect error 10.242.42.164:6858/39817, (111) Connection refused -725> 2014-09-18 18:08:36.960003 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).fault (111) Connection refused -724> 2014-09-18 18:08:36.959987 7f4c773de700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).fault (111) Connection refused -723> 2014-09-18 18:08:36.974129 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5135 ==== pg_notify(2.1a2(20) epoch 11715) v5 ==== 1798+0+0 (3888766600 0 0) 0xe76f0c0 con 0x1088aec0 -722> 2014-09-18 18:08:36.974147 7f4c9f251700 5 -- op tracker -- , seq: 86344, time: 2014-09-18 18:08:36.974061, event: header_read, op: pg_notify(2.1a2(20) epoch 11715) -721> 2014-09-18 18:08:36.974158 7f4c9f251700 5 -- op tracker -- , seq: 86344, time: 2014-09-18 18:08:36.974062, event: throttled, op: pg_notify(2.1a2(20) epoch 11715) -720> 2014-09-18 18:08:36.974165 7f4c9f251700 5 -- op tracker -- , seq: 86344, time: 2014-09-18 18:08:36.974113, event: all_read, op: pg_notify(2.1a2(20) epoch 11715) -719> 2014-09-18 18:08:36.974172 7f4c9f251700 5 -- op tracker -- , seq: 86344, time: 2014-09-18 18:08:36.974142, event: dispatched, op: pg_notify(2.1a2(20) epoch 11715) -718> 2014-09-18 18:08:36.974182 7f4c9f251700 5 -- op tracker -- , seq: 86344, time: 2014-09-18 18:08:36.974179, event: waiting_for_osdmap, op: pg_notify(2.1a2(20) epoch 11715) -717> 2014-09-18 18:08:36.974195 7f4c9f251700 5 -- op tracker -- , seq: 86344, time: 2014-09-18 18:08:36.974192, event: started, op: pg_notify(2.1a2(20) epoch 11715) -716> 2014-09-18 18:08:36.974235 7f4c9f251700 5 -- op tracker -- , seq: 86344, time: 2014-09-18 18:08:36.974233, event: done, op: pg_notify(2.1a2(20) epoch 11715) -715> 2014-09-18 18:08:36.974267 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -714> 2014-09-18 18:08:36.974276 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x143b4600 con 0x111771e0 -713> 2014-09-18 18:08:36.985748 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1292 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xeeea400 con 0x1f102840 -712> 2014-09-18 18:08:36.985769 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -711> 2014-09-18 18:08:36.985788 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.4 10.242.42.164:6819/22324 1293 ==== pg_notify(2.4f6(15) epoch 11715) v5 ==== 1543+0+0 (2985096385 0 0) 0xe1a2780 con 0x1f102840 -710> 2014-09-18 18:08:36.985799 7f4c9f251700 5 -- op tracker -- , seq: 86345, time: 2014-09-18 18:08:36.985746, event: header_read, op: pg_notify(2.4f6(15) epoch 11715) -709> 2014-09-18 18:08:36.985807 7f4c9f251700 5 -- op tracker -- , seq: 86345, time: 2014-09-18 18:08:36.985746, event: throttled, op: pg_notify(2.4f6(15) epoch 11715) -708> 2014-09-18 18:08:36.985816 7f4c9f251700 5 -- op tracker -- , seq: 86345, time: 2014-09-18 18:08:36.985778, event: all_read, op: pg_notify(2.4f6(15) epoch 11715) -707> 2014-09-18 18:08:36.985824 7f4c9f251700 5 -- op tracker -- , seq: 86345, time: 2014-09-18 18:08:36.985796, event: dispatched, op: pg_notify(2.4f6(15) epoch 11715) -706> 2014-09-18 18:08:36.985832 7f4c9f251700 5 -- op tracker -- , seq: 86345, time: 2014-09-18 18:08:36.985831, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11715) -705> 2014-09-18 18:08:36.985840 7f4c9f251700 5 -- op tracker -- , seq: 86345, time: 2014-09-18 18:08:36.985838, event: started, op: pg_notify(2.4f6(15) epoch 11715) -704> 2014-09-18 18:08:36.985858 7f4c9f251700 5 -- op tracker -- , seq: 86345, time: 2014-09-18 18:08:36.985857, event: done, op: pg_notify(2.4f6(15) epoch 11715) -703> 2014-09-18 18:08:36.985892 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -702> 2014-09-18 18:08:36.985904 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x119f7c00 con 0x111771e0 -701> 2014-09-18 18:08:36.988134 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3260 ==== pg_notify(2.74e(11) epoch 11715) v5 ==== 1303+0+0 (2639101182 0 0) 0x2b93e1e0 con 0x1395ec60 -700> 2014-09-18 18:08:36.988149 7f4c9f251700 5 -- op tracker -- , seq: 86346, time: 2014-09-18 18:08:36.988070, event: header_read, op: pg_notify(2.74e(11) epoch 11715) -699> 2014-09-18 18:08:36.988157 7f4c9f251700 5 -- op tracker -- , seq: 86346, time: 2014-09-18 18:08:36.988071, event: throttled, op: pg_notify(2.74e(11) epoch 11715) -698> 2014-09-18 18:08:36.988163 7f4c9f251700 5 -- op tracker -- , seq: 86346, time: 2014-09-18 18:08:36.988108, event: all_read, op: pg_notify(2.74e(11) epoch 11715) -697> 2014-09-18 18:08:36.988169 7f4c9f251700 5 -- op tracker -- , seq: 86346, time: 2014-09-18 18:08:36.988145, event: dispatched, op: pg_notify(2.74e(11) epoch 11715) -696> 2014-09-18 18:08:36.988175 7f4c9f251700 5 -- op tracker -- , seq: 86346, time: 2014-09-18 18:08:36.988174, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11715) -695> 2014-09-18 18:08:36.988183 7f4c9f251700 5 -- op tracker -- , seq: 86346, time: 2014-09-18 18:08:36.988182, event: started, op: pg_notify(2.74e(11) epoch 11715) -694> 2014-09-18 18:08:36.988207 7f4c9f251700 5 -- op tracker -- , seq: 86346, time: 2014-09-18 18:08:36.988204, event: done, op: pg_notify(2.74e(11) epoch 11715) -693> 2014-09-18 18:08:36.988246 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -692> 2014-09-18 18:08:36.988256 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x143b3600 con 0x111771e0 -691> 2014-09-18 18:08:37.002523 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5136 ==== pg_notify(2.4c1(15) epoch 11715) v5 ==== 1507+0+0 (1154921543 0 0) 0xe76d860 con 0x1088aec0 -690> 2014-09-18 18:08:37.002721 7f4c9f251700 5 -- op tracker -- , seq: 86347, time: 2014-09-18 18:08:37.002411, event: header_read, op: pg_notify(2.4c1(15) epoch 11715) -689> 2014-09-18 18:08:37.002730 7f4c9f251700 5 -- op tracker -- , seq: 86347, time: 2014-09-18 18:08:37.002412, event: throttled, op: pg_notify(2.4c1(15) epoch 11715) -688> 2014-09-18 18:08:37.002737 7f4c9f251700 5 -- op tracker -- , seq: 86347, time: 2014-09-18 18:08:37.002504, event: all_read, op: pg_notify(2.4c1(15) epoch 11715) -687> 2014-09-18 18:08:37.002741 7f4c9f251700 5 -- op tracker -- , seq: 86347, time: 2014-09-18 18:08:37.002715, event: dispatched, op: pg_notify(2.4c1(15) epoch 11715) -686> 2014-09-18 18:08:37.002749 7f4c9f251700 5 -- op tracker -- , seq: 86347, time: 2014-09-18 18:08:37.002748, event: waiting_for_osdmap, op: pg_notify(2.4c1(15) epoch 11715) -685> 2014-09-18 18:08:37.002759 7f4c9f251700 5 -- op tracker -- , seq: 86347, time: 2014-09-18 18:08:37.002757, event: started, op: pg_notify(2.4c1(15) epoch 11715) -684> 2014-09-18 18:08:37.002778 7f4c9f251700 5 -- op tracker -- , seq: 86347, time: 2014-09-18 18:08:37.002776, event: done, op: pg_notify(2.4c1(15) epoch 11715) -683> 2014-09-18 18:08:37.002892 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -682> 2014-09-18 18:08:37.002902 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x119f7e00 con 0x111771e0 -681> 2014-09-18 18:08:37.012313 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5137 ==== pg_notify(2.62(44) epoch 11715) v5 ==== 3220+0+0 (1242916863 0 0) 0xe76e760 con 0x1088aec0 -680> 2014-09-18 18:08:37.012336 7f4c9f251700 5 -- op tracker -- , seq: 86348, time: 2014-09-18 18:08:37.012243, event: header_read, op: pg_notify(2.62(44) epoch 11715) -679> 2014-09-18 18:08:37.012359 7f4c9f251700 5 -- op tracker -- , seq: 86348, time: 2014-09-18 18:08:37.012244, event: throttled, op: pg_notify(2.62(44) epoch 11715) -678> 2014-09-18 18:08:37.012368 7f4c9f251700 5 -- op tracker -- , seq: 86348, time: 2014-09-18 18:08:37.012296, event: all_read, op: pg_notify(2.62(44) epoch 11715) -677> 2014-09-18 18:08:37.012377 7f4c9f251700 5 -- op tracker -- , seq: 86348, time: 2014-09-18 18:08:37.012331, event: dispatched, op: pg_notify(2.62(44) epoch 11715) -676> 2014-09-18 18:08:37.012385 7f4c9f251700 5 -- op tracker -- , seq: 86348, time: 2014-09-18 18:08:37.012383, event: waiting_for_osdmap, op: pg_notify(2.62(44) epoch 11715) -675> 2014-09-18 18:08:37.012394 7f4c9f251700 5 -- op tracker -- , seq: 86348, time: 2014-09-18 18:08:37.012393, event: started, op: pg_notify(2.62(44) epoch 11715) -674> 2014-09-18 18:08:37.012414 7f4c9f251700 5 -- op tracker -- , seq: 86348, time: 2014-09-18 18:08:37.012413, event: done, op: pg_notify(2.62(44) epoch 11715) -673> 2014-09-18 18:08:37.012435 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -672> 2014-09-18 18:08:37.012446 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x143b4a00 con 0x111771e0 -671> 2014-09-18 18:08:37.022118 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.32 10.242.42.176:0/1099 114 ==== osd_ping(ping e11715 stamp 2014-09-18 23:37:41.769863) v2 ==== 47+0+0 (1276189138 0 0) 0x2b9301c0 con 0x12abd080 -670> 2014-09-18 18:08:37.022164 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.176:0/1099 -- osd_ping(ping_reply e11715 stamp 2014-09-18 23:37:41.769863) v2 -- ?+0 0xe0bd840 con 0x12abd080 -669> 2014-09-18 18:08:37.022160 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.32 10.242.42.176:0/1099 114 ==== osd_ping(ping e11715 stamp 2014-09-18 23:37:41.769863) v2 ==== 47+0+0 (1276189138 0 0) 0xe1a2b40 con 0x110bb2e0 -668> 2014-09-18 18:08:37.022195 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.176:0/1099 -- osd_ping(ping_reply e11715 stamp 2014-09-18 23:37:41.769863) v2 -- ?+0 0xf7a8ee0 con 0x110bb2e0 -667> 2014-09-18 18:08:37.022738 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5138 ==== pg_notify(2.3b6(10) epoch 11715) v5 ==== 1260+0+0 (1669275590 0 0) 0xe76d0e0 con 0x1088aec0 -666> 2014-09-18 18:08:37.022755 7f4c9f251700 5 -- op tracker -- , seq: 86349, time: 2014-09-18 18:08:37.022692, event: header_read, op: pg_notify(2.3b6(10) epoch 11715) -665> 2014-09-18 18:08:37.022763 7f4c9f251700 5 -- op tracker -- , seq: 86349, time: 2014-09-18 18:08:37.022692, event: throttled, op: pg_notify(2.3b6(10) epoch 11715) -664> 2014-09-18 18:08:37.022769 7f4c9f251700 5 -- op tracker -- , seq: 86349, time: 2014-09-18 18:08:37.022725, event: all_read, op: pg_notify(2.3b6(10) epoch 11715) -663> 2014-09-18 18:08:37.022775 7f4c9f251700 5 -- op tracker -- , seq: 86349, time: 2014-09-18 18:08:37.022751, event: dispatched, op: pg_notify(2.3b6(10) epoch 11715) -662> 2014-09-18 18:08:37.022782 7f4c9f251700 5 -- op tracker -- , seq: 86349, time: 2014-09-18 18:08:37.022780, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11715) -661> 2014-09-18 18:08:37.022794 7f4c9f251700 5 -- op tracker -- , seq: 86349, time: 2014-09-18 18:08:37.022791, event: started, op: pg_notify(2.3b6(10) epoch 11715) -660> 2014-09-18 18:08:37.022822 7f4c9f251700 5 -- op tracker -- , seq: 86349, time: 2014-09-18 18:08:37.022819, event: done, op: pg_notify(2.3b6(10) epoch 11715) -659> 2014-09-18 18:08:37.022838 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -658> 2014-09-18 18:08:37.022848 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x119f8000 con 0x111771e0 -657> 2014-09-18 18:08:37.033197 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5139 ==== pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11715) v5 ==== 4538+0+0 (3784163492 0 0) 0xe76fa20 con 0x1088aec0 -656> 2014-09-18 18:08:37.033214 7f4c9f251700 5 -- op tracker -- , seq: 86350, time: 2014-09-18 18:08:37.033077, event: header_read, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11715) -655> 2014-09-18 18:08:37.033222 7f4c9f251700 5 -- op tracker -- , seq: 86350, time: 2014-09-18 18:08:37.033081, event: throttled, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11715) -654> 2014-09-18 18:08:37.033226 7f4c9f251700 5 -- op tracker -- , seq: 86350, time: 2014-09-18 18:08:37.033171, event: all_read, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11715) -653> 2014-09-18 18:08:37.033231 7f4c9f251700 5 -- op tracker -- , seq: 86350, time: 2014-09-18 18:08:37.033210, event: dispatched, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11715) -652> 2014-09-18 18:08:37.033238 7f4c9f251700 5 -- op tracker -- , seq: 86350, time: 2014-09-18 18:08:37.033236, event: waiting_for_osdmap, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11715) -651> 2014-09-18 18:08:37.033252 7f4c9f251700 5 -- op tracker -- , seq: 86350, time: 2014-09-18 18:08:37.033250, event: started, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11715) -650> 2014-09-18 18:08:37.033305 7f4c9f251700 5 -- op tracker -- , seq: 86350, time: 2014-09-18 18:08:37.033303, event: done, op: pg_notify(2.6f8(10),2.517(11),2.519(23) epoch 11715) -649> 2014-09-18 18:08:37.033320 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -648> 2014-09-18 18:08:37.033321 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.6 10.242.42.166:6808/22481 5140 ==== pg_query(2.237 epoch 11715) v3 ==== 144+0+0 (800850753 0 0) 0xe76eee0 con 0x1088aec0 -647> 2014-09-18 18:08:37.033334 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0xdbfa800 con 0x111771e0 -646> 2014-09-18 18:08:37.033345 7f4c9f251700 5 -- op tracker -- , seq: 86351, time: 2014-09-18 18:08:37.033197, event: header_read, op: pg_query(2.237 epoch 11715) -645> 2014-09-18 18:08:37.033348 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -644> 2014-09-18 18:08:37.033353 7f4c9f251700 5 -- op tracker -- , seq: 86351, time: 2014-09-18 18:08:37.033197, event: throttled, op: pg_query(2.237 epoch 11715) -643> 2014-09-18 18:08:37.033365 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x119f9a00 con 0x111771e0 -642> 2014-09-18 18:08:37.033372 7f4c9f251700 5 -- op tracker -- , seq: 86351, time: 2014-09-18 18:08:37.033212, event: all_read, op: pg_query(2.237 epoch 11715) -641> 2014-09-18 18:08:37.033380 7f4c9f251700 5 -- op tracker -- , seq: 86351, time: 2014-09-18 18:08:37.033342, event: dispatched, op: pg_query(2.237 epoch 11715) -640> 2014-09-18 18:08:37.033386 7f4c9f251700 5 -- op tracker -- , seq: 86351, time: 2014-09-18 18:08:37.033385, event: waiting_for_osdmap, op: pg_query(2.237 epoch 11715) -639> 2014-09-18 18:08:37.033394 7f4c9f251700 5 -- op tracker -- , seq: 86351, time: 2014-09-18 18:08:37.033393, event: started, op: pg_query(2.237 epoch 11715) -638> 2014-09-18 18:08:37.033407 7f4c9f251700 5 -- op tracker -- , seq: 86351, time: 2014-09-18 18:08:37.033406, event: done, op: pg_query(2.237 epoch 11715) -637> 2014-09-18 18:08:37.033936 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.237(131) epoch 11715) v5 -- ?+0 0x21cfc940 con 0x1088aec0 -636> 2014-09-18 18:08:37.033951 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -635> 2014-09-18 18:08:37.033968 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1259b600 con 0x111771e0 -634> 2014-09-18 18:08:37.077150 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3706 ==== pg_info(1 pgs e11715:2.78f) v4 ==== 738+0+0 (676955059 0 0) 0xfcc81e0 con 0xcbecec0 -633> 2014-09-18 18:08:37.077177 7f4c9f251700 5 -- op tracker -- , seq: 86352, time: 2014-09-18 18:08:37.077057, event: header_read, op: pg_info(1 pgs e11715:2.78f) -632> 2014-09-18 18:08:37.077185 7f4c9f251700 5 -- op tracker -- , seq: 86352, time: 2014-09-18 18:08:37.077060, event: throttled, op: pg_info(1 pgs e11715:2.78f) -631> 2014-09-18 18:08:37.077203 7f4c9f251700 5 -- op tracker -- , seq: 86352, time: 2014-09-18 18:08:37.077117, event: all_read, op: pg_info(1 pgs e11715:2.78f) -630> 2014-09-18 18:08:37.077207 7f4c9f251700 5 -- op tracker -- , seq: 86352, time: 2014-09-18 18:08:37.077170, event: dispatched, op: pg_info(1 pgs e11715:2.78f) -629> 2014-09-18 18:08:37.077212 7f4c9f251700 5 -- op tracker -- , seq: 86352, time: 2014-09-18 18:08:37.077211, event: waiting_for_osdmap, op: pg_info(1 pgs e11715:2.78f) -628> 2014-09-18 18:08:37.077220 7f4c9f251700 5 -- op tracker -- , seq: 86352, time: 2014-09-18 18:08:37.077219, event: started, op: pg_info(1 pgs e11715:2.78f) -627> 2014-09-18 18:08:37.077241 7f4c9f251700 5 -- op tracker -- , seq: 86352, time: 2014-09-18 18:08:37.077240, event: done, op: pg_info(1 pgs e11715:2.78f) -626> 2014-09-18 18:08:37.077450 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6857/39817 -- pg_info(1 pgs e11715:2.78f) v4 -- ?+0 0x2b93ffe0 con 0x12b3b2e0 -625> 2014-09-18 18:08:37.077475 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11715:2.78f) v4 -- ?+0 0x853fde0 con 0x1395f8c0 -624> 2014-09-18 18:08:37.077490 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11715:2.78f) v4 -- ?+0 0x853eb20 con 0xcbecec0 -623> 2014-09-18 18:08:37.077553 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -622> 2014-09-18 18:08:37.077560 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x119f8200 con 0x111771e0 -621> 2014-09-18 18:08:37.077596 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11715 n=505 ec=188 les/c 11715/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on activate: bft=1,27 from 0//0//-1 -620> 2014-09-18 18:08:37.077630 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).connect error 10.242.42.164:6857/39817, (111) Connection refused -619> 2014-09-18 18:08:37.077644 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11715 n=505 ec=188 les/c 11715/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 1 from 0//0//-1 -618> 2014-09-18 18:08:37.077650 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).fault (111) Connection refused -617> 2014-09-18 18:08:37.077662 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11715 n=505 ec=188 les/c 11715/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target shard 27 from 0//0//-1 -616> 2014-09-18 18:08:37.077672 7f4c775e0700 0 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).fault -615> 2014-09-18 18:08:37.077689 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -614> 2014-09-18 18:08:37.077692 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x119f8800 con 0x111771e0 -613> 2014-09-18 18:08:37.077732 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).connect error 10.242.42.164:6857/39817, (111) Connection refused -612> 2014-09-18 18:08:37.077758 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).fault (111) Connection refused -611> 2014-09-18 18:08:37.077703 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11715 n=505 ec=188 les/c 11715/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active/Activating 0.383216 10 0.000845 -610> 2014-09-18 18:08:37.077811 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.78f( v 6789'22964 (4127'19523,6789'22964] local-les=11715 n=505 ec=188 les/c 11715/11020 11714/11714/11258) [27,42,1]/[42,35] r=0 lpr=11714 pi=5697-11713/157 bft=1,27 crt=0'0 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] enter Started/Primary/Active/WaitLocalBackfillReserved -609> 2014-09-18 18:08:37.077843 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -608> 2014-09-18 18:08:37.077849 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1236e800 con 0x111771e0 -607> 2014-09-18 18:08:37.139140 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3261 ==== pg_notify(2.799(9),2.4d0(10) epoch 11715) v5 ==== 2509+0+0 (753437082 0 0) 0x2b940940 con 0x1395ec60 -606> 2014-09-18 18:08:37.139159 7f4c9f251700 5 -- op tracker -- , seq: 86353, time: 2014-09-18 18:08:37.139063, event: header_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11715) -605> 2014-09-18 18:08:37.139168 7f4c9f251700 5 -- op tracker -- , seq: 86353, time: 2014-09-18 18:08:37.139065, event: throttled, op: pg_notify(2.799(9),2.4d0(10) epoch 11715) -604> 2014-09-18 18:08:37.139172 7f4c9f251700 5 -- op tracker -- , seq: 86353, time: 2014-09-18 18:08:37.139114, event: all_read, op: pg_notify(2.799(9),2.4d0(10) epoch 11715) -603> 2014-09-18 18:08:37.139177 7f4c9f251700 5 -- op tracker -- , seq: 86353, time: 2014-09-18 18:08:37.139155, event: dispatched, op: pg_notify(2.799(9),2.4d0(10) epoch 11715) -602> 2014-09-18 18:08:37.139181 7f4c9f251700 5 -- op tracker -- , seq: 86353, time: 2014-09-18 18:08:37.139179, event: waiting_for_osdmap, op: pg_notify(2.799(9),2.4d0(10) epoch 11715) -601> 2014-09-18 18:08:37.139200 7f4c9f251700 5 -- op tracker -- , seq: 86353, time: 2014-09-18 18:08:37.139198, event: started, op: pg_notify(2.799(9),2.4d0(10) epoch 11715) -600> 2014-09-18 18:08:37.139226 7f4c9f251700 5 -- op tracker -- , seq: 86353, time: 2014-09-18 18:08:37.139224, event: done, op: pg_notify(2.799(9),2.4d0(10) epoch 11715) -599> 2014-09-18 18:08:37.139308 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -598> 2014-09-18 18:08:37.139318 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1259ac00 con 0x111771e0 -597> 2014-09-18 18:08:37.139962 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3262 ==== pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11715) v5 ==== 8086+0+0 (3189623008 0 0) 0x2b9401c0 con 0x1395ec60 -596> 2014-09-18 18:08:37.139974 7f4c9f251700 5 -- op tracker -- , seq: 86354, time: 2014-09-18 18:08:37.139826, event: header_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11715) -595> 2014-09-18 18:08:37.140028 7f4c9f251700 5 -- op tracker -- , seq: 86354, time: 2014-09-18 18:08:37.139828, event: throttled, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11715) -594> 2014-09-18 18:08:37.140032 7f4c9f251700 5 -- op tracker -- , seq: 86354, time: 2014-09-18 18:08:37.139938, event: all_read, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11715) -593> 2014-09-18 18:08:37.140037 7f4c9f251700 5 -- op tracker -- , seq: 86354, time: 2014-09-18 18:08:37.139971, event: dispatched, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11715) -592> 2014-09-18 18:08:37.140042 7f4c9f251700 5 -- op tracker -- , seq: 86354, time: 2014-09-18 18:08:37.140040, event: waiting_for_osdmap, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11715) -591> 2014-09-18 18:08:37.140053 7f4c9f251700 5 -- op tracker -- , seq: 86354, time: 2014-09-18 18:08:37.140051, event: started, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11715) -590> 2014-09-18 18:08:37.140112 7f4c9f251700 5 -- op tracker -- , seq: 86354, time: 2014-09-18 18:08:37.140110, event: done, op: pg_notify(2.4b1(13),2.4c1(15),2.78f(55),2.4c6(15) epoch 11715) -589> 2014-09-18 18:08:37.140130 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3263 ==== pg_query(2.4c0 epoch 11715) v3 ==== 144+0+0 (2046328476 0 0) 0x2b941480 con 0x1395ec60 -588> 2014-09-18 18:08:37.140140 7f4c9f251700 5 -- op tracker -- , seq: 86355, time: 2014-09-18 18:08:37.139965, event: header_read, op: pg_query(2.4c0 epoch 11715) -587> 2014-09-18 18:08:37.140147 7f4c9f251700 5 -- op tracker -- , seq: 86355, time: 2014-09-18 18:08:37.139966, event: throttled, op: pg_query(2.4c0 epoch 11715) -586> 2014-09-18 18:08:37.140269 7f4c9f251700 5 -- op tracker -- , seq: 86355, time: 2014-09-18 18:08:37.139985, event: all_read, op: pg_query(2.4c0 epoch 11715) -585> 2014-09-18 18:08:37.140276 7f4c9f251700 5 -- op tracker -- , seq: 86355, time: 2014-09-18 18:08:37.140138, event: dispatched, op: pg_query(2.4c0 epoch 11715) -584> 2014-09-18 18:08:37.140283 7f4c9f251700 5 -- op tracker -- , seq: 86355, time: 2014-09-18 18:08:37.140281, event: waiting_for_osdmap, op: pg_query(2.4c0 epoch 11715) -583> 2014-09-18 18:08:37.140283 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -582> 2014-09-18 18:08:37.140291 7f4c9f251700 5 -- op tracker -- , seq: 86355, time: 2014-09-18 18:08:37.140289, event: started, op: pg_query(2.4c0 epoch 11715) -581> 2014-09-18 18:08:37.140290 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1259bc00 con 0x111771e0 -580> 2014-09-18 18:08:37.140312 7f4c9f251700 5 -- op tracker -- , seq: 86355, time: 2014-09-18 18:08:37.140311, event: done, op: pg_query(2.4c0 epoch 11715) -579> 2014-09-18 18:08:37.140318 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -578> 2014-09-18 18:08:37.140324 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3264 ==== pg_notify(2.204(64) epoch 11715) v5 ==== 4270+0+0 (1224785509 0 0) 0xddf2b40 con 0x1395ec60 -577> 2014-09-18 18:08:37.140328 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1236c800 con 0x111771e0 -576> 2014-09-18 18:08:37.140331 7f4c9f251700 5 -- op tracker -- , seq: 86356, time: 2014-09-18 18:08:37.140028, event: header_read, op: pg_notify(2.204(64) epoch 11715) -575> 2014-09-18 18:08:37.140342 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.4c0(9) epoch 11715) v5 -- ?+0 0x2b94c580 con 0x1395ec60 -574> 2014-09-18 18:08:37.140348 7f4c9f251700 5 -- op tracker -- , seq: 86356, time: 2014-09-18 18:08:37.140029, event: throttled, op: pg_notify(2.204(64) epoch 11715) -573> 2014-09-18 18:08:37.140354 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -572> 2014-09-18 18:08:37.140356 7f4c9f251700 5 -- op tracker -- , seq: 86356, time: 2014-09-18 18:08:37.140099, event: all_read, op: pg_notify(2.204(64) epoch 11715) -571> 2014-09-18 18:08:37.140357 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1259b400 con 0x111771e0 -570> 2014-09-18 18:08:37.140369 7f4c9f251700 5 -- op tracker -- , seq: 86356, time: 2014-09-18 18:08:37.140329, event: dispatched, op: pg_notify(2.204(64) epoch 11715) -569> 2014-09-18 18:08:37.140374 7f4c9f251700 5 -- op tracker -- , seq: 86356, time: 2014-09-18 18:08:37.140373, event: waiting_for_osdmap, op: pg_notify(2.204(64) epoch 11715) -568> 2014-09-18 18:08:37.140381 7f4c9f251700 5 -- op tracker -- , seq: 86356, time: 2014-09-18 18:08:37.140380, event: started, op: pg_notify(2.204(64) epoch 11715) -567> 2014-09-18 18:08:37.140397 7f4c9f251700 5 -- op tracker -- , seq: 86356, time: 2014-09-18 18:08:37.140396, event: done, op: pg_notify(2.204(64) epoch 11715) -566> 2014-09-18 18:08:37.140433 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -565> 2014-09-18 18:08:37.140440 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1236f400 con 0x111771e0 -564> 2014-09-18 18:08:37.160129 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).connect error 10.242.42.164:6858/39817, (111) Connection refused -563> 2014-09-18 18:08:37.160148 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).fault (111) Connection refused -562> 2014-09-18 18:08:37.160263 7f4c773de700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).connect error 10.242.42.164:6859/39817, (111) Connection refused -561> 2014-09-18 18:08:37.160284 7f4c773de700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).fault (111) Connection refused -560> 2014-09-18 18:08:37.195941 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.13 10.242.42.168:0/10879 697 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:37.062403) v2 ==== 47+0+0 (3218866893 0 0) 0x2b928ee0 con 0x140f7b80 -559> 2014-09-18 18:08:37.195976 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:37.062403) v2 -- ?+0 0x2b9301c0 con 0x140f7b80 -558> 2014-09-18 18:08:37.195990 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.13 10.242.42.168:0/10879 697 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:37.062403) v2 ==== 47+0+0 (3218866893 0 0) 0x24f781e0 con 0x109f9de0 -557> 2014-09-18 18:08:37.196019 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.168:0/10879 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:37.062403) v2 -- ?+0 0xe1a2b40 con 0x109f9de0 -556> 2014-09-18 18:08:37.201548 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3265 ==== pg_notify(2.376(10) epoch 11715) v5 ==== 1296+0+0 (1718467983 0 0) 0x2b94c580 con 0x1395ec60 -555> 2014-09-18 18:08:37.201567 7f4c9f251700 5 -- op tracker -- , seq: 86357, time: 2014-09-18 18:08:37.201420, event: header_read, op: pg_notify(2.376(10) epoch 11715) -554> 2014-09-18 18:08:37.201578 7f4c9f251700 5 -- op tracker -- , seq: 86357, time: 2014-09-18 18:08:37.201421, event: throttled, op: pg_notify(2.376(10) epoch 11715) -553> 2014-09-18 18:08:37.201585 7f4c9f251700 5 -- op tracker -- , seq: 86357, time: 2014-09-18 18:08:37.201447, event: all_read, op: pg_notify(2.376(10) epoch 11715) -552> 2014-09-18 18:08:37.201592 7f4c9f251700 5 -- op tracker -- , seq: 86357, time: 2014-09-18 18:08:37.201563, event: dispatched, op: pg_notify(2.376(10) epoch 11715) -551> 2014-09-18 18:08:37.201600 7f4c9f251700 5 -- op tracker -- , seq: 86357, time: 2014-09-18 18:08:37.201598, event: waiting_for_osdmap, op: pg_notify(2.376(10) epoch 11715) -550> 2014-09-18 18:08:37.201608 7f4c9f251700 5 -- op tracker -- , seq: 86357, time: 2014-09-18 18:08:37.201607, event: started, op: pg_notify(2.376(10) epoch 11715) -549> 2014-09-18 18:08:37.201649 7f4c9f251700 5 -- op tracker -- , seq: 86357, time: 2014-09-18 18:08:37.201647, event: done, op: pg_notify(2.376(10) epoch 11715) -548> 2014-09-18 18:08:37.201676 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -547> 2014-09-18 18:08:37.201689 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1df16600 con 0x111771e0 -546> 2014-09-18 18:08:37.277943 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).connect error 10.242.42.164:6857/39817, (111) Connection refused -545> 2014-09-18 18:08:37.277979 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).fault (111) Connection refused -544> 2014-09-18 18:08:37.326475 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 709 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x11bb8640 con 0x1f103e40 -543> 2014-09-18 18:08:37.326513 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -542> 2014-09-18 18:08:37.326521 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 710 ==== pg_notify(2.4f6(15) epoch 11715) v5 ==== 1543+0+0 (1166490647 0 0) 0xe1ad840 con 0x1f103e40 -541> 2014-09-18 18:08:37.326550 7f4c9f251700 5 -- op tracker -- , seq: 86358, time: 2014-09-18 18:08:37.326469, event: header_read, op: pg_notify(2.4f6(15) epoch 11715) -540> 2014-09-18 18:08:37.326566 7f4c9f251700 5 -- op tracker -- , seq: 86358, time: 2014-09-18 18:08:37.326469, event: throttled, op: pg_notify(2.4f6(15) epoch 11715) -539> 2014-09-18 18:08:37.326571 7f4c9f251700 5 -- op tracker -- , seq: 86358, time: 2014-09-18 18:08:37.326503, event: all_read, op: pg_notify(2.4f6(15) epoch 11715) -538> 2014-09-18 18:08:37.326575 7f4c9f251700 5 -- op tracker -- , seq: 86358, time: 2014-09-18 18:08:37.326527, event: dispatched, op: pg_notify(2.4f6(15) epoch 11715) -537> 2014-09-18 18:08:37.326580 7f4c9f251700 5 -- op tracker -- , seq: 86358, time: 2014-09-18 18:08:37.326579, event: waiting_for_osdmap, op: pg_notify(2.4f6(15) epoch 11715) -536> 2014-09-18 18:08:37.326588 7f4c9f251700 5 -- op tracker -- , seq: 86358, time: 2014-09-18 18:08:37.326586, event: started, op: pg_notify(2.4f6(15) epoch 11715) -535> 2014-09-18 18:08:37.326609 7f4c9f251700 5 -- op tracker -- , seq: 86358, time: 2014-09-18 18:08:37.326608, event: done, op: pg_notify(2.4f6(15) epoch 11715) -534> 2014-09-18 18:08:37.326663 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -533> 2014-09-18 18:08:37.326674 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1236ce00 con 0x111771e0 -532> 2014-09-18 18:08:37.329651 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 711 ==== pg_notify(2.799(10) epoch 11715) v5 ==== 1280+0+0 (810957233 0 0) 0xe1ac1c0 con 0x1f103e40 -531> 2014-09-18 18:08:37.329671 7f4c9f251700 5 -- op tracker -- , seq: 86359, time: 2014-09-18 18:08:37.329562, event: header_read, op: pg_notify(2.799(10) epoch 11715) -530> 2014-09-18 18:08:37.329679 7f4c9f251700 5 -- op tracker -- , seq: 86359, time: 2014-09-18 18:08:37.329563, event: throttled, op: pg_notify(2.799(10) epoch 11715) -529> 2014-09-18 18:08:37.329685 7f4c9f251700 5 -- op tracker -- , seq: 86359, time: 2014-09-18 18:08:37.329616, event: all_read, op: pg_notify(2.799(10) epoch 11715) -528> 2014-09-18 18:08:37.329693 7f4c9f251700 5 -- op tracker -- , seq: 86359, time: 2014-09-18 18:08:37.329666, event: dispatched, op: pg_notify(2.799(10) epoch 11715) -527> 2014-09-18 18:08:37.329700 7f4c9f251700 5 -- op tracker -- , seq: 86359, time: 2014-09-18 18:08:37.329698, event: waiting_for_osdmap, op: pg_notify(2.799(10) epoch 11715) -526> 2014-09-18 18:08:37.329708 7f4c9f251700 5 -- op tracker -- , seq: 86359, time: 2014-09-18 18:08:37.329706, event: started, op: pg_notify(2.799(10) epoch 11715) -525> 2014-09-18 18:08:37.329727 7f4c9f251700 5 -- op tracker -- , seq: 86359, time: 2014-09-18 18:08:37.329725, event: done, op: pg_notify(2.799(10) epoch 11715) -524> 2014-09-18 18:08:37.329777 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -523> 2014-09-18 18:08:37.329786 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1df17a00 con 0x111771e0 -522> 2014-09-18 18:08:37.330438 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.3 10.242.42.164:6823/20623 712 ==== pg_notify(2.4c6(15) epoch 11715) v5 ==== 1511+0+0 (2713662297 0 0) 0xe1aaf00 con 0x1f103e40 -521> 2014-09-18 18:08:37.330457 7f4c9f251700 5 -- op tracker -- , seq: 86360, time: 2014-09-18 18:08:37.330385, event: header_read, op: pg_notify(2.4c6(15) epoch 11715) -520> 2014-09-18 18:08:37.330468 7f4c9f251700 5 -- op tracker -- , seq: 86360, time: 2014-09-18 18:08:37.330386, event: throttled, op: pg_notify(2.4c6(15) epoch 11715) -519> 2014-09-18 18:08:37.330474 7f4c9f251700 5 -- op tracker -- , seq: 86360, time: 2014-09-18 18:08:37.330417, event: all_read, op: pg_notify(2.4c6(15) epoch 11715) -518> 2014-09-18 18:08:37.330480 7f4c9f251700 5 -- op tracker -- , seq: 86360, time: 2014-09-18 18:08:37.330453, event: dispatched, op: pg_notify(2.4c6(15) epoch 11715) -517> 2014-09-18 18:08:37.330487 7f4c9f251700 5 -- op tracker -- , seq: 86360, time: 2014-09-18 18:08:37.330485, event: waiting_for_osdmap, op: pg_notify(2.4c6(15) epoch 11715) -516> 2014-09-18 18:08:37.330495 7f4c9f251700 5 -- op tracker -- , seq: 86360, time: 2014-09-18 18:08:37.330493, event: started, op: pg_notify(2.4c6(15) epoch 11715) -515> 2014-09-18 18:08:37.330514 7f4c9f251700 5 -- op tracker -- , seq: 86360, time: 2014-09-18 18:08:37.330513, event: done, op: pg_notify(2.4c6(15) epoch 11715) -514> 2014-09-18 18:08:37.330531 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -513> 2014-09-18 18:08:37.330548 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1236d800 con 0x111771e0 -512> 2014-09-18 18:08:37.439771 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1738 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xf85b440 con 0x4900000 -511> 2014-09-18 18:08:37.439799 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -510> 2014-09-18 18:08:37.439874 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.20 10.242.42.172:6822/33871 1739 ==== pg_notify(2.e0(79) epoch 11715) v5 ==== 5119+0+0 (208618616 0 0) 0xf1ec580 con 0x4900000 -509> 2014-09-18 18:08:37.439887 7f4c9f251700 5 -- op tracker -- , seq: 86361, time: 2014-09-18 18:08:37.439777, event: header_read, op: pg_notify(2.e0(79) epoch 11715) -508> 2014-09-18 18:08:37.439898 7f4c9f251700 5 -- op tracker -- , seq: 86361, time: 2014-09-18 18:08:37.439777, event: throttled, op: pg_notify(2.e0(79) epoch 11715) -507> 2014-09-18 18:08:37.439907 7f4c9f251700 5 -- op tracker -- , seq: 86361, time: 2014-09-18 18:08:37.439863, event: all_read, op: pg_notify(2.e0(79) epoch 11715) -506> 2014-09-18 18:08:37.439914 7f4c9f251700 5 -- op tracker -- , seq: 86361, time: 2014-09-18 18:08:37.439882, event: dispatched, op: pg_notify(2.e0(79) epoch 11715) -505> 2014-09-18 18:08:37.439922 7f4c9f251700 5 -- op tracker -- , seq: 86361, time: 2014-09-18 18:08:37.439921, event: waiting_for_osdmap, op: pg_notify(2.e0(79) epoch 11715) -504> 2014-09-18 18:08:37.439933 7f4c9f251700 5 -- op tracker -- , seq: 86361, time: 2014-09-18 18:08:37.439932, event: started, op: pg_notify(2.e0(79) epoch 11715) -503> 2014-09-18 18:08:37.439956 7f4c9f251700 5 -- op tracker -- , seq: 86361, time: 2014-09-18 18:08:37.439954, event: done, op: pg_notify(2.e0(79) epoch 11715) -502> 2014-09-18 18:08:37.439987 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -501> 2014-09-18 18:08:37.439999 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1df14c00 con 0x111771e0 -500> 2014-09-18 18:08:37.460146 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.55 10.242.42.172:0/19404 1106 ==== osd_ping(ping e11715 stamp 2014-09-18 23:37:26.564484) v2 ==== 47+0+0 (3573977207 0 0) 0x1368c960 con 0xeddec60 -499> 2014-09-18 18:08:37.460153 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.55 10.242.42.172:0/19404 1106 ==== osd_ping(ping e11715 stamp 2014-09-18 23:37:26.564484) v2 ==== 47+0+0 (3573977207 0 0) 0xf8730e0 con 0x4b7ab00 -498> 2014-09-18 18:08:37.460199 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.172:0/19404 -- osd_ping(ping_reply e11715 stamp 2014-09-18 23:37:26.564484) v2 -- ?+0 0x24f781e0 con 0x4b7ab00 -497> 2014-09-18 18:08:37.460229 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.172:0/19404 -- osd_ping(ping_reply e11715 stamp 2014-09-18 23:37:26.564484) v2 -- ?+0 0x2b928ee0 con 0xeddec60 -496> 2014-09-18 18:08:37.490328 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.35 10.242.42.164:0/17421 166 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:37.489189) v2 ==== 47+0+0 (47936436 0 0) 0xf096b40 con 0x1d5bcd60 -495> 2014-09-18 18:08:37.490337 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.35 10.242.42.164:0/17421 166 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:37.489189) v2 ==== 47+0+0 (47936436 0 0) 0xf0b8760 con 0x142e9700 -494> 2014-09-18 18:08:37.490373 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/17421 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:37.489189) v2 -- ?+0 0x1368c960 con 0x1d5bcd60 -493> 2014-09-18 18:08:37.490403 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/17421 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:37.489189) v2 -- ?+0 0xf8730e0 con 0x142e9700 -492> 2014-09-18 18:08:37.497579 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.46 10.242.42.178:0/4507 119 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:37.496989) v2 ==== 47+0+0 (1230090616 0 0) 0xf4effe0 con 0x10a5c3c0 -491> 2014-09-18 18:08:37.497594 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.46 10.242.42.178:0/4507 119 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:37.496989) v2 ==== 47+0+0 (1230090616 0 0) 0xf2b5660 con 0x110bb5a0 -490> 2014-09-18 18:08:37.497617 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.178:0/4507 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:37.496989) v2 -- ?+0 0xf096b40 con 0x10a5c3c0 -489> 2014-09-18 18:08:37.497758 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.178:0/4507 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:37.496989) v2 -- ?+0 0xf0b8760 con 0x110bb5a0 -488> 2014-09-18 18:08:37.560287 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).connect error 10.242.42.164:6858/39817, (111) Connection refused -487> 2014-09-18 18:08:37.560324 7f4c71984700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6858/39817 pipe(0x14b4e000 sd=70 :0 s=1 pgs=0 cs=0 l=1 c=0x23a779c0).fault (111) Connection refused -486> 2014-09-18 18:08:37.560400 7f4c773de700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).connect error 10.242.42.164:6859/39817, (111) Connection refused -485> 2014-09-18 18:08:37.560434 7f4c773de700 2 -- 10.242.42.178:0/2483 >> 10.242.42.164:6859/39817 pipe(0x14b50100 sd=54 :0 s=1 pgs=0 cs=0 l=1 c=0x23a77b20).fault (111) Connection refused -484> 2014-09-18 18:08:37.562366 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 979 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0x10d19180 con 0x10a878c0 -483> 2014-09-18 18:08:37.562390 7f4c9f251700 3 osd.42 11715 handle_osd_map epochs [11715,11715], i have 11715, src has [1,11715] -482> 2014-09-18 18:08:37.562398 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 980 ==== pg_notify(2.204 epoch 11715) v5 ==== 722+0+0 (3204286541 0 0) 0x24fd2580 con 0x10a878c0 -481> 2014-09-18 18:08:37.562409 7f4c9f251700 5 -- op tracker -- , seq: 86362, time: 2014-09-18 18:08:37.562338, event: header_read, op: pg_notify(2.204 epoch 11715) -480> 2014-09-18 18:08:37.562430 7f4c9f251700 5 -- op tracker -- , seq: 86362, time: 2014-09-18 18:08:37.562338, event: throttled, op: pg_notify(2.204 epoch 11715) -479> 2014-09-18 18:08:37.562436 7f4c9f251700 5 -- op tracker -- , seq: 86362, time: 2014-09-18 18:08:37.562358, event: all_read, op: pg_notify(2.204 epoch 11715) -478> 2014-09-18 18:08:37.562442 7f4c9f251700 5 -- op tracker -- , seq: 86362, time: 2014-09-18 18:08:37.562405, event: dispatched, op: pg_notify(2.204 epoch 11715) -477> 2014-09-18 18:08:37.562448 7f4c9f251700 5 -- op tracker -- , seq: 86362, time: 2014-09-18 18:08:37.562447, event: waiting_for_osdmap, op: pg_notify(2.204 epoch 11715) -476> 2014-09-18 18:08:37.562458 7f4c9f251700 5 -- op tracker -- , seq: 86362, time: 2014-09-18 18:08:37.562456, event: started, op: pg_notify(2.204 epoch 11715) -475> 2014-09-18 18:08:37.562478 7f4c9f251700 5 -- op tracker -- , seq: 86362, time: 2014-09-18 18:08:37.562477, event: done, op: pg_notify(2.204 epoch 11715) -474> 2014-09-18 18:08:37.562516 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.870045 6 0.000376 -473> 2014-09-18 18:08:37.562554 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -472> 2014-09-18 18:08:37.562715 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 bft=0,27 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000162 0 0.000000 -471> 2014-09-18 18:08:37.562734 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 bft=0,27 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -470> 2014-09-18 18:08:37.562750 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 bft=0,27 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000015 0 0.000000 -469> 2014-09-18 18:08:37.562761 7f4c97241700 5 osd.42 pg_epoch: 11715 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 bft=0,27 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -468> 2014-09-18 18:08:37.562935 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -467> 2014-09-18 18:08:37.562942 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1221aa00 con 0x111771e0 -466> 2014-09-18 18:08:37.563148 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -465> 2014-09-18 18:08:37.563156 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1df15e00 con 0x111771e0 -464> 2014-09-18 18:08:37.565827 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.0 10.242.42.164:6801/15878 981 ==== pg_notify(2.517(11),2.4c6(15) epoch 11715) v5 ==== 2834+0+0 (3845042622 0 0) 0x1624f4a0 con 0x10a878c0 -463> 2014-09-18 18:08:37.565845 7f4c9f251700 5 -- op tracker -- , seq: 86363, time: 2014-09-18 18:08:37.565758, event: header_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11715) -462> 2014-09-18 18:08:37.565854 7f4c9f251700 5 -- op tracker -- , seq: 86363, time: 2014-09-18 18:08:37.565759, event: throttled, op: pg_notify(2.517(11),2.4c6(15) epoch 11715) -461> 2014-09-18 18:08:37.565861 7f4c9f251700 5 -- op tracker -- , seq: 86363, time: 2014-09-18 18:08:37.565805, event: all_read, op: pg_notify(2.517(11),2.4c6(15) epoch 11715) -460> 2014-09-18 18:08:37.565867 7f4c9f251700 5 -- op tracker -- , seq: 86363, time: 2014-09-18 18:08:37.565840, event: dispatched, op: pg_notify(2.517(11),2.4c6(15) epoch 11715) -459> 2014-09-18 18:08:37.565873 7f4c9f251700 5 -- op tracker -- , seq: 86363, time: 2014-09-18 18:08:37.565872, event: waiting_for_osdmap, op: pg_notify(2.517(11),2.4c6(15) epoch 11715) -458> 2014-09-18 18:08:37.565881 7f4c9f251700 5 -- op tracker -- , seq: 86363, time: 2014-09-18 18:08:37.565880, event: started, op: pg_notify(2.517(11),2.4c6(15) epoch 11715) -457> 2014-09-18 18:08:37.565930 7f4c9f251700 5 -- op tracker -- , seq: 86363, time: 2014-09-18 18:08:37.565928, event: done, op: pg_notify(2.517(11),2.4c6(15) epoch 11715) -456> 2014-09-18 18:08:37.565940 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -455> 2014-09-18 18:08:37.565951 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1221b400 con 0x111771e0 -454> 2014-09-18 18:08:37.565972 7f4c96a40700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -453> 2014-09-18 18:08:37.565982 7f4c96a40700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11715 {2.376=[42,58,9],2.763=[13,58,24]} v11715) v1 -- ?+0 0x1df17400 con 0x111771e0 -452> 2014-09-18 18:08:37.616233 7f4c9b249700 1 -- 10.242.42.178:6811/2483 <== osd.0 10.242.42.164:0/15878 162 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:37.615717) v2 ==== 47+0+0 (2689429952 0 0) 0xf5765a0 con 0x1d5bc7e0 -451> 2014-09-18 18:08:37.616252 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 <== osd.0 10.242.42.164:0/15878 162 ==== osd_ping(ping e11715 stamp 2014-09-18 18:08:37.615717) v2 ==== 47+0+0 (2689429952 0 0) 0x2b9221e0 con 0x142e8260 -450> 2014-09-18 18:08:37.616263 7f4c9b249700 1 -- 10.242.42.178:6811/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:37.615717) v2 -- ?+0 0xf4effe0 con 0x1d5bc7e0 -449> 2014-09-18 18:08:37.616302 7f4c9ca4c700 1 -- 10.242.42.178:6813/2483 --> 10.242.42.164:0/15878 -- osd_ping(ping_reply e11715 stamp 2014-09-18 18:08:37.615717) v2 -- ?+0 0xf2b5660 con 0x142e8260 -448> 2014-09-18 18:08:37.678179 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).connect error 10.242.42.164:6857/39817, (111) Connection refused -447> 2014-09-18 18:08:37.678214 7f4c775e0700 2 -- 10.242.42.178:6807/2483 >> 10.242.42.164:6857/39817 pipe(0x10b659c0 sd=59 :6807 s=1 pgs=32 cs=2 l=0 c=0x12b3b2e0).fault (111) Connection refused -446> 2014-09-18 18:08:37.685694 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 34 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x15e31680 con 0x111771e0 -445> 2014-09-18 18:08:37.685808 7f4ca0a54700 3 osd.42 11715 handle_osd_map epochs [11716,11716], i have 11715, src has [1,11716] -444> 2014-09-18 18:08:37.688084 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -443> 2014-09-18 18:08:37.688094 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11716 {2.376=[42,58,9],2.763=[13,58,24]} v11716) v1 -- ?+0 0x12218c00 con 0x111771e0 -442> 2014-09-18 18:08:37.688116 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 552 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0xe269600 con 0x1395ef20 -441> 2014-09-18 18:08:37.688528 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -440> 2014-09-18 18:08:37.688544 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 553 ==== pg_log(0.34 epoch 11716 log log((0'0,0'0], crt=0'0) query_epoch 11716) v4 ==== 802+0+0 (2973945909 0 0) 0x10bea700 con 0x1395ef20 -439> 2014-09-18 18:08:37.688564 7f4c9f251700 5 -- op tracker -- , seq: 86364, time: 2014-09-18 18:08:37.688119, event: header_read, op: pg_log(0.34 epoch 11716 log log((0'0,0'0], crt=0'0) query_epoch 11716) -438> 2014-09-18 18:08:37.688579 7f4c9f251700 5 -- op tracker -- , seq: 86364, time: 2014-09-18 18:08:37.688120, event: throttled, op: pg_log(0.34 epoch 11716 log log((0'0,0'0], crt=0'0) query_epoch 11716) -437> 2014-09-18 18:08:37.688588 7f4c9f251700 5 -- op tracker -- , seq: 86364, time: 2014-09-18 18:08:37.688147, event: all_read, op: pg_log(0.34 epoch 11716 log log((0'0,0'0], crt=0'0) query_epoch 11716) -436> 2014-09-18 18:08:37.688596 7f4c9f251700 5 -- op tracker -- , seq: 86364, time: 2014-09-18 18:08:37.688558, event: dispatched, op: pg_log(0.34 epoch 11716 log log((0'0,0'0], crt=0'0) query_epoch 11716) -435> 2014-09-18 18:08:37.688606 7f4c9f251700 5 -- op tracker -- , seq: 86364, time: 2014-09-18 18:08:37.688603, event: waiting_for_osdmap, op: pg_log(0.34 epoch 11716 log log((0'0,0'0], crt=0'0) query_epoch 11716) -434> 2014-09-18 18:08:37.688617 7f4c9f251700 5 -- op tracker -- , seq: 86364, time: 2014-09-18 18:08:37.688613, event: started, op: pg_log(0.34 epoch 11716 log log((0'0,0'0], crt=0'0) query_epoch 11716) -433> 2014-09-18 18:08:37.688636 7f4c9f251700 5 -- op tracker -- , seq: 86364, time: 2014-09-18 18:08:37.688633, event: done, op: pg_log(0.34 epoch 11716 log log((0'0,0'0], crt=0'0) query_epoch 11716) -432> 2014-09-18 18:08:37.688649 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 57 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0xe339840 con 0x1395f340 -431> 2014-09-18 18:08:37.688660 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -430> 2014-09-18 18:08:37.688629 7f4ca0a54700 1 -- 10.242.42.178:6801/2483 <== mon.2 10.242.42.188:6789/0 35 ==== osd_map(11712..11716 src has 1..11716) v3 ==== 6539+0+0 (787435063 0 0) 0x11174ac0 con 0x111771e0 -429> 2014-09-18 18:08:37.688667 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.11 10.242.42.168:6804/17298 58 ==== pg_notify(2.1d8(9) epoch 11716) v5 ==== 1261+0+0 (2298600024 0 0) 0xfa270e0 con 0x1395f340 -428> 2014-09-18 18:08:37.688675 7f4ca0a54700 3 osd.42 11716 handle_osd_map epochs [11712,11716], i have 11716, src has [1,11716] -427> 2014-09-18 18:08:37.688698 7f4c9f251700 5 -- op tracker -- , seq: 86365, time: 2014-09-18 18:08:37.688455, event: header_read, op: pg_notify(2.1d8(9) epoch 11716) -426> 2014-09-18 18:08:37.688707 7f4c9f251700 5 -- op tracker -- , seq: 86365, time: 2014-09-18 18:08:37.688456, event: throttled, op: pg_notify(2.1d8(9) epoch 11716) -425> 2014-09-18 18:08:37.688714 7f4c9f251700 5 -- op tracker -- , seq: 86365, time: 2014-09-18 18:08:37.688494, event: all_read, op: pg_notify(2.1d8(9) epoch 11716) -424> 2014-09-18 18:08:37.688726 7f4c9f251700 5 -- op tracker -- , seq: 86365, time: 2014-09-18 18:08:37.688675, event: dispatched, op: pg_notify(2.1d8(9) epoch 11716) -423> 2014-09-18 18:08:37.688737 7f4c9f251700 5 -- op tracker -- , seq: 86365, time: 2014-09-18 18:08:37.688734, event: waiting_for_osdmap, op: pg_notify(2.1d8(9) epoch 11716) -422> 2014-09-18 18:08:37.688761 7f4c9f251700 5 -- op tracker -- , seq: 86365, time: 2014-09-18 18:08:37.688758, event: started, op: pg_notify(2.1d8(9) epoch 11716) -421> 2014-09-18 18:08:37.688782 7f4c9f251700 5 -- op tracker -- , seq: 86365, time: 2014-09-18 18:08:37.688780, event: done, op: pg_notify(2.1d8(9) epoch 11716) -420> 2014-09-18 18:08:37.688799 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1055 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x11bb9600 con 0x14358260 -419> 2014-09-18 18:08:37.688811 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -418> 2014-09-18 18:08:37.688818 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1056 ==== pg_notify(2.719(17) epoch 11716) v5 ==== 1601+0+0 (446595105 0 0) 0x24f6fde0 con 0x14358260 -417> 2014-09-18 18:08:37.688830 7f4c9f251700 5 -- op tracker -- , seq: 86366, time: 2014-09-18 18:08:37.688725, event: header_read, op: pg_notify(2.719(17) epoch 11716) -416> 2014-09-18 18:08:37.688839 7f4c9f251700 5 -- op tracker -- , seq: 86366, time: 2014-09-18 18:08:37.688726, event: throttled, op: pg_notify(2.719(17) epoch 11716) -415> 2014-09-18 18:08:37.688846 7f4c9f251700 5 -- op tracker -- , seq: 86366, time: 2014-09-18 18:08:37.688796, event: all_read, op: pg_notify(2.719(17) epoch 11716) -414> 2014-09-18 18:08:37.688854 7f4c9f251700 5 -- op tracker -- , seq: 86366, time: 2014-09-18 18:08:37.688826, event: dispatched, op: pg_notify(2.719(17) epoch 11716) -413> 2014-09-18 18:08:37.688862 7f4c9f251700 5 -- op tracker -- , seq: 86366, time: 2014-09-18 18:08:37.688860, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11716) -412> 2014-09-18 18:08:37.688873 7f4c9f251700 5 -- op tracker -- , seq: 86366, time: 2014-09-18 18:08:37.688870, event: started, op: pg_notify(2.719(17) epoch 11716) -411> 2014-09-18 18:08:37.688909 7f4c9f251700 5 -- op tracker -- , seq: 86366, time: 2014-09-18 18:08:37.688906, event: done, op: pg_notify(2.719(17) epoch 11716) -410> 2014-09-18 18:08:37.688923 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2377 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0xe267d40 con 0xdd8c940 -409> 2014-09-18 18:08:37.688936 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -408> 2014-09-18 18:08:37.688994 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2378 ==== pg_notify(2.457(21) epoch 11716) v5 ==== 1875+0+0 (981432886 0 0) 0xf1ed660 con 0xdd8c940 -407> 2014-09-18 18:08:37.689014 7f4c9f251700 5 -- op tracker -- , seq: 86367, time: 2014-09-18 18:08:37.688924, event: header_read, op: pg_notify(2.457(21) epoch 11716) -406> 2014-09-18 18:08:37.689026 7f4c9f251700 5 -- op tracker -- , seq: 86367, time: 2014-09-18 18:08:37.688925, event: throttled, op: pg_notify(2.457(21) epoch 11716) -405> 2014-09-18 18:08:37.689033 7f4c9f251700 5 -- op tracker -- , seq: 86367, time: 2014-09-18 18:08:37.688971, event: all_read, op: pg_notify(2.457(21) epoch 11716) -404> 2014-09-18 18:08:37.689039 7f4c9f251700 5 -- op tracker -- , seq: 86367, time: 2014-09-18 18:08:37.689009, event: dispatched, op: pg_notify(2.457(21) epoch 11716) -403> 2014-09-18 18:08:37.689047 7f4c9f251700 5 -- op tracker -- , seq: 86367, time: 2014-09-18 18:08:37.689044, event: waiting_for_osdmap, op: pg_notify(2.457(21) epoch 11716) -402> 2014-09-18 18:08:37.689056 7f4c9f251700 5 -- op tracker -- , seq: 86367, time: 2014-09-18 18:08:37.689053, event: started, op: pg_notify(2.457(21) epoch 11716) -401> 2014-09-18 18:08:37.689076 7f4c9f251700 5 -- op tracker -- , seq: 86367, time: 2014-09-18 18:08:37.689073, event: done, op: pg_notify(2.457(21) epoch 11716) -400> 2014-09-18 18:08:37.689431 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x12bf0000 con 0x1088aec0 -399> 2014-09-18 18:08:37.689510 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.43f(14) epoch 11716) v5 -- ?+0 0xf91af00 con 0x1088aec0 -398> 2014-09-18 18:08:37.689595 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x12bf06c0 con 0x140f95a0 -397> 2014-09-18 18:08:37.689662 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.30f(11) epoch 11716) v5 -- ?+0 0x2b94cb20 con 0x140f95a0 -396> 2014-09-18 18:08:37.689734 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 800 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x2d7fbf80 con 0xe9a27e0 -395> 2014-09-18 18:08:37.689751 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -394> 2014-09-18 18:08:37.689822 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x12bf3a80 con 0x1395edc0 -393> 2014-09-18 18:08:37.689853 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6829/1204 -- pg_notify(2.6b3(79) epoch 11716) v5 -- ?+0 0xf2b1a20 con 0x1395edc0 -392> 2014-09-18 18:08:37.689919 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0xe2ee640 con 0x1395f1e0 -391> 2014-09-18 18:08:37.689954 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6817/7086 -- pg_notify(2.3c6(24) epoch 11716) v5 -- ?+0 0xf2b1840 con 0x1395f1e0 -390> 2014-09-18 18:08:37.689984 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -389> 2014-09-18 18:08:37.689993 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11716 {2.376=[42,58,9],2.763=[13,58,24]} v11716) v1 -- ?+0 0x1259a200 con 0x111771e0 -388> 2014-09-18 18:08:37.690155 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3707 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x10d166c0 con 0xcbecec0 -387> 2014-09-18 18:08:37.690181 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -386> 2014-09-18 18:08:37.690188 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 49 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x24dbed00 con 0x142f8260 -385> 2014-09-18 18:08:37.690197 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -384> 2014-09-18 18:08:37.690338 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.53 10.242.42.170:6816/2318 801 ==== pg_notify(2.2d8(174) epoch 11716) v5 ==== 10808+0+0 (718826160 0 0) 0x14220b20 con 0xe9a27e0 -383> 2014-09-18 18:08:37.690354 7f4c9f251700 5 -- op tracker -- , seq: 86368, time: 2014-09-18 18:08:37.689819, event: header_read, op: pg_notify(2.2d8(174) epoch 11716) -382> 2014-09-18 18:08:37.690364 7f4c9f251700 5 -- op tracker -- , seq: 86368, time: 2014-09-18 18:08:37.689820, event: throttled, op: pg_notify(2.2d8(174) epoch 11716) -381> 2014-09-18 18:08:37.690371 7f4c9f251700 5 -- op tracker -- , seq: 86368, time: 2014-09-18 18:08:37.690083, event: all_read, op: pg_notify(2.2d8(174) epoch 11716) -380> 2014-09-18 18:08:37.690376 7f4c9f251700 5 -- op tracker -- , seq: 86368, time: 2014-09-18 18:08:37.690349, event: dispatched, op: pg_notify(2.2d8(174) epoch 11716) -379> 2014-09-18 18:08:37.690383 7f4c9f251700 5 -- op tracker -- , seq: 86368, time: 2014-09-18 18:08:37.690381, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11716) -378> 2014-09-18 18:08:37.690392 7f4c9f251700 5 -- op tracker -- , seq: 86368, time: 2014-09-18 18:08:37.690390, event: started, op: pg_notify(2.2d8(174) epoch 11716) -377> 2014-09-18 18:08:37.690411 7f4c9f251700 5 -- op tracker -- , seq: 86368, time: 2014-09-18 18:08:37.690409, event: done, op: pg_notify(2.2d8(174) epoch 11716) -376> 2014-09-18 18:08:37.690435 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3708 ==== pg_notify(2.4f6(12) epoch 11716) v5 ==== 1382+0+0 (3315879697 0 0) 0x853eb20 con 0xcbecec0 -375> 2014-09-18 18:08:37.690448 7f4c9f251700 5 -- op tracker -- , seq: 86369, time: 2014-09-18 18:08:37.690024, event: header_read, op: pg_notify(2.4f6(12) epoch 11716) -374> 2014-09-18 18:08:37.690456 7f4c9f251700 5 -- op tracker -- , seq: 86369, time: 2014-09-18 18:08:37.690025, event: throttled, op: pg_notify(2.4f6(12) epoch 11716) -373> 2014-09-18 18:08:37.690462 7f4c9f251700 5 -- op tracker -- , seq: 86369, time: 2014-09-18 18:08:37.690071, event: all_read, op: pg_notify(2.4f6(12) epoch 11716) -372> 2014-09-18 18:08:37.690467 7f4c9f251700 5 -- op tracker -- , seq: 86369, time: 2014-09-18 18:08:37.690445, event: dispatched, op: pg_notify(2.4f6(12) epoch 11716) -371> 2014-09-18 18:08:37.690475 7f4c9f251700 5 -- op tracker -- , seq: 86369, time: 2014-09-18 18:08:37.690472, event: waiting_for_osdmap, op: pg_notify(2.4f6(12) epoch 11716) -370> 2014-09-18 18:08:37.690482 7f4c9f251700 5 -- op tracker -- , seq: 86369, time: 2014-09-18 18:08:37.690480, event: started, op: pg_notify(2.4f6(12) epoch 11716) -369> 2014-09-18 18:08:37.690517 7f4c9f251700 5 -- op tracker -- , seq: 86369, time: 2014-09-18 18:08:37.690512, event: done, op: pg_notify(2.4f6(12) epoch 11716) -368> 2014-09-18 18:08:37.690536 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 50 ==== pg_notify(2.376(59) epoch 11716) v5 ==== 4019+0+0 (1638015776 0 0) 0x21cea5a0 con 0x142f8260 -367> 2014-09-18 18:08:37.690558 7f4c9f251700 5 -- op tracker -- , seq: 86370, time: 2014-09-18 18:08:37.690160, event: header_read, op: pg_notify(2.376(59) epoch 11716) -366> 2014-09-18 18:08:37.690566 7f4c9f251700 5 -- op tracker -- , seq: 86370, time: 2014-09-18 18:08:37.690161, event: throttled, op: pg_notify(2.376(59) epoch 11716) -365> 2014-09-18 18:08:37.690572 7f4c9f251700 5 -- op tracker -- , seq: 86370, time: 2014-09-18 18:08:37.690256, event: all_read, op: pg_notify(2.376(59) epoch 11716) -364> 2014-09-18 18:08:37.690578 7f4c9f251700 5 -- op tracker -- , seq: 86370, time: 2014-09-18 18:08:37.690555, event: dispatched, op: pg_notify(2.376(59) epoch 11716) -363> 2014-09-18 18:08:37.690585 7f4c9f251700 5 -- op tracker -- , seq: 86370, time: 2014-09-18 18:08:37.690584, event: waiting_for_osdmap, op: pg_notify(2.376(59) epoch 11716) -362> 2014-09-18 18:08:37.690593 7f4c9f251700 5 -- op tracker -- , seq: 86370, time: 2014-09-18 18:08:37.690591, event: started, op: pg_notify(2.376(59) epoch 11716) -361> 2014-09-18 18:08:37.690608 7f4c9f251700 5 -- op tracker -- , seq: 86370, time: 2014-09-18 18:08:37.690606, event: done, op: pg_notify(2.376(59) epoch 11716) -360> 2014-09-18 18:08:37.690664 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0xe2ee880 con 0x1395ef20 -359> 2014-09-18 18:08:37.690672 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1210 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x2db47840 con 0x1088b440 -358> 2014-09-18 18:08:37.690690 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -357> 2014-09-18 18:08:37.690691 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_notify(0.34(23) epoch 11716) v5 -- ?+0 0xf2b0ee0 con 0x1395ef20 -356> 2014-09-18 18:08:37.690738 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -355> 2014-09-18 18:08:37.690750 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11716 {2.376=[42,58,9],2.763=[13,58,24]} v11716) v1 -- ?+0 0x1259a800 con 0x111771e0 -354> 2014-09-18 18:08:37.690890 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 bft=2,27 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.845451 4 0.000298 -353> 2014-09-18 18:08:37.690940 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 bft=2,27 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.998355 0 0.000000 -352> 2014-09-18 18:08:37.690965 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.22 10.242.42.172:6815/37668 1211 ==== pg_notify(2.2d8(174) epoch 11716) v5 ==== 10808+0+0 (718826160 0 0) 0xf8125a0 con 0x1088b440 -351> 2014-09-18 18:08:37.690982 7f4c9f251700 5 -- op tracker -- , seq: 86371, time: 2014-09-18 18:08:37.690793, event: header_read, op: pg_notify(2.2d8(174) epoch 11716) -350> 2014-09-18 18:08:37.690997 7f4c9f251700 5 -- op tracker -- , seq: 86371, time: 2014-09-18 18:08:37.690793, event: throttled, op: pg_notify(2.2d8(174) epoch 11716) -349> 2014-09-18 18:08:37.691004 7f4c9f251700 5 -- op tracker -- , seq: 86371, time: 2014-09-18 18:08:37.690948, event: all_read, op: pg_notify(2.2d8(174) epoch 11716) -348> 2014-09-18 18:08:37.691018 7f4c9f251700 5 -- op tracker -- , seq: 86371, time: 2014-09-18 18:08:37.690978, event: dispatched, op: pg_notify(2.2d8(174) epoch 11716) -347> 2014-09-18 18:08:37.691025 7f4c9f251700 5 -- op tracker -- , seq: 86371, time: 2014-09-18 18:08:37.691023, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11716) -346> 2014-09-18 18:08:37.691034 7f4c9f251700 5 -- op tracker -- , seq: 86371, time: 2014-09-18 18:08:37.691032, event: started, op: pg_notify(2.2d8(174) epoch 11716) -345> 2014-09-18 18:08:37.690962 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11505 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 bft=2,27 crt=6745'27699 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -344> 2014-09-18 18:08:37.691056 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 bft=0,27 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.128295 3 0.000401 -343> 2014-09-18 18:08:37.691210 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 bft=0,27 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 0.998879 0 0.000000 -342> 2014-09-18 18:08:37.691225 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11491 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 bft=0,27 crt=6779'26713 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Active -341> 2014-09-18 18:08:37.691722 7f4c97241700 0 log_channel(default) log [INF] : 2.2d8 restarting backfill on osd.2 from (0'0,0'0] MAX to 6825'27712 -340> 2014-09-18 18:08:37.691823 7f4c96a40700 0 log_channel(default) log [INF] : 2.204 restarting backfill on osd.0 from (0'0,0'0] MAX to 6810'26722 -339> 2014-09-18 18:08:37.693054 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x1533dcc0 con 0x10a878c0 -338> 2014-09-18 18:08:37.693074 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6801/15878 -- pg_log(2.204 epoch 11716 log log((4155'23721,6810'26722], crt=6779'26713) query_epoch 11716) v4 -- ?+0 0x10f71500 con 0x10a878c0 -337> 2014-09-18 18:08:37.693514 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6817/19139 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x2db45b00 con 0x10f60520 -336> 2014-09-18 18:08:37.693532 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6817/19139 -- pg_log(2.2d8 epoch 11716 log log((4157'24703,6825'27712], crt=6745'27699) query_epoch 11716) v4 -- ?+0 0x22ed9500 con 0x10f60520 -335> 2014-09-18 18:08:37.693864 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.2d8( v 6825'27712 (4157'24703,6825'27712] local-les=11716 n=502 ec=188 les/c 11505/9946 11715/11715/11491) [2,27,42]/[42] r=0 lpr=11715 pi=3414-11714/174 bft=2,27 crt=6745'27699 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -334> 2014-09-18 18:08:37.693776 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.204( v 6810'26722 (4155'23721,6810'26722] local-les=11716 n=533 ec=188 les/c 11491/10727 11715/11715/9945) [42,27,0]/[42,9] r=0 lpr=11715 pi=10726-11714/10 bft=0,27 crt=6779'26713 lcod 0'0 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -333> 2014-09-18 18:08:37.696871 7f4c9f251700 5 -- op tracker -- , seq: 86371, time: 2014-09-18 18:08:37.696864, event: done, op: pg_notify(2.2d8(174) epoch 11716) -332> 2014-09-18 18:08:37.696963 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] exit Started/Primary/Peering/WaitActingChange 0.851797 1 0.000328 -331> 2014-09-18 18:08:37.696981 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] exit Started/Primary 0.851823 0 0.000000 -330> 2014-09-18 18:08:37.696985 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1278 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0xeee8900 con 0xe8db760 -329> 2014-09-18 18:08:37.696990 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] exit Started 1.010512 0 0.000000 -328> 2014-09-18 18:08:37.697006 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -327> 2014-09-18 18:08:37.697013 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 309 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x12bf06c0 con 0x140f95a0 -326> 2014-09-18 18:08:37.697025 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -325> 2014-09-18 18:08:37.697012 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11715/11715) [42,2,58] r=0 lpr=11715 pi=6625-11714/56 crt=0'0 mlcod 0'0 inactive] enter Reset -324> 2014-09-18 18:08:37.697032 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.10 10.242.42.168:6845/1010695 504 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0xcc03cc0 con 0x12b53020 -323> 2014-09-18 18:08:37.697040 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -322> 2014-09-18 18:08:37.697043 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.56 10.242.42.172:6837/36458 1057 ==== pg_info(1 pgs e11716:2.b5(24)) v4 ==== 2030+0+0 (493810373 0 0) 0xe1aab40 con 0x14358260 -321> 2014-09-18 18:08:37.697055 7f4c9f251700 5 -- op tracker -- , seq: 86372, time: 2014-09-18 18:08:37.694404, event: header_read, op: pg_info(1 pgs e11716:2.b5(24)) -320> 2014-09-18 18:08:37.697063 7f4c9f251700 5 -- op tracker -- , seq: 86372, time: 2014-09-18 18:08:37.694405, event: throttled, op: pg_info(1 pgs e11716:2.b5(24)) -319> 2014-09-18 18:08:37.697070 7f4c9f251700 5 -- op tracker -- , seq: 86372, time: 2014-09-18 18:08:37.694456, event: all_read, op: pg_info(1 pgs e11716:2.b5(24)) -318> 2014-09-18 18:08:37.697076 7f4c9f251700 5 -- op tracker -- , seq: 86372, time: 2014-09-18 18:08:37.697052, event: dispatched, op: pg_info(1 pgs e11716:2.b5(24)) -317> 2014-09-18 18:08:37.697082 7f4c9f251700 5 -- op tracker -- , seq: 86372, time: 2014-09-18 18:08:37.697080, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.b5(24)) -316> 2014-09-18 18:08:37.697090 7f4c9f251700 5 -- op tracker -- , seq: 86372, time: 2014-09-18 18:08:37.697088, event: started, op: pg_info(1 pgs e11716:2.b5(24)) -315> 2014-09-18 18:08:37.697106 7f4c9f251700 5 -- op tracker -- , seq: 86372, time: 2014-09-18 18:08:37.697104, event: done, op: pg_info(1 pgs e11716:2.b5(24)) -314> 2014-09-18 18:08:37.697123 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3709 ==== pg_info(1 pgs e11716:2.654(11)) v4 ==== 1327+0+0 (2485865603 0 0) 0x1213a940 con 0xcbecec0 -313> 2014-09-18 18:08:37.697134 7f4c9f251700 5 -- op tracker -- , seq: 86373, time: 2014-09-18 18:08:37.696577, event: header_read, op: pg_info(1 pgs e11716:2.654(11)) -312> 2014-09-18 18:08:37.697142 7f4c9f251700 5 -- op tracker -- , seq: 86373, time: 2014-09-18 18:08:37.696578, event: throttled, op: pg_info(1 pgs e11716:2.654(11)) -311> 2014-09-18 18:08:37.697150 7f4c9f251700 5 -- op tracker -- , seq: 86373, time: 2014-09-18 18:08:37.696613, event: all_read, op: pg_info(1 pgs e11716:2.654(11)) -310> 2014-09-18 18:08:37.697159 7f4c9f251700 5 -- op tracker -- , seq: 86373, time: 2014-09-18 18:08:37.697132, event: dispatched, op: pg_info(1 pgs e11716:2.654(11)) -309> 2014-09-18 18:08:37.697170 7f4c9f251700 5 -- op tracker -- , seq: 86373, time: 2014-09-18 18:08:37.697167, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.654(11)) -308> 2014-09-18 18:08:37.697183 7f4c9f251700 5 -- op tracker -- , seq: 86373, time: 2014-09-18 18:08:37.697180, event: started, op: pg_info(1 pgs e11716:2.654(11)) -307> 2014-09-18 18:08:37.697204 7f4c9f251700 5 -- op tracker -- , seq: 86373, time: 2014-09-18 18:08:37.697201, event: done, op: pg_info(1 pgs e11716:2.654(11)) -306> 2014-09-18 18:08:37.697221 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.45 10.242.42.178:6817/7086 3 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0xe2ee640 con 0x1395f1e0 -305> 2014-09-18 18:08:37.697237 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -304> 2014-09-18 18:08:37.697245 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2379 ==== pg_notify(2.2c7(139) epoch 11716) v5 ==== 8651+0+0 (1563004678 0 0) 0xf1ed0c0 con 0xdd8c940 -303> 2014-09-18 18:08:37.697260 7f4c9f251700 5 -- op tracker -- , seq: 86374, time: 2014-09-18 18:08:37.695487, event: header_read, op: pg_notify(2.2c7(139) epoch 11716) -302> 2014-09-18 18:08:37.697269 7f4c9f251700 5 -- op tracker -- , seq: 86374, time: 2014-09-18 18:08:37.695488, event: throttled, op: pg_notify(2.2c7(139) epoch 11716) -301> 2014-09-18 18:08:37.697274 7f4c9f251700 5 -- op tracker -- , seq: 86374, time: 2014-09-18 18:08:37.695677, event: all_read, op: pg_notify(2.2c7(139) epoch 11716) -300> 2014-09-18 18:08:37.697280 7f4c9f251700 5 -- op tracker -- , seq: 86374, time: 2014-09-18 18:08:37.697256, event: dispatched, op: pg_notify(2.2c7(139) epoch 11716) -299> 2014-09-18 18:08:37.697287 7f4c9f251700 5 -- op tracker -- , seq: 86374, time: 2014-09-18 18:08:37.697285, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11716) -298> 2014-09-18 18:08:37.697296 7f4c9f251700 5 -- op tracker -- , seq: 86374, time: 2014-09-18 18:08:37.697293, event: started, op: pg_notify(2.2c7(139) epoch 11716) -297> 2014-09-18 18:08:37.697318 7f4c9f251700 5 -- op tracker -- , seq: 86374, time: 2014-09-18 18:08:37.697316, event: done, op: pg_notify(2.2c7(139) epoch 11716) -296> 2014-09-18 18:08:37.697308 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11716/11716) [42,2,58]/[13,58,24] r=-1 lpr=11716 pi=6625-11715/57 crt=0'0 remapped NOTIFY] exit Reset 0.000296 1 0.000105 -295> 2014-09-18 18:08:37.697334 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11716/11716) [42,2,58]/[13,58,24] r=-1 lpr=11716 pi=6625-11715/57 crt=0'0 remapped NOTIFY] enter Started -294> 2014-09-18 18:08:37.697342 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2610 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x14408b40 con 0x4900b00 -293> 2014-09-18 18:08:37.697343 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11716/11716) [42,2,58]/[13,58,24] r=-1 lpr=11716 pi=6625-11715/57 crt=0'0 remapped NOTIFY] enter Start -292> 2014-09-18 18:08:37.697364 7f4c96a40700 1 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11716/11716) [42,2,58]/[13,58,24] r=-1 lpr=11716 pi=6625-11715/57 crt=0'0 remapped NOTIFY] state: transitioning to Stray -291> 2014-09-18 18:08:37.697356 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -290> 2014-09-18 18:08:37.697399 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2025 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x10e22000 con 0xe2d9a20 -289> 2014-09-18 18:08:37.697413 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -288> 2014-09-18 18:08:37.697424 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.237(131) epoch 11716) v5 -- ?+0 0x19f22d20 con 0x1088aec0 -287> 2014-09-18 18:08:37.697376 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11716/11716) [42,2,58]/[13,58,24] r=-1 lpr=11716 pi=6625-11715/57 crt=0'0 remapped NOTIFY] exit Start 0.000033 0 0.000000 -286> 2014-09-18 18:08:37.697450 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0xe2edb00 con 0x1395f8c0 -285> 2014-09-18 18:08:37.697448 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.763( empty local-les=0 n=0 ec=188 les/c 11403/11403 11715/11716/11716) [42,2,58]/[13,58,24] r=-1 lpr=11716 pi=6625-11715/57 crt=0'0 remapped NOTIFY] enter Started/Stray -284> 2014-09-18 18:08:37.697476 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11716:2.2d8(174)) v4 -- ?+0 0x19f24ee0 con 0x1395f8c0 -283> 2014-09-18 18:08:37.697426 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.63 10.242.42.178:6803/2487 1279 ==== pg_notify(2.17e(17) epoch 11716) v5 ==== 1593+0+0 (473903931 0 0) 0xf1e8ee0 con 0xe8db760 -282> 2014-09-18 18:08:37.697542 7f4c9f251700 5 -- op tracker -- , seq: 86375, time: 2014-09-18 18:08:37.692246, event: header_read, op: pg_notify(2.17e(17) epoch 11716) -281> 2014-09-18 18:08:37.697553 7f4c9f251700 5 -- op tracker -- , seq: 86375, time: 2014-09-18 18:08:37.692246, event: throttled, op: pg_notify(2.17e(17) epoch 11716) -280> 2014-09-18 18:08:37.697563 7f4c9f251700 5 -- op tracker -- , seq: 86375, time: 2014-09-18 18:08:37.692293, event: all_read, op: pg_notify(2.17e(17) epoch 11716) -279> 2014-09-18 18:08:37.697573 7f4c9f251700 5 -- op tracker -- , seq: 86375, time: 2014-09-18 18:08:37.697536, event: dispatched, op: pg_notify(2.17e(17) epoch 11716) -278> 2014-09-18 18:08:37.697583 7f4c9f251700 5 -- op tracker -- , seq: 86375, time: 2014-09-18 18:08:37.697580, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11716) -277> 2014-09-18 18:08:37.697594 7f4c9f251700 5 -- op tracker -- , seq: 86375, time: 2014-09-18 18:08:37.697592, event: started, op: pg_notify(2.17e(17) epoch 11716) -276> 2014-09-18 18:08:37.697625 7f4c9f251700 5 -- op tracker -- , seq: 86375, time: 2014-09-18 18:08:37.697623, event: done, op: pg_notify(2.17e(17) epoch 11716) -275> 2014-09-18 18:08:37.697642 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 310 ==== pg_info(1 pgs e11716:2.30f(11)) v4 ==== 1355+0+0 (1661189465 0 0) 0x2b94cb20 con 0x140f95a0 -274> 2014-09-18 18:08:37.697656 7f4c9f251700 5 -- op tracker -- , seq: 86376, time: 2014-09-18 18:08:37.695143, event: header_read, op: pg_info(1 pgs e11716:2.30f(11)) -273> 2014-09-18 18:08:37.697666 7f4c9f251700 5 -- op tracker -- , seq: 86376, time: 2014-09-18 18:08:37.695144, event: throttled, op: pg_info(1 pgs e11716:2.30f(11)) -272> 2014-09-18 18:08:37.697675 7f4c9f251700 5 -- op tracker -- , seq: 86376, time: 2014-09-18 18:08:37.695183, event: all_read, op: pg_info(1 pgs e11716:2.30f(11)) -271> 2014-09-18 18:08:37.697685 7f4c9f251700 5 -- op tracker -- , seq: 86376, time: 2014-09-18 18:08:37.697651, event: dispatched, op: pg_info(1 pgs e11716:2.30f(11)) -270> 2014-09-18 18:08:37.697695 7f4c9f251700 5 -- op tracker -- , seq: 86376, time: 2014-09-18 18:08:37.697693, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.30f(11)) -269> 2014-09-18 18:08:37.697707 7f4c9f251700 5 -- op tracker -- , seq: 86376, time: 2014-09-18 18:08:37.697704, event: started, op: pg_info(1 pgs e11716:2.30f(11)) -268> 2014-09-18 18:08:37.697726 7f4c9f251700 5 -- op tracker -- , seq: 86376, time: 2014-09-18 18:08:37.697724, event: done, op: pg_info(1 pgs e11716:2.30f(11)) -267> 2014-09-18 18:08:37.697740 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.10 10.242.42.168:6845/1010695 505 ==== pg_info(1 pgs e11716:2.250(7)) v4 ==== 1103+0+0 (2900770596 0 0) 0xe958d00 con 0x12b53020 -266> 2014-09-18 18:08:37.697752 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -265> 2014-09-18 18:08:37.697763 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_pgtemp(e11716 {2.376=[42,58,9]} v11716) v1 -- ?+0 0xdbfa600 con 0x111771e0 -264> 2014-09-18 18:08:37.697752 7f4c9f251700 5 -- op tracker -- , seq: 86377, time: 2014-09-18 18:08:37.697021, event: header_read, op: pg_info(1 pgs e11716:2.250(7)) -263> 2014-09-18 18:08:37.697777 7f4c9f251700 5 -- op tracker -- , seq: 86377, time: 2014-09-18 18:08:37.697022, event: throttled, op: pg_info(1 pgs e11716:2.250(7)) -262> 2014-09-18 18:08:37.697787 7f4c9f251700 5 -- op tracker -- , seq: 86377, time: 2014-09-18 18:08:37.697056, event: all_read, op: pg_info(1 pgs e11716:2.250(7)) -261> 2014-09-18 18:08:37.697797 7f4c9f251700 5 -- op tracker -- , seq: 86377, time: 2014-09-18 18:08:37.697748, event: dispatched, op: pg_info(1 pgs e11716:2.250(7)) -260> 2014-09-18 18:08:37.697807 7f4c9f251700 5 -- op tracker -- , seq: 86377, time: 2014-09-18 18:08:37.697805, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.250(7)) -259> 2014-09-18 18:08:37.697819 7f4c9f251700 5 -- op tracker -- , seq: 86377, time: 2014-09-18 18:08:37.697816, event: started, op: pg_info(1 pgs e11716:2.250(7)) -258> 2014-09-18 18:08:37.697897 7f4c9f251700 5 -- op tracker -- , seq: 86377, time: 2014-09-18 18:08:37.697894, event: done, op: pg_info(1 pgs e11716:2.250(7)) -257> 2014-09-18 18:08:37.698050 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3710 ==== pg_notify(2.3b6(10) epoch 11716) v5 ==== 1260+0+0 (1098065193 0 0) 0x1213a760 con 0xcbecec0 -256> 2014-09-18 18:08:37.698065 7f4c9f251700 5 -- op tracker -- , seq: 86378, time: 2014-09-18 18:08:37.697078, event: header_read, op: pg_notify(2.3b6(10) epoch 11716) -255> 2014-09-18 18:08:37.698076 7f4c9f251700 5 -- op tracker -- , seq: 86378, time: 2014-09-18 18:08:37.697079, event: throttled, op: pg_notify(2.3b6(10) epoch 11716) -254> 2014-09-18 18:08:37.698086 7f4c9f251700 5 -- op tracker -- , seq: 86378, time: 2014-09-18 18:08:37.697111, event: all_read, op: pg_notify(2.3b6(10) epoch 11716) -253> 2014-09-18 18:08:37.698096 7f4c9f251700 5 -- op tracker -- , seq: 86378, time: 2014-09-18 18:08:37.698061, event: dispatched, op: pg_notify(2.3b6(10) epoch 11716) -252> 2014-09-18 18:08:37.698106 7f4c9f251700 5 -- op tracker -- , seq: 86378, time: 2014-09-18 18:08:37.698103, event: waiting_for_osdmap, op: pg_notify(2.3b6(10) epoch 11716) -251> 2014-09-18 18:08:37.698118 7f4c9f251700 5 -- op tracker -- , seq: 86378, time: 2014-09-18 18:08:37.698115, event: started, op: pg_notify(2.3b6(10) epoch 11716) -250> 2014-09-18 18:08:37.698138 7f4c9f251700 5 -- op tracker -- , seq: 86378, time: 2014-09-18 18:08:37.698136, event: done, op: pg_notify(2.3b6(10) epoch 11716) -249> 2014-09-18 18:08:37.698018 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] exit Started/Primary/Peering/WaitActingChange 0.944532 2 0.000287 -248> 2014-09-18 18:08:37.698148 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] exit Started/Primary 0.944669 0 0.000000 -247> 2014-09-18 18:08:37.698163 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] exit Started 1.003820 0 0.000000 -246> 2014-09-18 18:08:37.698151 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3322 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x10d17b00 con 0x4900580 -245> 2014-09-18 18:08:37.698176 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11715/9945) [42,24,58] r=0 lpr=11715 pi=11490-11714/1 crt=6764'25987 lcod 0'0 mlcod 0'0 inactive] enter Reset -244> 2014-09-18 18:08:37.698184 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -243> 2014-09-18 18:08:37.698184 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.5a2(14),2.756(28),2.763(57) epoch 11716) v5 -- ?+0 0x2b9410c0 con 0x140f95a0 -242> 2014-09-18 18:08:37.698197 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.39 10.242.42.172:6816/2660 2611 ==== pg_notify(2.71c(12) epoch 11716) v5 ==== 1424+0+0 (4112249701 0 0) 0xf05bfe0 con 0x4900b00 -241> 2014-09-18 18:08:37.698211 7f4c9f251700 5 -- op tracker -- , seq: 86379, time: 2014-09-18 18:08:37.696189, event: header_read, op: pg_notify(2.71c(12) epoch 11716) -240> 2014-09-18 18:08:37.698211 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x11172000 con 0x1395ec60 -239> 2014-09-18 18:08:37.698214 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped] exit Reset 0.000038 1 0.000193 -238> 2014-09-18 18:08:37.698225 7f4c9f251700 5 -- op tracker -- , seq: 86379, time: 2014-09-18 18:08:37.696190, event: throttled, op: pg_notify(2.71c(12) epoch 11716) -237> 2014-09-18 18:08:37.698233 7f4c9f251700 5 -- op tracker -- , seq: 86379, time: 2014-09-18 18:08:37.696247, event: all_read, op: pg_notify(2.71c(12) epoch 11716) -236> 2014-09-18 18:08:37.698241 7f4c9f251700 5 -- op tracker -- , seq: 86379, time: 2014-09-18 18:08:37.698206, event: dispatched, op: pg_notify(2.71c(12) epoch 11716) -235> 2014-09-18 18:08:37.698251 7f4c9f251700 5 -- op tracker -- , seq: 86379, time: 2014-09-18 18:08:37.698248, event: waiting_for_osdmap, op: pg_notify(2.71c(12) epoch 11716) -234> 2014-09-18 18:08:37.698273 7f4c9f251700 5 -- op tracker -- , seq: 86379, time: 2014-09-18 18:08:37.698270, event: started, op: pg_notify(2.71c(12) epoch 11716) -233> 2014-09-18 18:08:37.698293 7f4c9f251700 5 -- op tracker -- , seq: 86379, time: 2014-09-18 18:08:37.698291, event: done, op: pg_notify(2.71c(12) epoch 11716) -232> 2014-09-18 18:08:37.698308 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.55 10.242.42.172:6829/19404 2026 ==== pg_notify(2.26d(9) epoch 11716) v5 ==== 1257+0+0 (1772439243 0 0) 0x1310c960 con 0xe2d9a20 -231> 2014-09-18 18:08:37.698321 7f4c9f251700 5 -- op tracker -- , seq: 86380, time: 2014-09-18 18:08:37.693661, event: header_read, op: pg_notify(2.26d(9) epoch 11716) -230> 2014-09-18 18:08:37.698334 7f4c9f251700 5 -- op tracker -- , seq: 86380, time: 2014-09-18 18:08:37.693662, event: throttled, op: pg_notify(2.26d(9) epoch 11716) -229> 2014-09-18 18:08:37.698345 7f4c9f251700 5 -- op tracker -- , seq: 86380, time: 2014-09-18 18:08:37.693724, event: all_read, op: pg_notify(2.26d(9) epoch 11716) -228> 2014-09-18 18:08:37.698355 7f4c9f251700 5 -- op tracker -- , seq: 86380, time: 2014-09-18 18:08:37.698317, event: dispatched, op: pg_notify(2.26d(9) epoch 11716) -227> 2014-09-18 18:08:37.698365 7f4c9f251700 5 -- op tracker -- , seq: 86380, time: 2014-09-18 18:08:37.698362, event: waiting_for_osdmap, op: pg_notify(2.26d(9) epoch 11716) -226> 2014-09-18 18:08:37.698377 7f4c9f251700 5 -- op tracker -- , seq: 86380, time: 2014-09-18 18:08:37.698374, event: started, op: pg_notify(2.26d(9) epoch 11716) -225> 2014-09-18 18:08:37.698356 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11716:2.204(10)) v4 -- ?+0 0x2b93e1e0 con 0x1395ec60 -224> 2014-09-18 18:08:37.698397 7f4c9f251700 5 -- op tracker -- , seq: 86380, time: 2014-09-18 18:08:37.698395, event: done, op: pg_notify(2.26d(9) epoch 11716) -223> 2014-09-18 18:08:37.698229 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped] enter Started -222> 2014-09-18 18:08:37.698410 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 311 ==== pg_info(1 pgs e11716:2.756(17)) v4 ==== 1641+0+0 (2027987582 0 0) 0x128cc940 con 0x140f95a0 -221> 2014-09-18 18:08:37.698415 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped] enter Start -220> 2014-09-18 18:08:37.698425 7f4c9f251700 5 -- op tracker -- , seq: 86381, time: 2014-09-18 18:08:37.695458, event: header_read, op: pg_info(1 pgs e11716:2.756(17)) -219> 2014-09-18 18:08:37.698427 7f4c97241700 1 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped] state: transitioning to Primary -218> 2014-09-18 18:08:37.698436 7f4c9f251700 5 -- op tracker -- , seq: 86381, time: 2014-09-18 18:08:37.695458, event: throttled, op: pg_info(1 pgs e11716:2.756(17)) -217> 2014-09-18 18:08:37.698440 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped] exit Start 0.000025 0 0.000000 -216> 2014-09-18 18:08:37.698446 7f4c9f251700 5 -- op tracker -- , seq: 86381, time: 2014-09-18 18:08:37.695518, event: all_read, op: pg_info(1 pgs e11716:2.756(17)) -215> 2014-09-18 18:08:37.698448 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.174:6801/21548 -- pg_info(1 pgs e11716:2.204(10)) v4 -- ?+0 0xf4872c0 con 0x1395f8c0 -214> 2014-09-18 18:08:37.698454 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary -213> 2014-09-18 18:08:37.698459 7f4c9f251700 5 -- op tracker -- , seq: 86381, time: 2014-09-18 18:08:37.698418, event: dispatched, op: pg_info(1 pgs e11716:2.756(17)) -212> 2014-09-18 18:08:37.698465 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped] enter Started/Primary/Peering -211> 2014-09-18 18:08:37.698471 7f4c9f251700 5 -- op tracker -- , seq: 86381, time: 2014-09-18 18:08:37.698467, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.756(17)) -210> 2014-09-18 18:08:37.698483 7f4c9f251700 5 -- op tracker -- , seq: 86381, time: 2014-09-18 18:08:37.698478, event: started, op: pg_info(1 pgs e11716:2.756(17)) -209> 2014-09-18 18:08:37.698475 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -208> 2014-09-18 18:08:37.698501 7f4c9f251700 5 -- op tracker -- , seq: 86381, time: 2014-09-18 18:08:37.698499, event: done, op: pg_info(1 pgs e11716:2.756(17)) -207> 2014-09-18 18:08:37.698515 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3323 ==== pg_notify(2.4f6(184) epoch 11716) v5 ==== 11296+0+0 (2594191586 0 0) 0xf72ed20 con 0x4900580 -206> 2014-09-18 18:08:37.698527 7f4c9f251700 5 -- op tracker -- , seq: 86382, time: 2014-09-18 18:08:37.697756, event: header_read, op: pg_notify(2.4f6(184) epoch 11716) -205> 2014-09-18 18:08:37.698549 7f4c9f251700 5 -- op tracker -- , seq: 86382, time: 2014-09-18 18:08:37.697757, event: throttled, op: pg_notify(2.4f6(184) epoch 11716) -204> 2014-09-18 18:08:37.698559 7f4c9f251700 5 -- op tracker -- , seq: 86382, time: 2014-09-18 18:08:37.698013, event: all_read, op: pg_notify(2.4f6(184) epoch 11716) -203> 2014-09-18 18:08:37.698569 7f4c9f251700 5 -- op tracker -- , seq: 86382, time: 2014-09-18 18:08:37.698523, event: dispatched, op: pg_notify(2.4f6(184) epoch 11716) -202> 2014-09-18 18:08:37.698579 7f4c9f251700 5 -- op tracker -- , seq: 86382, time: 2014-09-18 18:08:37.698576, event: waiting_for_osdmap, op: pg_notify(2.4f6(184) epoch 11716) -201> 2014-09-18 18:08:37.698591 7f4c9f251700 5 -- op tracker -- , seq: 86382, time: 2014-09-18 18:08:37.698588, event: started, op: pg_notify(2.4f6(184) epoch 11716) -200> 2014-09-18 18:08:37.698772 7f4c9f251700 5 -- op tracker -- , seq: 86382, time: 2014-09-18 18:08:37.698769, event: done, op: pg_notify(2.4f6(184) epoch 11716) -199> 2014-09-18 18:08:37.699219 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.45 10.242.42.178:6817/7086 4 ==== pg_log(2.3c6 epoch 11716 log log((4155'24411,6840'27412], crt=6790'27403) query_epoch 11716) v4 ==== 308324+0+0 (1162585662 0 0) 0x22ff1800 con 0x1395f1e0 -198> 2014-09-18 18:08:37.699357 7f4c9f251700 5 -- op tracker -- , seq: 86383, time: 2014-09-18 18:08:37.695730, event: header_read, op: pg_log(2.3c6 epoch 11716 log log((4155'24411,6840'27412], crt=6790'27403) query_epoch 11716) -197> 2014-09-18 18:08:37.699424 7f4c9f251700 5 -- op tracker -- , seq: 86383, time: 2014-09-18 18:08:37.695731, event: throttled, op: pg_log(2.3c6 epoch 11716 log log((4155'24411,6840'27412], crt=6790'27403) query_epoch 11716) -196> 2014-09-18 18:08:37.699459 7f4c9f251700 5 -- op tracker -- , seq: 86383, time: 2014-09-18 18:08:37.698709, event: all_read, op: pg_log(2.3c6 epoch 11716 log log((4155'24411,6840'27412], crt=6790'27403) query_epoch 11716) -195> 2014-09-18 18:08:37.699470 7f4c9f251700 5 -- op tracker -- , seq: 86383, time: 2014-09-18 18:08:37.699339, event: dispatched, op: pg_log(2.3c6 epoch 11716 log log((4155'24411,6840'27412], crt=6790'27403) query_epoch 11716) -194> 2014-09-18 18:08:37.699496 7f4c97241700 10 monclient: _send_mon_message to mon.rack2-ram-3 at 10.242.42.188:6789/0 -193> 2014-09-18 18:08:37.699505 7f4c97241700 1 -- 10.242.42.178:6801/2483 --> 10.242.42.188:6789/0 -- osd_alive(want up_thru 11716 have 11716) v1 -- ?+0 0xf91cb20 con 0x111771e0 -192> 2014-09-18 18:08:37.699518 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.4c0(9) epoch 11716) v5 -- ?+0 0xf91c940 con 0x1395ec60 -191> 2014-09-18 18:08:37.699534 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x14409d40 con 0xcbecec0 -190> 2014-09-18 18:08:37.699544 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_notify(2.654(12) epoch 11716) v5 -- ?+0 0xf91d840 con 0xcbecec0 -189> 2014-09-18 18:08:37.699481 7f4c9f251700 5 -- op tracker -- , seq: 86383, time: 2014-09-18 18:08:37.699478, event: waiting_for_osdmap, op: pg_log(2.3c6 epoch 11716 log log((4155'24411,6840'27412], crt=6790'27403) query_epoch 11716) -188> 2014-09-18 18:08:37.699591 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_query(2.376 epoch 11716) v3 -- ?+0 0xf974760 con 0x1395ec60 -187> 2014-09-18 18:08:37.699606 7f4c9f251700 5 -- op tracker -- , seq: 86383, time: 2014-09-18 18:08:37.699602, event: started, op: pg_log(2.3c6 epoch 11716 log log((4155'24411,6840'27412], crt=6790'27403) query_epoch 11716) -186> 2014-09-18 18:08:37.699626 7f4c9f251700 5 -- op tracker -- , seq: 86383, time: 2014-09-18 18:08:37.699623, event: done, op: pg_log(2.3c6 epoch 11716 log log((4155'24411,6840'27412], crt=6790'27403) query_epoch 11716) -185> 2014-09-18 18:08:37.699660 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x12bf1680 con 0xdd8c940 -184> 2014-09-18 18:08:37.699676 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6802/5612 -- pg_query(2.376 epoch 11716) v3 -- ?+0 0xf973fe0 con 0xdd8c940 -183> 2014-09-18 18:08:37.699640 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3711 ==== pg_notify(2.517(11) epoch 11716) v5 ==== 1343+0+0 (125962759 0 0) 0x128cc3a0 con 0xcbecec0 -182> 2014-09-18 18:08:37.699701 7f4c9f251700 5 -- op tracker -- , seq: 86384, time: 2014-09-18 18:08:37.698796, event: header_read, op: pg_notify(2.517(11) epoch 11716) -181> 2014-09-18 18:08:37.699711 7f4c9f251700 5 -- op tracker -- , seq: 86384, time: 2014-09-18 18:08:37.698797, event: throttled, op: pg_notify(2.517(11) epoch 11716) -180> 2014-09-18 18:08:37.699720 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6835/46756 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0x12bf2400 con 0x142f8260 -179> 2014-09-18 18:08:37.699719 7f4c9f251700 5 -- op tracker -- , seq: 86384, time: 2014-09-18 18:08:37.698826, event: all_read, op: pg_notify(2.517(11) epoch 11716) -178> 2014-09-18 18:08:37.699731 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6835/46756 -- pg_query(2.376 epoch 11716) v3 -- ?+0 0xf972000 con 0x142f8260 -177> 2014-09-18 18:08:37.699741 7f4c9f251700 5 -- op tracker -- , seq: 86384, time: 2014-09-18 18:08:37.699696, event: dispatched, op: pg_notify(2.517(11) epoch 11716) -176> 2014-09-18 18:08:37.699747 7f4c9f251700 5 -- op tracker -- , seq: 86384, time: 2014-09-18 18:08:37.699746, event: waiting_for_osdmap, op: pg_notify(2.517(11) epoch 11716) -175> 2014-09-18 18:08:37.699755 7f4c9f251700 5 -- op tracker -- , seq: 86384, time: 2014-09-18 18:08:37.699754, event: started, op: pg_notify(2.517(11) epoch 11716) -174> 2014-09-18 18:08:37.699769 7f4c9f251700 5 -- op tracker -- , seq: 86384, time: 2014-09-18 18:08:37.699768, event: done, op: pg_notify(2.517(11) epoch 11716) -173> 2014-09-18 18:08:37.700001 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3324 ==== pg_notify(2.17e(17) epoch 11716) v5 ==== 1593+0+0 (473903931 0 0) 0xf730940 con 0x4900580 -172> 2014-09-18 18:08:37.700002 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.012391 7 0.001142 -171> 2014-09-18 18:08:37.700023 7f4c9f251700 5 -- op tracker -- , seq: 86385, time: 2014-09-18 18:08:37.699855, event: header_read, op: pg_notify(2.17e(17) epoch 11716) -170> 2014-09-18 18:08:37.700028 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive -169> 2014-09-18 18:08:37.700038 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[0.34( empty local-les=11491 n=0 ec=1 les/c 11491/11491 11715/11715/11715) [32,42,0] r=1 lpr=11715 pi=7889-11714/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -168> 2014-09-18 18:08:37.700048 7f4c9f251700 5 -- op tracker -- , seq: 86385, time: 2014-09-18 18:08:37.699856, event: throttled, op: pg_notify(2.17e(17) epoch 11716) -167> 2014-09-18 18:08:37.700057 7f4c9f251700 5 -- op tracker -- , seq: 86385, time: 2014-09-18 18:08:37.699986, event: all_read, op: pg_notify(2.17e(17) epoch 11716) -166> 2014-09-18 18:08:37.700062 7f4c9f251700 5 -- op tracker -- , seq: 86385, time: 2014-09-18 18:08:37.700018, event: dispatched, op: pg_notify(2.17e(17) epoch 11716) -165> 2014-09-18 18:08:37.700070 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_notify(2.67b(5) epoch 11716) v5 -- ?+0 0xf1ec580 con 0x1395ec60 -164> 2014-09-18 18:08:37.700076 7f4c9f251700 5 -- op tracker -- , seq: 86385, time: 2014-09-18 18:08:37.700074, event: waiting_for_osdmap, op: pg_notify(2.17e(17) epoch 11716) -163> 2014-09-18 18:08:37.700085 7f4c9f251700 5 -- op tracker -- , seq: 86385, time: 2014-09-18 18:08:37.700083, event: started, op: pg_notify(2.17e(17) epoch 11716) -162> 2014-09-18 18:08:37.700105 7f4c9f251700 5 -- op tracker -- , seq: 86385, time: 2014-09-18 18:08:37.700099, event: done, op: pg_notify(2.17e(17) epoch 11716) -161> 2014-09-18 18:08:37.700133 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6845/1010695 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0xe2efa80 con 0x12b53020 -160> 2014-09-18 18:08:37.700149 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6845/1010695 -- pg_notify(2.250(16) epoch 11716) v5 -- ?+0 0xfa28580 con 0x12b53020 -159> 2014-09-18 18:08:37.700190 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0xe2edd40 con 0x23a74f20 -158> 2014-09-18 18:08:37.700207 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.178:6819/4507 -- pg_notify(2.68e(181) epoch 11716) v5 -- ?+0 0xfa27a40 con 0x23a74f20 -157> 2014-09-18 18:08:37.700243 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- osd_map(11716..11716 src has 1..11716) v3 -- ?+0 0xe2ed200 con 0x14358260 -156> 2014-09-18 18:08:37.700261 7f4c96a40700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- pg_notify(2.b5(21) epoch 11716) v5 -- ?+0 0x24fd2580 con 0x14358260 -155> 2014-09-18 18:08:37.700304 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3712 ==== pg_notify(2.2c7(139) epoch 11716) v5 ==== 8651+0+0 (2627629293 0 0) 0xf91d840 con 0xcbecec0 -154> 2014-09-18 18:08:37.700321 7f4c9f251700 5 -- op tracker -- , seq: 86386, time: 2014-09-18 18:08:37.700140, event: header_read, op: pg_notify(2.2c7(139) epoch 11716) -153> 2014-09-18 18:08:37.700333 7f4c9f251700 5 -- op tracker -- , seq: 86386, time: 2014-09-18 18:08:37.700141, event: throttled, op: pg_notify(2.2c7(139) epoch 11716) -152> 2014-09-18 18:08:37.700342 7f4c9f251700 5 -- op tracker -- , seq: 86386, time: 2014-09-18 18:08:37.700286, event: all_read, op: pg_notify(2.2c7(139) epoch 11716) -151> 2014-09-18 18:08:37.700350 7f4c9f251700 5 -- op tracker -- , seq: 86386, time: 2014-09-18 18:08:37.700317, event: dispatched, op: pg_notify(2.2c7(139) epoch 11716) -150> 2014-09-18 18:08:37.700360 7f4c9f251700 5 -- op tracker -- , seq: 86386, time: 2014-09-18 18:08:37.700358, event: waiting_for_osdmap, op: pg_notify(2.2c7(139) epoch 11716) -149> 2014-09-18 18:08:37.700371 7f4c9f251700 5 -- op tracker -- , seq: 86386, time: 2014-09-18 18:08:37.700368, event: started, op: pg_notify(2.2c7(139) epoch 11716) -148> 2014-09-18 18:08:37.700397 7f4c9f251700 5 -- op tracker -- , seq: 86386, time: 2014-09-18 18:08:37.700394, event: done, op: pg_notify(2.2c7(139) epoch 11716) -147> 2014-09-18 18:08:37.700312 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] exit Started/Stray 1.004219 7 0.000240 -146> 2014-09-18 18:08:37.700190 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] exit Started/Stray 1.006303 7 0.000222 -145> 2014-09-18 18:08:37.700434 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] enter Started/ReplicaActive -144> 2014-09-18 18:08:37.700442 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -143> 2014-09-18 18:08:37.700454 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.b5( v 6775'26858 (4157'23883,6775'26858] local-les=11456 n=524 ec=188 les/c 11456/10730 11715/11715/11153) [11,27,9]/[56,42] r=1 lpr=11715 pi=10199-11714/21 crt=6775'26858 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -142> 2014-09-18 18:08:37.700462 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.654( v 6778'24660 (4157'21684,6778'24660] local-les=11712 n=529 ec=188 les/c 11505/11100 11715/11715/11332) [60,27,20]/[35,42] r=1 lpr=11715 pi=10768-11714/12 crt=6775'24651 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -141> 2014-09-18 18:08:37.700485 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.58 10.242.42.176:6835/46756 51 ==== pg_notify(2.376(59) epoch 11716) v5 ==== 4019+0+0 (1638015776 0 0) 0xf972000 con 0x142f8260 -140> 2014-09-18 18:08:37.700502 7f4c9f251700 5 -- op tracker -- , seq: 86387, time: 2014-09-18 18:08:37.700364, event: header_read, op: pg_notify(2.376(59) epoch 11716) -139> 2014-09-18 18:08:37.700515 7f4c9f251700 5 -- op tracker -- , seq: 86387, time: 2014-09-18 18:08:37.700365, event: throttled, op: pg_notify(2.376(59) epoch 11716) -138> 2014-09-18 18:08:37.700523 7f4c9f251700 5 -- op tracker -- , seq: 86387, time: 2014-09-18 18:08:37.700457, event: all_read, op: pg_notify(2.376(59) epoch 11716) -137> 2014-09-18 18:08:37.700530 7f4c9f251700 5 -- op tracker -- , seq: 86387, time: 2014-09-18 18:08:37.700497, event: dispatched, op: pg_notify(2.376(59) epoch 11716) -136> 2014-09-18 18:08:37.700537 7f4c9f251700 5 -- op tracker -- , seq: 86387, time: 2014-09-18 18:08:37.700535, event: waiting_for_osdmap, op: pg_notify(2.376(59) epoch 11716) -135> 2014-09-18 18:08:37.700545 7f4c9f251700 5 -- op tracker -- , seq: 86387, time: 2014-09-18 18:08:37.700543, event: started, op: pg_notify(2.376(59) epoch 11716) -134> 2014-09-18 18:08:37.700562 7f4c9f251700 5 -- op tracker -- , seq: 86387, time: 2014-09-18 18:08:37.700560, event: done, op: pg_notify(2.376(59) epoch 11716) -133> 2014-09-18 18:08:37.700580 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.24 10.242.42.172:6802/5612 2380 ==== pg_notify(2.376 epoch 11716) v5 ==== 722+0+0 (3083931827 0 0) 0xf973fe0 con 0xdd8c940 -132> 2014-09-18 18:08:37.700596 7f4c9f251700 5 -- op tracker -- , seq: 86388, time: 2014-09-18 18:08:37.700497, event: header_read, op: pg_notify(2.376 epoch 11716) -131> 2014-09-18 18:08:37.700607 7f4c9f251700 5 -- op tracker -- , seq: 86388, time: 2014-09-18 18:08:37.700498, event: throttled, op: pg_notify(2.376 epoch 11716) -130> 2014-09-18 18:08:37.700616 7f4c9f251700 5 -- op tracker -- , seq: 86388, time: 2014-09-18 18:08:37.700519, event: all_read, op: pg_notify(2.376 epoch 11716) -129> 2014-09-18 18:08:37.700620 7f4c9f251700 5 -- op tracker -- , seq: 86388, time: 2014-09-18 18:08:37.700591, event: dispatched, op: pg_notify(2.376 epoch 11716) -128> 2014-09-18 18:08:37.700627 7f4c9f251700 5 -- op tracker -- , seq: 86388, time: 2014-09-18 18:08:37.700625, event: waiting_for_osdmap, op: pg_notify(2.376 epoch 11716) -127> 2014-09-18 18:08:37.700632 7f4c9f251700 5 -- op tracker -- , seq: 86388, time: 2014-09-18 18:08:37.700631, event: started, op: pg_notify(2.376 epoch 11716) -126> 2014-09-18 18:08:37.700645 7f4c9f251700 5 -- op tracker -- , seq: 86388, time: 2014-09-18 18:08:37.700644, event: done, op: pg_notify(2.376 epoch 11716) -125> 2014-09-18 18:08:37.700660 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11506/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started/Stray 1.009335 7 0.000228 -124> 2014-09-18 18:08:37.700676 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11506/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -123> 2014-09-18 18:08:37.700687 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.30f( v 6742'23732 (4157'20731,6742'23732] lb 0//0//-1 local-les=0 n=0 ec=188 les/c 11506/10658 11715/11715/10657) [58,48,42]/[13,44] r=-1 lpr=11715 pi=10657-11714/11 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -122> 2014-09-18 18:08:37.700843 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] exit Started/Stray 1.004805 7 0.000325 -121> 2014-09-18 18:08:37.700856 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.23 10.242.42.172:6819/11463 3325 ==== pg_notify(2.1bb(9) epoch 11716) v5 ==== 1189+0+0 (577234994 0 0) 0xf730b20 con 0x4900580 -120> 2014-09-18 18:08:37.700865 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] enter Started/ReplicaActive -119> 2014-09-18 18:08:37.700872 7f4c9f251700 5 -- op tracker -- , seq: 86389, time: 2014-09-18 18:08:37.700804, event: header_read, op: pg_notify(2.1bb(9) epoch 11716) -118> 2014-09-18 18:08:37.700891 7f4c9f251700 5 -- op tracker -- , seq: 86389, time: 2014-09-18 18:08:37.700805, event: throttled, op: pg_notify(2.1bb(9) epoch 11716) -117> 2014-09-18 18:08:37.700885 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.250( v 6704'25402 (4156'22291,6704'25402] local-les=11507 n=547 ec=188 les/c 11507/10663 11715/11715/10662) [10,27,9]/[10,42] r=1 lpr=11715 pi=10123-11714/16 crt=6704'25402 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -116> 2014-09-18 18:08:37.700906 7f4c9f251700 5 -- op tracker -- , seq: 86389, time: 2014-09-18 18:08:37.700842, event: all_read, op: pg_notify(2.1bb(9) epoch 11716) -115> 2014-09-18 18:08:37.700913 7f4c9f251700 5 -- op tracker -- , seq: 86389, time: 2014-09-18 18:08:37.700867, event: dispatched, op: pg_notify(2.1bb(9) epoch 11716) -114> 2014-09-18 18:08:37.700920 7f4c9f251700 5 -- op tracker -- , seq: 86389, time: 2014-09-18 18:08:37.700918, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11716) -113> 2014-09-18 18:08:37.700929 7f4c9f251700 5 -- op tracker -- , seq: 86389, time: 2014-09-18 18:08:37.700926, event: started, op: pg_notify(2.1bb(9) epoch 11716) -112> 2014-09-18 18:08:37.700948 7f4c9f251700 5 -- op tracker -- , seq: 86389, time: 2014-09-18 18:08:37.700946, event: done, op: pg_notify(2.1bb(9) epoch 11716) -111> 2014-09-18 18:08:37.701028 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6808/22481 -- pg_notify(2.4e8(8) epoch 11716) v5 -- ?+0 0xf1ef860 con 0x1088aec0 -110> 2014-09-18 18:08:37.701361 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] exit Started/Stray 1.009890 7 0.000770 -109> 2014-09-18 18:08:37.701384 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] enter Started/ReplicaActive -108> 2014-09-18 18:08:37.701411 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 955 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0xe2edb00 con 0x1395f8c0 -107> 2014-09-18 18:08:37.701428 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -106> 2014-09-18 18:08:37.701433 7f4c96a40700 5 osd.42 pg_epoch: 11716 pg[2.756( v 6789'26114 (4075'22603,6789'26114] local-les=11709 n=512 ec=188 les/c 11709/10658 11715/11715/10657) [6,30,51]/[13,42] r=1 lpr=11715 pi=9952-11714/28 crt=6789'26114 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -105> 2014-09-18 18:08:37.703234 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.176:6814/1099 -- pg_info(1 pgs e11716:0.34) v4 -- ?+0 0x12112000 con 0x1395ef20 -104> 2014-09-18 18:08:37.703350 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 956 ==== pg_notify(2.204(10) epoch 11716) v5 ==== 1276+0+0 (3045109478 0 0) 0xe1a5840 con 0x1395f8c0 -103> 2014-09-18 18:08:37.703382 7f4c9f251700 5 -- op tracker -- , seq: 86390, time: 2014-09-18 18:08:37.701410, event: header_read, op: pg_notify(2.204(10) epoch 11716) -102> 2014-09-18 18:08:37.703396 7f4c9f251700 5 -- op tracker -- , seq: 86390, time: 2014-09-18 18:08:37.701411, event: throttled, op: pg_notify(2.204(10) epoch 11716) -101> 2014-09-18 18:08:37.703407 7f4c9f251700 5 -- op tracker -- , seq: 86390, time: 2014-09-18 18:08:37.701446, event: all_read, op: pg_notify(2.204(10) epoch 11716) -100> 2014-09-18 18:08:37.703417 7f4c9f251700 5 -- op tracker -- , seq: 86390, time: 2014-09-18 18:08:37.703375, event: dispatched, op: pg_notify(2.204(10) epoch 11716) -99> 2014-09-18 18:08:37.703427 7f4c9f251700 5 -- op tracker -- , seq: 86390, time: 2014-09-18 18:08:37.703425, event: waiting_for_osdmap, op: pg_notify(2.204(10) epoch 11716) -98> 2014-09-18 18:08:37.703441 7f4c9f251700 5 -- op tracker -- , seq: 86390, time: 2014-09-18 18:08:37.703439, event: started, op: pg_notify(2.204(10) epoch 11716) -97> 2014-09-18 18:08:37.703471 7f4c9f251700 5 -- op tracker -- , seq: 86390, time: 2014-09-18 18:08:37.703468, event: done, op: pg_notify(2.204(10) epoch 11716) -96> 2014-09-18 18:08:37.703488 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3713 ==== pg_notify(2.40a(162) epoch 11716) v5 ==== 9736+0+0 (2837257804 0 0) 0x128cd480 con 0xcbecec0 -95> 2014-09-18 18:08:37.703505 7f4c9f251700 5 -- op tracker -- , seq: 86391, time: 2014-09-18 18:08:37.702301, event: header_read, op: pg_notify(2.40a(162) epoch 11716) -94> 2014-09-18 18:08:37.703514 7f4c9f251700 5 -- op tracker -- , seq: 86391, time: 2014-09-18 18:08:37.702302, event: throttled, op: pg_notify(2.40a(162) epoch 11716) -93> 2014-09-18 18:08:37.703525 7f4c9f251700 5 -- op tracker -- , seq: 86391, time: 2014-09-18 18:08:37.702449, event: all_read, op: pg_notify(2.40a(162) epoch 11716) -92> 2014-09-18 18:08:37.703532 7f4c9f251700 5 -- op tracker -- , seq: 86391, time: 2014-09-18 18:08:37.703500, event: dispatched, op: pg_notify(2.40a(162) epoch 11716) -91> 2014-09-18 18:08:37.703539 7f4c9f251700 5 -- op tracker -- , seq: 86391, time: 2014-09-18 18:08:37.703537, event: waiting_for_osdmap, op: pg_notify(2.40a(162) epoch 11716) -90> 2014-09-18 18:08:37.703546 7f4c9f251700 5 -- op tracker -- , seq: 86391, time: 2014-09-18 18:08:37.703545, event: started, op: pg_notify(2.40a(162) epoch 11716) -89> 2014-09-18 18:08:37.703564 7f4c9f251700 5 -- op tracker -- , seq: 86391, time: 2014-09-18 18:08:37.703562, event: done, op: pg_notify(2.40a(162) epoch 11716) -88> 2014-09-18 18:08:37.703584 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3714 ==== pg_notify(2.186(12) epoch 11716) v5 ==== 1386+0+0 (120060266 0 0) 0x128cd0c0 con 0xcbecec0 -87> 2014-09-18 18:08:37.703594 7f4c9f251700 5 -- op tracker -- , seq: 86392, time: 2014-09-18 18:08:37.703252, event: header_read, op: pg_notify(2.186(12) epoch 11716) -86> 2014-09-18 18:08:37.703602 7f4c9f251700 5 -- op tracker -- , seq: 86392, time: 2014-09-18 18:08:37.703253, event: throttled, op: pg_notify(2.186(12) epoch 11716) -85> 2014-09-18 18:08:37.703606 7f4c9f251700 5 -- op tracker -- , seq: 86392, time: 2014-09-18 18:08:37.703286, event: all_read, op: pg_notify(2.186(12) epoch 11716) -84> 2014-09-18 18:08:37.703612 7f4c9f251700 5 -- op tracker -- , seq: 86392, time: 2014-09-18 18:08:37.703591, event: dispatched, op: pg_notify(2.186(12) epoch 11716) -83> 2014-09-18 18:08:37.703618 7f4c9f251700 5 -- op tracker -- , seq: 86392, time: 2014-09-18 18:08:37.703617, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11716) -82> 2014-09-18 18:08:37.703626 7f4c9f251700 5 -- op tracker -- , seq: 86392, time: 2014-09-18 18:08:37.703624, event: started, op: pg_notify(2.186(12) epoch 11716) -81> 2014-09-18 18:08:37.703631 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.164:6811/17421 -- pg_info(1 pgs e11716:2.654) v4 -- ?+0 0x12115660 con 0xcbecec0 -80> 2014-09-18 18:08:37.703648 7f4c9f251700 5 -- op tracker -- , seq: 86392, time: 2014-09-18 18:08:37.703646, event: done, op: pg_notify(2.186(12) epoch 11716) -79> 2014-09-18 18:08:37.703660 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.32 10.242.42.176:6814/1099 554 ==== pg_notify(2.4d0(10) epoch 11716) v5 ==== 1292+0+0 (1092780459 0 0) 0xf2b0ee0 con 0x1395ef20 -78> 2014-09-18 18:08:37.703673 7f4c9f251700 5 -- op tracker -- , seq: 86393, time: 2014-09-18 18:08:37.703445, event: header_read, op: pg_notify(2.4d0(10) epoch 11716) -77> 2014-09-18 18:08:37.703680 7f4c9f251700 5 -- op tracker -- , seq: 86393, time: 2014-09-18 18:08:37.703446, event: throttled, op: pg_notify(2.4d0(10) epoch 11716) -76> 2014-09-18 18:08:37.703686 7f4c9f251700 5 -- op tracker -- , seq: 86393, time: 2014-09-18 18:08:37.703483, event: all_read, op: pg_notify(2.4d0(10) epoch 11716) -75> 2014-09-18 18:08:37.703692 7f4c9f251700 5 -- op tracker -- , seq: 86393, time: 2014-09-18 18:08:37.703669, event: dispatched, op: pg_notify(2.4d0(10) epoch 11716) -74> 2014-09-18 18:08:37.703697 7f4c9f251700 5 -- op tracker -- , seq: 86393, time: 2014-09-18 18:08:37.703695, event: waiting_for_osdmap, op: pg_notify(2.4d0(10) epoch 11716) -73> 2014-09-18 18:08:37.703704 7f4c9f251700 5 -- op tracker -- , seq: 86393, time: 2014-09-18 18:08:37.703702, event: started, op: pg_notify(2.4d0(10) epoch 11716) -72> 2014-09-18 18:08:37.703804 7f4c9f251700 5 -- op tracker -- , seq: 86393, time: 2014-09-18 18:08:37.703802, event: done, op: pg_notify(2.4d0(10) epoch 11716) -71> 2014-09-18 18:08:37.703818 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3715 ==== pg_notify(2.719(17),2.71c(12),2.457(21) epoch 11716) v5 ==== 4800+0+0 (3557483733 0 0) 0xf2b5480 con 0xcbecec0 -70> 2014-09-18 18:08:37.703889 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.172:6837/36458 -- pg_info(1 pgs e11716:2.b5) v4 -- ?+0 0xe1aa1e0 con 0x14358260 -69> 2014-09-18 18:08:37.703829 7f4c9f251700 5 -- op tracker -- , seq: 86394, time: 2014-09-18 18:08:37.703589, event: header_read, op: pg_notify(2.719(17),2.71c(12),2.457(21) epoch 11716) -68> 2014-09-18 18:08:37.703916 7f4c9f251700 5 -- op tracker -- , seq: 86394, time: 2014-09-18 18:08:37.703590, event: throttled, op: pg_notify(2.719(17),2.71c(12),2.457(21) epoch 11716) -67> 2014-09-18 18:08:37.703923 7f4c9f251700 5 -- op tracker -- , seq: 86394, time: 2014-09-18 18:08:37.703668, event: all_read, op: pg_notify(2.719(17),2.71c(12),2.457(21) epoch 11716) -66> 2014-09-18 18:08:37.703930 7f4c9f251700 5 -- op tracker -- , seq: 86394, time: 2014-09-18 18:08:37.703827, event: dispatched, op: pg_notify(2.719(17),2.71c(12),2.457(21) epoch 11716) -65> 2014-09-18 18:08:37.703936 7f4c9f251700 5 -- op tracker -- , seq: 86394, time: 2014-09-18 18:08:37.703934, event: waiting_for_osdmap, op: pg_notify(2.719(17),2.71c(12),2.457(21) epoch 11716) -64> 2014-09-18 18:08:37.703944 7f4c9f251700 5 -- op tracker -- , seq: 86394, time: 2014-09-18 18:08:37.703942, event: started, op: pg_notify(2.719(17),2.71c(12),2.457(21) epoch 11716) -63> 2014-09-18 18:08:37.703973 7f4c9f251700 5 -- op tracker -- , seq: 86394, time: 2014-09-18 18:08:37.703971, event: done, op: pg_notify(2.719(17),2.71c(12),2.457(21) epoch 11716) -62> 2014-09-18 18:08:37.704025 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_info(1 pgs e11716:2.30f) v4 -- ?+0 0xf866d20 con 0x140f95a0 -61> 2014-09-18 18:08:37.704098 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6845/1010695 -- pg_info(1 pgs e11716:2.250) v4 -- ?+0 0x11b0ef00 con 0x12b53020 -60> 2014-09-18 18:08:37.704120 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_info(1 pgs e11716:2.756) v4 -- ?+0 0xde5f0e0 con 0x140f95a0 -59> 2014-09-18 18:08:37.704273 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.13 10.242.42.168:6840/1010879 312 ==== pg_query(2.763 epoch 11716) v3 ==== 144+0+0 (2643137185 0 0) 0x2b9410c0 con 0x140f95a0 -58> 2014-09-18 18:08:37.704285 7f4c9f251700 5 -- op tracker -- , seq: 86395, time: 2014-09-18 18:08:37.704240, event: header_read, op: pg_query(2.763 epoch 11716) -57> 2014-09-18 18:08:37.704293 7f4c9f251700 5 -- op tracker -- , seq: 86395, time: 2014-09-18 18:08:37.704241, event: throttled, op: pg_query(2.763 epoch 11716) -56> 2014-09-18 18:08:37.704297 7f4c9f251700 5 -- op tracker -- , seq: 86395, time: 2014-09-18 18:08:37.704261, event: all_read, op: pg_query(2.763 epoch 11716) -55> 2014-09-18 18:08:37.704303 7f4c9f251700 5 -- op tracker -- , seq: 86395, time: 2014-09-18 18:08:37.704282, event: dispatched, op: pg_query(2.763 epoch 11716) -54> 2014-09-18 18:08:37.704308 7f4c9f251700 5 -- op tracker -- , seq: 86395, time: 2014-09-18 18:08:37.704307, event: waiting_for_osdmap, op: pg_query(2.763 epoch 11716) -53> 2014-09-18 18:08:37.704316 7f4c9f251700 5 -- op tracker -- , seq: 86395, time: 2014-09-18 18:08:37.704315, event: started, op: pg_query(2.763 epoch 11716) -52> 2014-09-18 18:08:37.704354 7f4c9f251700 5 -- op tracker -- , seq: 86395, time: 2014-09-18 18:08:37.704352, event: done, op: pg_query(2.763 epoch 11716) -51> 2014-09-18 18:08:37.704520 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.35 10.242.42.164:6811/17421 3716 ==== pg_notify(2.1bb(9) epoch 11716) v5 ==== 1189+0+0 (2397985313 0 0) 0x12115660 con 0xcbecec0 -50> 2014-09-18 18:08:37.704531 7f4c9f251700 5 -- op tracker -- , seq: 86396, time: 2014-09-18 18:08:37.704473, event: header_read, op: pg_notify(2.1bb(9) epoch 11716) -49> 2014-09-18 18:08:37.704538 7f4c9f251700 5 -- op tracker -- , seq: 86396, time: 2014-09-18 18:08:37.704474, event: throttled, op: pg_notify(2.1bb(9) epoch 11716) -48> 2014-09-18 18:08:37.704543 7f4c9f251700 5 -- op tracker -- , seq: 86396, time: 2014-09-18 18:08:37.704509, event: all_read, op: pg_notify(2.1bb(9) epoch 11716) -47> 2014-09-18 18:08:37.704549 7f4c9f251700 5 -- op tracker -- , seq: 86396, time: 2014-09-18 18:08:37.704528, event: dispatched, op: pg_notify(2.1bb(9) epoch 11716) -46> 2014-09-18 18:08:37.704554 7f4c9f251700 5 -- op tracker -- , seq: 86396, time: 2014-09-18 18:08:37.704552, event: waiting_for_osdmap, op: pg_notify(2.1bb(9) epoch 11716) -45> 2014-09-18 18:08:37.704566 7f4c9f251700 5 -- op tracker -- , seq: 86396, time: 2014-09-18 18:08:37.704563, event: started, op: pg_notify(2.1bb(9) epoch 11716) -44> 2014-09-18 18:08:37.704590 7f4c9f251700 5 -- op tracker -- , seq: 86396, time: 2014-09-18 18:08:37.704587, event: done, op: pg_notify(2.1bb(9) epoch 11716) -43> 2014-09-18 18:08:37.704693 7f4c97241700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.168:6840/1010879 -- pg_notify(2.763(57) epoch 11716) v5 -- ?+0 0x12c11e00 con 0x140f95a0 -42> 2014-09-18 18:08:37.704821 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 957 ==== pg_notify(2.2d8(174) epoch 11716) v5 ==== 10808+0+0 (920023174 0 0) 0xe1a50c0 con 0x1395f8c0 -41> 2014-09-18 18:08:37.704995 7f4c9f251700 5 -- op tracker -- , seq: 86397, time: 2014-09-18 18:08:37.704657, event: header_read, op: pg_notify(2.2d8(174) epoch 11716) -40> 2014-09-18 18:08:37.705006 7f4c9f251700 5 -- op tracker -- , seq: 86397, time: 2014-09-18 18:08:37.704658, event: throttled, op: pg_notify(2.2d8(174) epoch 11716) -39> 2014-09-18 18:08:37.705011 7f4c9f251700 5 -- op tracker -- , seq: 86397, time: 2014-09-18 18:08:37.704806, event: all_read, op: pg_notify(2.2d8(174) epoch 11716) -38> 2014-09-18 18:08:37.705018 7f4c9f251700 5 -- op tracker -- , seq: 86397, time: 2014-09-18 18:08:37.704986, event: dispatched, op: pg_notify(2.2d8(174) epoch 11716) -37> 2014-09-18 18:08:37.705023 7f4c9f251700 5 -- op tracker -- , seq: 86397, time: 2014-09-18 18:08:37.705022, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11716) -36> 2014-09-18 18:08:37.705029 7f4c9f251700 5 -- op tracker -- , seq: 86397, time: 2014-09-18 18:08:37.705028, event: started, op: pg_notify(2.2d8(174) epoch 11716) -35> 2014-09-18 18:08:37.705060 7f4c9f251700 5 -- op tracker -- , seq: 86397, time: 2014-09-18 18:08:37.705057, event: done, op: pg_notify(2.2d8(174) epoch 11716) -34> 2014-09-18 18:08:37.712926 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 352 ==== osd_map(11715..11715 src has 1..11715) v3 ==== 3478+0+0 (3560729998 0 0) 0xe939a80 con 0x12562260 -33> 2014-09-18 18:08:37.712954 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11715,11715], i have 11716, src has [1,11715] -32> 2014-09-18 18:08:37.713001 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.47 10.242.42.164:6831/25184 353 ==== pg_notify(2.719(17) epoch 11715) v5 ==== 1601+0+0 (3573084045 0 0) 0x117385a0 con 0x12562260 -31> 2014-09-18 18:08:37.713017 7f4c9f251700 5 -- op tracker -- , seq: 86398, time: 2014-09-18 18:08:37.712935, event: header_read, op: pg_notify(2.719(17) epoch 11715) -30> 2014-09-18 18:08:37.713029 7f4c9f251700 5 -- op tracker -- , seq: 86398, time: 2014-09-18 18:08:37.712936, event: throttled, op: pg_notify(2.719(17) epoch 11715) -29> 2014-09-18 18:08:37.713034 7f4c9f251700 5 -- op tracker -- , seq: 86398, time: 2014-09-18 18:08:37.712979, event: all_read, op: pg_notify(2.719(17) epoch 11715) -28> 2014-09-18 18:08:37.713039 7f4c9f251700 5 -- op tracker -- , seq: 86398, time: 2014-09-18 18:08:37.713013, event: dispatched, op: pg_notify(2.719(17) epoch 11715) -27> 2014-09-18 18:08:37.713045 7f4c9f251700 5 -- op tracker -- , seq: 86398, time: 2014-09-18 18:08:37.713043, event: waiting_for_osdmap, op: pg_notify(2.719(17) epoch 11715) -26> 2014-09-18 18:08:37.713053 7f4c9f251700 5 -- op tracker -- , seq: 86398, time: 2014-09-18 18:08:37.713052, event: started, op: pg_notify(2.719(17) epoch 11715) -25> 2014-09-18 18:08:37.713092 7f4c9f251700 5 -- op tracker -- , seq: 86398, time: 2014-09-18 18:08:37.713072, event: done, op: pg_notify(2.719(17) epoch 11715) -24> 2014-09-18 18:08:37.716414 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3266 ==== osd_map(11716..11716 src has 1..11716) v3 ==== 1237+0+0 (716772192 0 0) 0x11172000 con 0x1395ec60 -23> 2014-09-18 18:08:37.716433 7f4c9f251700 3 osd.42 11716 handle_osd_map epochs [11716,11716], i have 11716, src has [1,11716] -22> 2014-09-18 18:08:37.716472 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3267 ==== pg_notify(2.71(19) epoch 11716) v5 ==== 1795+0+0 (2121635773 0 0) 0xf1ec580 con 0x1395ec60 -21> 2014-09-18 18:08:37.716484 7f4c9f251700 5 -- op tracker -- , seq: 86399, time: 2014-09-18 18:08:37.716428, event: header_read, op: pg_notify(2.71(19) epoch 11716) -20> 2014-09-18 18:08:37.716493 7f4c9f251700 5 -- op tracker -- , seq: 86399, time: 2014-09-18 18:08:37.716429, event: throttled, op: pg_notify(2.71(19) epoch 11716) -19> 2014-09-18 18:08:37.716497 7f4c9f251700 5 -- op tracker -- , seq: 86399, time: 2014-09-18 18:08:37.716461, event: all_read, op: pg_notify(2.71(19) epoch 11716) -18> 2014-09-18 18:08:37.716503 7f4c9f251700 5 -- op tracker -- , seq: 86399, time: 2014-09-18 18:08:37.716481, event: dispatched, op: pg_notify(2.71(19) epoch 11716) -17> 2014-09-18 18:08:37.716508 7f4c9f251700 5 -- op tracker -- , seq: 86399, time: 2014-09-18 18:08:37.716507, event: waiting_for_osdmap, op: pg_notify(2.71(19) epoch 11716) -16> 2014-09-18 18:08:37.716516 7f4c9f251700 5 -- op tracker -- , seq: 86399, time: 2014-09-18 18:08:37.716515, event: started, op: pg_notify(2.71(19) epoch 11716) -15> 2014-09-18 18:08:37.716548 7f4c9f251700 5 -- op tracker -- , seq: 86399, time: 2014-09-18 18:08:37.716546, event: done, op: pg_notify(2.71(19) epoch 11716) -14> 2014-09-18 18:08:37.717440 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3268 ==== pg_notify(2.376(11) epoch 11716) v5 ==== 1355+0+0 (2012410958 0 0) 0xf974760 con 0x1395ec60 -13> 2014-09-18 18:08:37.717457 7f4c9f251700 5 -- op tracker -- , seq: 86400, time: 2014-09-18 18:08:37.717396, event: header_read, op: pg_notify(2.376(11) epoch 11716) -12> 2014-09-18 18:08:37.717467 7f4c9f251700 5 -- op tracker -- , seq: 86400, time: 2014-09-18 18:08:37.717397, event: throttled, op: pg_notify(2.376(11) epoch 11716) -11> 2014-09-18 18:08:37.717472 7f4c9f251700 5 -- op tracker -- , seq: 86400, time: 2014-09-18 18:08:37.717427, event: all_read, op: pg_notify(2.376(11) epoch 11716) -10> 2014-09-18 18:08:37.717478 7f4c9f251700 5 -- op tracker -- , seq: 86400, time: 2014-09-18 18:08:37.717453, event: dispatched, op: pg_notify(2.376(11) epoch 11716) -9> 2014-09-18 18:08:37.717484 7f4c9f251700 5 -- op tracker -- , seq: 86400, time: 2014-09-18 18:08:37.717482, event: waiting_for_osdmap, op: pg_notify(2.376(11) epoch 11716) -8> 2014-09-18 18:08:37.717492 7f4c9f251700 5 -- op tracker -- , seq: 86400, time: 2014-09-18 18:08:37.717490, event: started, op: pg_notify(2.376(11) epoch 11716) -7> 2014-09-18 18:08:37.717510 7f4c9f251700 5 -- op tracker -- , seq: 86400, time: 2014-09-18 18:08:37.717509, event: done, op: pg_notify(2.376(11) epoch 11716) -6> 2014-09-18 18:08:37.717559 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.019084 5 0.000408 -5> 2014-09-18 18:08:37.717758 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -4> 2014-09-18 18:08:37.717804 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 bft=24 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000045 0 0.000000 -3> 2014-09-18 18:08:37.717911 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 bft=24 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -2> 2014-09-18 18:08:37.717962 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 bft=24 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000052 0 0.000000 -1> 2014-09-18 18:08:37.717975 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.376( v 6772'25996 (4155'22997,6772'25996] local-les=11491 n=524 ec=188 les/c 11491/11491 11715/11716/9945) [42,24,58]/[42,58,9] r=0 lpr=11716 pi=11490-11715/2 bft=24 crt=6764'25987 lcod 0'0 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru 0> 2014-09-18 18:08:37.717934 7f4c96a40700 -1 osd/PGLog.h: In function 'void PGLog::IndexedLog::claim_log_and_clear_rollback_info(const pg_log_t&)' thread 7f4c96a40700 time 2014-09-18 18:08:37.701817 osd/PGLog.h: 87: FAILED assert(rollback_info_trimmed_to == head) ceph version 0.84-sd-sprint4 (3215c520e1306f50d0094b5646636c02456c9df4) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xb776db] 2: (PG::RecoveryState::Stray::react(PG::MLogRec const&)+0x1f9) [0x7e1d19] 3: (boost::statechart::simple_state, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x1f4) [0x81a944] 4: (boost::statechart::state_machine, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x805d6b] 5: (PG::handle_peering_event(std::tr1::shared_ptr, PG::RecoveryCtx*)+0x1ce) [0x7b5b8e] 6: (OSD::process_peering_events(std::list > const&, ThreadPool::TPHandle&)+0x2b0) [0x6a3140] 7: (OSD::PeeringWQ::_process(std::list > const&, ThreadPool::TPHandle&)+0x18) [0x6f83c8] 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa46) [0xb68866] 9: (ThreadPool::WorkThread::entry()+0x10) [0xb69910] 10: (()+0x8182) [0x7f4cb21da182] 11: (clone()+0x6d) [0x7f4cb05c030d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.42.log --- end dump of recent events --- 2014-09-18 18:08:37.801977 7f4c96a40700 -1 *** Caught signal (Aborted) ** in thread 7f4c96a40700 ceph version 0.84-sd-sprint4 (3215c520e1306f50d0094b5646636c02456c9df4) 1: /usr/bin/ceph-osd() [0xa8c7ff] 2: (()+0x10340) [0x7f4cb21e2340] 3: (gsignal()+0x39) [0x7f4cb04fbf79] 4: (abort()+0x148) [0x7f4cb04ff388] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f4cb0e076b5] 6: (()+0x5e836) [0x7f4cb0e05836] 7: (()+0x5e863) [0x7f4cb0e05863] 8: (()+0x5eaa2) [0x7f4cb0e05aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xb778c8] 10: (PG::RecoveryState::Stray::react(PG::MLogRec const&)+0x1f9) [0x7e1d19] 11: (boost::statechart::simple_state, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x1f4) [0x81a944] 12: (boost::statechart::state_machine, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x805d6b] 13: (PG::handle_peering_event(std::tr1::shared_ptr, PG::RecoveryCtx*)+0x1ce) [0x7b5b8e] 14: (OSD::process_peering_events(std::list > const&, ThreadPool::TPHandle&)+0x2b0) [0x6a3140] 15: (OSD::PeeringWQ::_process(std::list > const&, ThreadPool::TPHandle&)+0x18) [0x6f83c8] 16: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa46) [0xb68866] 17: (ThreadPool::WorkThread::entry()+0x10) [0xb69910] 18: (()+0x8182) [0x7f4cb21da182] 19: (clone()+0x6d) [0x7f4cb05c030d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -101> 2014-09-18 18:08:37.719116 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 958 ==== pg_info(1 pgs e11716:2.2d8) v4 ==== 734+0+0 (3719967135 0 0) 0xe76f0c0 con 0x1395f8c0 -100> 2014-09-18 18:08:37.719133 7f4c9f251700 5 -- op tracker -- , seq: 86401, time: 2014-09-18 18:08:37.719072, event: header_read, op: pg_info(1 pgs e11716:2.2d8) -99> 2014-09-18 18:08:37.719142 7f4c9f251700 5 -- op tracker -- , seq: 86401, time: 2014-09-18 18:08:37.719073, event: throttled, op: pg_info(1 pgs e11716:2.2d8) -98> 2014-09-18 18:08:37.719146 7f4c9f251700 5 -- op tracker -- , seq: 86401, time: 2014-09-18 18:08:37.719102, event: all_read, op: pg_info(1 pgs e11716:2.2d8) -97> 2014-09-18 18:08:37.719150 7f4c9f251700 5 -- op tracker -- , seq: 86401, time: 2014-09-18 18:08:37.719129, event: dispatched, op: pg_info(1 pgs e11716:2.2d8) -96> 2014-09-18 18:08:37.719155 7f4c9f251700 5 -- op tracker -- , seq: 86401, time: 2014-09-18 18:08:37.719153, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.2d8) -95> 2014-09-18 18:08:37.719162 7f4c9f251700 5 -- op tracker -- , seq: 86401, time: 2014-09-18 18:08:37.719161, event: started, op: pg_info(1 pgs e11716:2.2d8) -94> 2014-09-18 18:08:37.719179 7f4c9f251700 5 -- op tracker -- , seq: 86401, time: 2014-09-18 18:08:37.719178, event: done, op: pg_info(1 pgs e11716:2.2d8) -93> 2014-09-18 18:08:37.721422 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.27 10.242.42.174:6801/21548 959 ==== pg_info(1 pgs e11716:2.204) v4 ==== 738+0+0 (2776262433 0 0) 0xe76c960 con 0x1395f8c0 -92> 2014-09-18 18:08:37.721431 7f4c9f251700 5 -- op tracker -- , seq: 86402, time: 2014-09-18 18:08:37.721375, event: header_read, op: pg_info(1 pgs e11716:2.204) -91> 2014-09-18 18:08:37.721436 7f4c9f251700 5 -- op tracker -- , seq: 86402, time: 2014-09-18 18:08:37.721375, event: throttled, op: pg_info(1 pgs e11716:2.204) -90> 2014-09-18 18:08:37.721448 7f4c9f251700 5 -- op tracker -- , seq: 86402, time: 2014-09-18 18:08:37.721400, event: all_read, op: pg_info(1 pgs e11716:2.204) -89> 2014-09-18 18:08:37.721456 7f4c9f251700 5 -- op tracker -- , seq: 86402, time: 2014-09-18 18:08:37.721429, event: dispatched, op: pg_info(1 pgs e11716:2.204) -88> 2014-09-18 18:08:37.721462 7f4c9f251700 5 -- op tracker -- , seq: 86402, time: 2014-09-18 18:08:37.721460, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.204) -87> 2014-09-18 18:08:37.721467 7f4c9f251700 5 -- op tracker -- , seq: 86402, time: 2014-09-18 18:08:37.721466, event: started, op: pg_info(1 pgs e11716:2.204) -86> 2014-09-18 18:08:37.721483 7f4c9f251700 5 -- op tracker -- , seq: 86402, time: 2014-09-18 18:08:37.721482, event: done, op: pg_info(1 pgs e11716:2.204) -85> 2014-09-18 18:08:37.735735 7f4caae41700 5 osd.42 11716 tick -84> 2014-09-18 18:08:37.740577 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3269 ==== pg_notify(2.204(64) epoch 11716) v5 ==== 4270+0+0 (3290542762 0 0) 0xf91c940 con 0x1395ec60 -83> 2014-09-18 18:08:37.740593 7f4c9f251700 5 -- op tracker -- , seq: 86403, time: 2014-09-18 18:08:37.740451, event: header_read, op: pg_notify(2.204(64) epoch 11716) -82> 2014-09-18 18:08:37.740616 7f4c9f251700 5 -- op tracker -- , seq: 86403, time: 2014-09-18 18:08:37.740452, event: throttled, op: pg_notify(2.204(64) epoch 11716) -81> 2014-09-18 18:08:37.740621 7f4c9f251700 5 -- op tracker -- , seq: 86403, time: 2014-09-18 18:08:37.740522, event: all_read, op: pg_notify(2.204(64) epoch 11716) -80> 2014-09-18 18:08:37.740625 7f4c9f251700 5 -- op tracker -- , seq: 86403, time: 2014-09-18 18:08:37.740588, event: dispatched, op: pg_notify(2.204(64) epoch 11716) -79> 2014-09-18 18:08:37.740630 7f4c9f251700 5 -- op tracker -- , seq: 86403, time: 2014-09-18 18:08:37.740629, event: waiting_for_osdmap, op: pg_notify(2.204(64) epoch 11716) -78> 2014-09-18 18:08:37.740636 7f4c9f251700 5 -- op tracker -- , seq: 86403, time: 2014-09-18 18:08:37.740635, event: started, op: pg_notify(2.204(64) epoch 11716) -77> 2014-09-18 18:08:37.740655 7f4c9f251700 5 -- op tracker -- , seq: 86403, time: 2014-09-18 18:08:37.740654, event: done, op: pg_notify(2.204(64) epoch 11716) -76> 2014-09-18 18:08:37.742797 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3270 ==== pg_notify(2.519(4) epoch 11716) v5 ==== 962+0+0 (1844456051 0 0) 0x2b93e1e0 con 0x1395ec60 -75> 2014-09-18 18:08:37.742810 7f4c9f251700 5 -- op tracker -- , seq: 86404, time: 2014-09-18 18:08:37.742749, event: header_read, op: pg_notify(2.519(4) epoch 11716) -74> 2014-09-18 18:08:37.742817 7f4c9f251700 5 -- op tracker -- , seq: 86404, time: 2014-09-18 18:08:37.742750, event: throttled, op: pg_notify(2.519(4) epoch 11716) -73> 2014-09-18 18:08:37.742823 7f4c9f251700 5 -- op tracker -- , seq: 86404, time: 2014-09-18 18:08:37.742785, event: all_read, op: pg_notify(2.519(4) epoch 11716) -72> 2014-09-18 18:08:37.742827 7f4c9f251700 5 -- op tracker -- , seq: 86404, time: 2014-09-18 18:08:37.742807, event: dispatched, op: pg_notify(2.519(4) epoch 11716) -71> 2014-09-18 18:08:37.742840 7f4c9f251700 5 -- op tracker -- , seq: 86404, time: 2014-09-18 18:08:37.742839, event: waiting_for_osdmap, op: pg_notify(2.519(4) epoch 11716) -70> 2014-09-18 18:08:37.742847 7f4c9f251700 5 -- op tracker -- , seq: 86404, time: 2014-09-18 18:08:37.742845, event: started, op: pg_notify(2.519(4) epoch 11716) -69> 2014-09-18 18:08:37.742867 7f4c9f251700 5 -- op tracker -- , seq: 86404, time: 2014-09-18 18:08:37.742865, event: done, op: pg_notify(2.519(4) epoch 11716) -68> 2014-09-18 18:08:37.744556 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3271 ==== pg_notify(2.2d8(174) epoch 11716) v5 ==== 10808+0+0 (4178935455 0 0) 0xfa27fe0 con 0x1395ec60 -67> 2014-09-18 18:08:37.744570 7f4c9f251700 5 -- op tracker -- , seq: 86405, time: 2014-09-18 18:08:37.744384, event: header_read, op: pg_notify(2.2d8(174) epoch 11716) -66> 2014-09-18 18:08:37.744578 7f4c9f251700 5 -- op tracker -- , seq: 86405, time: 2014-09-18 18:08:37.744385, event: throttled, op: pg_notify(2.2d8(174) epoch 11716) -65> 2014-09-18 18:08:37.744582 7f4c9f251700 5 -- op tracker -- , seq: 86405, time: 2014-09-18 18:08:37.744542, event: all_read, op: pg_notify(2.2d8(174) epoch 11716) -64> 2014-09-18 18:08:37.744588 7f4c9f251700 5 -- op tracker -- , seq: 86405, time: 2014-09-18 18:08:37.744567, event: dispatched, op: pg_notify(2.2d8(174) epoch 11716) -63> 2014-09-18 18:08:37.744592 7f4c9f251700 5 -- op tracker -- , seq: 86405, time: 2014-09-18 18:08:37.744591, event: waiting_for_osdmap, op: pg_notify(2.2d8(174) epoch 11716) -62> 2014-09-18 18:08:37.744608 7f4c9f251700 5 -- op tracker -- , seq: 86405, time: 2014-09-18 18:08:37.744606, event: started, op: pg_notify(2.2d8(174) epoch 11716) -61> 2014-09-18 18:08:37.744625 7f4c9f251700 5 -- op tracker -- , seq: 86405, time: 2014-09-18 18:08:37.744623, event: done, op: pg_notify(2.2d8(174) epoch 11716) -60> 2014-09-18 18:08:37.746222 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3272 ==== pg_notify(2.410(12) epoch 11716) v5 ==== 1376+0+0 (1518218661 0 0) 0xfa28d00 con 0x1395ec60 -59> 2014-09-18 18:08:37.746245 7f4c9f251700 5 -- op tracker -- , seq: 86406, time: 2014-09-18 18:08:37.746167, event: header_read, op: pg_notify(2.410(12) epoch 11716) -58> 2014-09-18 18:08:37.746260 7f4c9f251700 5 -- op tracker -- , seq: 86406, time: 2014-09-18 18:08:37.746168, event: throttled, op: pg_notify(2.410(12) epoch 11716) -57> 2014-09-18 18:08:37.746264 7f4c9f251700 5 -- op tracker -- , seq: 86406, time: 2014-09-18 18:08:37.746210, event: all_read, op: pg_notify(2.410(12) epoch 11716) -56> 2014-09-18 18:08:37.746268 7f4c9f251700 5 -- op tracker -- , seq: 86406, time: 2014-09-18 18:08:37.746242, event: dispatched, op: pg_notify(2.410(12) epoch 11716) -55> 2014-09-18 18:08:37.746273 7f4c9f251700 5 -- op tracker -- , seq: 86406, time: 2014-09-18 18:08:37.746272, event: waiting_for_osdmap, op: pg_notify(2.410(12) epoch 11716) -54> 2014-09-18 18:08:37.746279 7f4c9f251700 5 -- op tracker -- , seq: 86406, time: 2014-09-18 18:08:37.746277, event: started, op: pg_notify(2.410(12) epoch 11716) -53> 2014-09-18 18:08:37.746302 7f4c9f251700 5 -- op tracker -- , seq: 86406, time: 2014-09-18 18:08:37.746300, event: done, op: pg_notify(2.410(12) epoch 11716) -52> 2014-09-18 18:08:37.750477 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3273 ==== pg_notify(2.186(12) epoch 11716) v5 ==== 1386+0+0 (1358484302 0 0) 0x21cfa780 con 0x1395ec60 -51> 2014-09-18 18:08:37.750493 7f4c9f251700 5 -- op tracker -- , seq: 86407, time: 2014-09-18 18:08:37.750378, event: header_read, op: pg_notify(2.186(12) epoch 11716) -50> 2014-09-18 18:08:37.750512 7f4c9f251700 5 -- op tracker -- , seq: 86407, time: 2014-09-18 18:08:37.750378, event: throttled, op: pg_notify(2.186(12) epoch 11716) -49> 2014-09-18 18:08:37.750516 7f4c9f251700 5 -- op tracker -- , seq: 86407, time: 2014-09-18 18:08:37.750421, event: all_read, op: pg_notify(2.186(12) epoch 11716) -48> 2014-09-18 18:08:37.750521 7f4c9f251700 5 -- op tracker -- , seq: 86407, time: 2014-09-18 18:08:37.750489, event: dispatched, op: pg_notify(2.186(12) epoch 11716) -47> 2014-09-18 18:08:37.750525 7f4c9f251700 5 -- op tracker -- , seq: 86407, time: 2014-09-18 18:08:37.750524, event: waiting_for_osdmap, op: pg_notify(2.186(12) epoch 11716) -46> 2014-09-18 18:08:37.750531 7f4c9f251700 5 -- op tracker -- , seq: 86407, time: 2014-09-18 18:08:37.750530, event: started, op: pg_notify(2.186(12) epoch 11716) -45> 2014-09-18 18:08:37.750564 7f4c9f251700 5 -- op tracker -- , seq: 86407, time: 2014-09-18 18:08:37.750562, event: done, op: pg_notify(2.186(12) epoch 11716) -44> 2014-09-18 18:08:37.751232 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3274 ==== pg_notify(2.74e(11) epoch 11716) v5 ==== 1303+0+0 (2412510441 0 0) 0x21cfa1e0 con 0x1395ec60 -43> 2014-09-18 18:08:37.751253 7f4c9f251700 5 -- op tracker -- , seq: 86408, time: 2014-09-18 18:08:37.751147, event: header_read, op: pg_notify(2.74e(11) epoch 11716) -42> 2014-09-18 18:08:37.751268 7f4c9f251700 5 -- op tracker -- , seq: 86408, time: 2014-09-18 18:08:37.751148, event: throttled, op: pg_notify(2.74e(11) epoch 11716) -41> 2014-09-18 18:08:37.751277 7f4c9f251700 5 -- op tracker -- , seq: 86408, time: 2014-09-18 18:08:37.751175, event: all_read, op: pg_notify(2.74e(11) epoch 11716) -40> 2014-09-18 18:08:37.751284 7f4c9f251700 5 -- op tracker -- , seq: 86408, time: 2014-09-18 18:08:37.751247, event: dispatched, op: pg_notify(2.74e(11) epoch 11716) -39> 2014-09-18 18:08:37.751306 7f4c9f251700 5 -- op tracker -- , seq: 86408, time: 2014-09-18 18:08:37.751303, event: waiting_for_osdmap, op: pg_notify(2.74e(11) epoch 11716) -38> 2014-09-18 18:08:37.751319 7f4c9f251700 5 -- op tracker -- , seq: 86408, time: 2014-09-18 18:08:37.751316, event: started, op: pg_notify(2.74e(11) epoch 11716) -37> 2014-09-18 18:08:37.751343 7f4c9f251700 5 -- op tracker -- , seq: 86408, time: 2014-09-18 18:08:37.751341, event: done, op: pg_notify(2.74e(11) epoch 11716) -36> 2014-09-18 18:08:37.752392 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3275 ==== pg_notify(2.78f(55),2.4c6(15),2.799(9),2.4d0(10) epoch 11716) v5 ==== 7659+0+0 (2731110406 0 0) 0xfae6000 con 0x1395ec60 -35> 2014-09-18 18:08:37.752406 7f4c9f251700 5 -- op tracker -- , seq: 86409, time: 2014-09-18 18:08:37.752289, event: header_read, op: pg_notify(2.78f(55),2.4c6(15),2.799(9),2.4d0(10) epoch 11716) -34> 2014-09-18 18:08:37.752414 7f4c9f251700 5 -- op tracker -- , seq: 86409, time: 2014-09-18 18:08:37.752290, event: throttled, op: pg_notify(2.78f(55),2.4c6(15),2.799(9),2.4d0(10) epoch 11716) -33> 2014-09-18 18:08:37.752418 7f4c9f251700 5 -- op tracker -- , seq: 86409, time: 2014-09-18 18:08:37.752379, event: all_read, op: pg_notify(2.78f(55),2.4c6(15),2.799(9),2.4d0(10) epoch 11716) -32> 2014-09-18 18:08:37.752432 7f4c9f251700 5 -- op tracker -- , seq: 86409, time: 2014-09-18 18:08:37.752402, event: dispatched, op: pg_notify(2.78f(55),2.4c6(15),2.799(9),2.4d0(10) epoch 11716) -31> 2014-09-18 18:08:37.752437 7f4c9f251700 5 -- op tracker -- , seq: 86409, time: 2014-09-18 18:08:37.752435, event: waiting_for_osdmap, op: pg_notify(2.78f(55),2.4c6(15),2.799(9),2.4d0(10) epoch 11716) -30> 2014-09-18 18:08:37.752447 7f4c9f251700 5 -- op tracker -- , seq: 86409, time: 2014-09-18 18:08:37.752445, event: started, op: pg_notify(2.78f(55),2.4c6(15),2.799(9),2.4d0(10) epoch 11716) -29> 2014-09-18 18:08:37.752479 7f4c9f251700 5 -- op tracker -- , seq: 86409, time: 2014-09-18 18:08:37.752477, event: done, op: pg_notify(2.78f(55),2.4c6(15),2.799(9),2.4d0(10) epoch 11716) -28> 2014-09-18 18:08:37.754741 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3276 ==== pg_notify(2.4b1(13),2.4c1(15) epoch 11716) v5 ==== 2936+0+0 (2829392540 0 0) 0xfae65a0 con 0x1395ec60 -27> 2014-09-18 18:08:37.754753 7f4c9f251700 5 -- op tracker -- , seq: 86410, time: 2014-09-18 18:08:37.754684, event: header_read, op: pg_notify(2.4b1(13),2.4c1(15) epoch 11716) -26> 2014-09-18 18:08:37.754759 7f4c9f251700 5 -- op tracker -- , seq: 86410, time: 2014-09-18 18:08:37.754685, event: throttled, op: pg_notify(2.4b1(13),2.4c1(15) epoch 11716) -25> 2014-09-18 18:08:37.754763 7f4c9f251700 5 -- op tracker -- , seq: 86410, time: 2014-09-18 18:08:37.754728, event: all_read, op: pg_notify(2.4b1(13),2.4c1(15) epoch 11716) -24> 2014-09-18 18:08:37.754772 7f4c9f251700 5 -- op tracker -- , seq: 86410, time: 2014-09-18 18:08:37.754749, event: dispatched, op: pg_notify(2.4b1(13),2.4c1(15) epoch 11716) -23> 2014-09-18 18:08:37.754776 7f4c9f251700 5 -- op tracker -- , seq: 86410, time: 2014-09-18 18:08:37.754775, event: waiting_for_osdmap, op: pg_notify(2.4b1(13),2.4c1(15) epoch 11716) -22> 2014-09-18 18:08:37.754782 7f4c9f251700 5 -- op tracker -- , seq: 86410, time: 2014-09-18 18:08:37.754780, event: started, op: pg_notify(2.4b1(13),2.4c1(15) epoch 11716) -21> 2014-09-18 18:08:37.754804 7f4c9f251700 5 -- op tracker -- , seq: 86410, time: 2014-09-18 18:08:37.754802, event: done, op: pg_notify(2.4b1(13),2.4c1(15) epoch 11716) -20> 2014-09-18 18:08:37.754900 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3277 ==== pg_info(1 pgs e11716:2.4c0(108)) v4 ==== 6814+0+0 (3118287842 0 0) 0xfae81c0 con 0x1395ec60 -19> 2014-09-18 18:08:37.754910 7f4c9f251700 5 -- op tracker -- , seq: 86411, time: 2014-09-18 18:08:37.754774, event: header_read, op: pg_info(1 pgs e11716:2.4c0(108)) -18> 2014-09-18 18:08:37.754916 7f4c9f251700 5 -- op tracker -- , seq: 86411, time: 2014-09-18 18:08:37.754774, event: throttled, op: pg_info(1 pgs e11716:2.4c0(108)) -17> 2014-09-18 18:08:37.754920 7f4c9f251700 5 -- op tracker -- , seq: 86411, time: 2014-09-18 18:08:37.754889, event: all_read, op: pg_info(1 pgs e11716:2.4c0(108)) -16> 2014-09-18 18:08:37.754924 7f4c9f251700 5 -- op tracker -- , seq: 86411, time: 2014-09-18 18:08:37.754907, event: dispatched, op: pg_info(1 pgs e11716:2.4c0(108)) -15> 2014-09-18 18:08:37.754928 7f4c9f251700 5 -- op tracker -- , seq: 86411, time: 2014-09-18 18:08:37.754927, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.4c0(108)) -14> 2014-09-18 18:08:37.754933 7f4c9f251700 5 -- op tracker -- , seq: 86411, time: 2014-09-18 18:08:37.754932, event: started, op: pg_info(1 pgs e11716:2.4c0(108)) -13> 2014-09-18 18:08:37.754947 7f4c9f251700 5 -- op tracker -- , seq: 86411, time: 2014-09-18 18:08:37.754946, event: done, op: pg_info(1 pgs e11716:2.4c0(108)) -12> 2014-09-18 18:08:37.754953 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] exit Started/Stray 1.059426 7 0.000549 -11> 2014-09-18 18:08:37.754971 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive -10> 2014-09-18 18:08:37.754980 7f4c97241700 5 osd.42 pg_epoch: 11716 pg[2.4c0( v 6835'27134 (4119'23725,6835'27134] local-les=11491 n=496 ec=188 les/c 11491/10727 11715/11715/11490) [32,42,3]/[9,42] r=1 lpr=11715 pi=10726-11714/9 crt=6835'27134 lcod 0'0 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -9> 2014-09-18 18:08:37.756149 7f4ca4a20700 1 -- 10.242.42.178:6807/2483 --> 10.242.42.166:6826/24959 -- pg_info(1 pgs e11716:2.4c0) v4 -- ?+0 0xde5e960 con 0x1395ec60 -8> 2014-09-18 18:08:37.799686 7f4c9f251700 1 -- 10.242.42.178:6807/2483 <== osd.9 10.242.42.166:6826/24959 3278 ==== pg_info(1 pgs e11716:2.204) v4 ==== 738+0+0 (1073708628 0 0) 0xde5e960 con 0x1395ec60 -7> 2014-09-18 18:08:37.799710 7f4c9f251700 5 -- op tracker -- , seq: 86412, time: 2014-09-18 18:08:37.799598, event: header_read, op: pg_info(1 pgs e11716:2.204) -6> 2014-09-18 18:08:37.799728 7f4c9f251700 5 -- op tracker -- , seq: 86412, time: 2014-09-18 18:08:37.799600, event: throttled, op: pg_info(1 pgs e11716:2.204) -5> 2014-09-18 18:08:37.799732 7f4c9f251700 5 -- op tracker -- , seq: 86412, time: 2014-09-18 18:08:37.799633, event: all_read, op: pg_info(1 pgs e11716:2.204) -4> 2014-09-18 18:08:37.799736 7f4c9f251700 5 -- op tracker -- , seq: 86412, time: 2014-09-18 18:08:37.799703, event: dispatched, op: pg_info(1 pgs e11716:2.204) -3> 2014-09-18 18:08:37.799741 7f4c9f251700 5 -- op tracker -- , seq: 86412, time: 2014-09-18 18:08:37.799739, event: waiting_for_osdmap, op: pg_info(1 pgs e11716:2.204) -2> 2014-09-18 18:08:37.799757 7f4c9f251700 5 -- op tracker -- , seq: 86412, time: 2014-09-18 18:08:37.799756, event: started, op: pg_info(1 pgs e11716:2.204) -1> 2014-09-18 18:08:37.799779 7f4c9f251700 5 -- op tracker -- , seq: 86412, time: 2014-09-18 18:08:37.799777, event: done, op: pg_info(1 pgs e11716:2.204) 0> 2014-09-18 18:08:37.801977 7f4c96a40700 -1 *** Caught signal (Aborted) ** in thread 7f4c96a40700 ceph version 0.84-sd-sprint4 (3215c520e1306f50d0094b5646636c02456c9df4) 1: /usr/bin/ceph-osd() [0xa8c7ff] 2: (()+0x10340) [0x7f4cb21e2340] 3: (gsignal()+0x39) [0x7f4cb04fbf79] 4: (abort()+0x148) [0x7f4cb04ff388] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f4cb0e076b5] 6: (()+0x5e836) [0x7f4cb0e05836] 7: (()+0x5e863) [0x7f4cb0e05863] 8: (()+0x5eaa2) [0x7f4cb0e05aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xb778c8] 10: (PG::RecoveryState::Stray::react(PG::MLogRec const&)+0x1f9) [0x7e1d19] 11: (boost::statechart::simple_state, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x1f4) [0x81a944] 12: (boost::statechart::state_machine, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x805d6b] 13: (PG::handle_peering_event(std::tr1::shared_ptr, PG::RecoveryCtx*)+0x1ce) [0x7b5b8e] 14: (OSD::process_peering_events(std::list > const&, ThreadPool::TPHandle&)+0x2b0) [0x6a3140] 15: (OSD::PeeringWQ::_process(std::list > const&, ThreadPool::TPHandle&)+0x18) [0x6f83c8] 16: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa46) [0xb68866] 17: (ThreadPool::WorkThread::entry()+0x10) [0xb69910] 18: (()+0x8182) [0x7f4cb21da182] 19: (clone()+0x6d) [0x7f4cb05c030d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.42.log --- end dump of recent events --- 2014-09-18 18:08:38.458692 7faa5dabc900 0 ceph version 0.84-sd-sprint4 (3215c520e1306f50d0094b5646636c02456c9df4), process ceph-osd, pid 10205 2014-09-18 18:08:38.460375 7faa5dabc900 0 filestore(/var/lib/ceph/osd/ceph-42) backend xfs (magic 0x58465342) 2014-09-18 18:08:38.460386 7faa5dabc900 1 filestore(/var/lib/ceph/osd/ceph-42) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2014-09-18 18:08:38.461795 7faa5dabc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-42) detect_features: FIEMAP ioctl is supported and appears to work 2014-09-18 18:08:38.461804 7faa5dabc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-42) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-09-18 18:08:38.464650 7faa5dabc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-42) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2014-09-18 18:08:38.464756 7faa5dabc900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-42) detect_feature: extsize is disabled by conf 2014-09-18 18:08:38.532979 7faa5dabc900 0 filestore(/var/lib/ceph/osd/ceph-42) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2014-09-18 18:08:38.549176 7faa5dabc900 1 journal _open /var/lib/ceph/osd/ceph-42/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2014-09-18 18:08:38.555637 7faa5dabc900 1 journal _open /var/lib/ceph/osd/ceph-42/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2014-09-18 18:08:38.559441 7faa5dabc900 1 journal close /var/lib/ceph/osd/ceph-42/journal 2014-09-18 18:08:38.566863 7faa5dabc900 0 filestore(/var/lib/ceph/osd/ceph-42) backend xfs (magic 0x58465342) 2014-09-18 18:08:38.568789 7faa5dabc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-42) detect_features: FIEMAP ioctl is supported and appears to work 2014-09-18 18:08:38.568799 7faa5dabc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-42) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-09-18 18:08:38.570602 7faa5dabc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-42) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2014-09-18 18:08:38.570642 7faa5dabc900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-42) detect_feature: extsize is disabled by conf 2014-09-18 18:08:38.573086 7faa5dabc900 0 filestore(/var/lib/ceph/osd/ceph-42) mount: WRITEAHEAD journal mode explicitly enabled in conf 2014-09-18 18:08:38.575898 7faa5dabc900 1 journal _open /var/lib/ceph/osd/ceph-42/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2014-09-18 18:08:38.578169 7faa5dabc900 1 journal _open /var/lib/ceph/osd/ceph-42/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2014-09-18 18:08:38.581027 7faa5dabc900 0 cls/hello/cls_hello.cc:271: loading cls_hello 2014-09-18 18:08:38.583071 7faa5dabc900 0 osd.42 11716 crush map has features 1073741824, adjusting msgr requires for clients 2014-09-18 18:08:38.583078 7faa5dabc900 0 osd.42 11716 crush map has features 1073741824 was 8705, adjusting msgr requires for mons 2014-09-18 18:08:38.583079 7faa5dabc900 0 osd.42 11716 crush map has features 1073741824, adjusting msgr requires for osds 2014-09-18 18:08:38.583102 7faa5dabc900 0 osd.42 11716 load_pgs 2014-09-18 18:08:39.675908 7faa5dabc900 0 osd.42 11716 load_pgs opened 126 pgs 2014-09-18 18:08:39.680908 7faa4b21a700 0 osd.42 11716 ignoring osdmap until we have initialized 2014-09-18 18:08:39.681499 7faa4b21a700 0 osd.42 11716 ignoring osdmap until we have initialized 2014-09-18 18:08:39.827829 7faa5dabc900 0 osd.42 11716 done with init, starting boot process 2014-09-18 18:08:42.692512 7faa41a07700 0 log_channel(default) log [INF] : 2.376 restarting backfill on osd.24 from (0'0,0'0] MAX to 6772'25996 2014-09-18 18:08:42.984303 7faa3169e700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6887/40306 pipe(0x128698c0 sd=114 :6814 s=2 pgs=38 cs=1 l=0 c=0x127ef860).fault with nothing to send, going to standby 2014-09-18 18:08:42.984499 7faa251da700 0 -- 10.242.42.178:0/10205 >> 10.242.42.164:6888/40306 pipe(0x139b03c0 sd=219 :0 s=1 pgs=0 cs=0 l=1 c=0x13352000).fault 2014-09-18 18:08:42.984571 7faa250d9700 0 -- 10.242.42.178:0/10205 >> 10.242.42.164:6889/40306 pipe(0x139afe40 sd=226 :0 s=1 pgs=0 cs=0 l=1 c=0x133522c0).fault 2014-09-18 18:08:44.823559 7faa41206700 0 log_channel(default) log [INF] : 2.2d8 restarting backfill on osd.2 from (0'0,0'0] MAX to 6825'27712 2014-09-18 18:08:51.684889 7faa41206700 0 log_channel(default) log [INF] : 2.4d0 restarting backfill on osd.55 from (0'0,0'0] MAX to 6699'27238 2014-09-18 18:08:52.361334 7faa41a07700 0 log_channel(default) log [INF] : 2.17e restarting backfill on osd.27 from (0'0,0'0] MAX to 6789'26580 2014-09-18 18:08:52.430036 7faa41a07700 0 log_channel(default) log [INF] : 2.7fa restarting backfill on osd.4 from (0'0,0'0] MAX to 6761'32700 2014-09-18 18:08:52.529845 7faa41a07700 0 log_channel(default) log [INF] : 2.403 restarting backfill on osd.4 from (0'0,0'0] MAX to 6774'31098 2014-09-18 18:08:52.530387 7faa41a07700 0 log_channel(default) log [INF] : 2.403 restarting backfill on osd.58 from (0'0,0'0] MAX to 6774'31098 2014-09-18 18:08:53.814412 7faa41a07700 0 log_channel(default) log [INF] : 2.3d5 restarting backfill on osd.24 from (0'0,0'0] MAX to 6827'28556 2014-09-18 18:13:11.774017 7faa41a07700 0 log_channel(default) log [INF] : 2.3d5 restarting backfill on osd.6 from (0'0,0'0] MAX to 6827'28556 2014-09-18 18:13:11.780176 7faa41a07700 0 log_channel(default) log [INF] : 2.17e restarting backfill on osd.9 from (0'0,0'0] MAX to 6789'26580 2014-09-18 18:13:19.853643 7faa3b1fa700 0 log_channel(default) log [INF] : 2.179 scrub ok 2014-09-18 18:13:23.956373 7faa41206700 0 log_channel(default) log [INF] : 2.11b restarting backfill on osd.5 from (0'0,0'0] MAX to 6775'28078 2014-09-18 18:14:34.511648 7faa3b1fa700 0 log_channel(default) log [INF] : 2.6b3 scrub ok 2014-09-18 18:18:43.027039 7faa3169e700 0 -- 10.242.42.178:0/10205 >> 10.242.42.164:6824/20623 pipe(0x13840940 sd=247 :0 s=1 pgs=0 cs=0 l=1 c=0x13221700).fault 2014-09-18 18:18:43.027089 7faa267f0700 0 -- 10.242.42.178:0/10205 >> 10.242.42.164:6825/20623 pipe(0x186cd180 sd=261 :0 s=1 pgs=0 cs=0 l=1 c=0x13e8dfa0).fault 2014-09-18 18:18:43.027137 7faa2a931700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6823/20623 pipe(0x12cff080 sd=172 :45814 s=2 pgs=235 cs=1 l=0 c=0x12da5fa0).fault with nothing to send, going to standby 2014-09-18 18:18:45.871357 7faa35edf700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.166:6808/22481 pipe(0x5602940 sd=50 :44348 s=2 pgs=590 cs=1 l=0 c=0x12740160).fault with nothing to send, going to standby 2014-09-18 18:18:45.871661 7faa2d159700 0 -- 10.242.42.178:0/10205 >> 10.242.42.166:6810/22481 pipe(0x186cab00 sd=28 :0 s=1 pgs=0 cs=0 l=1 c=0x13e8de40).fault 2014-09-18 18:18:45.871882 7faa2e56d700 0 -- 10.242.42.178:0/10205 >> 10.242.42.166:6813/22481 pipe(0x138403c0 sd=146 :0 s=1 pgs=0 cs=0 l=1 c=0x12c41440).fault 2014-09-18 18:18:51.846223 7faa2aa32700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6823/20623 pipe(0x12cff080 sd=48 :45814 s=1 pgs=235 cs=2 l=0 c=0x12da5fa0).fault 2014-09-18 18:18:54.609005 7faa3b1fa700 0 log_channel(default) log [INF] : 2.186 scrub ok 2014-09-18 18:23:39.871331 7faa347c8700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.170:6808/2333 pipe(0x127958c0 sd=63 :45647 s=2 pgs=13357 cs=1 l=0 c=0x12798840).fault with nothing to send, going to standby 2014-09-18 18:23:41.702754 7faa32eb6700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.170:6818/2363 pipe(0x12796c00 sd=102 :6814 s=2 pgs=8087 cs=1 l=0 c=0x127ee260).fault with nothing to send, going to standby 2014-09-18 18:23:41.709277 7faa31ca4700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.178:6804/2488 pipe(0x12869b80 sd=113 :6814 s=2 pgs=4157 cs=1 l=0 c=0x127ef9c0).fault with nothing to send, going to standby 2014-09-18 18:23:41.709305 7faa2b33b700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.178:6836/3187 pipe(0x12794dc0 sd=167 :43708 s=2 pgs=9453 cs=1 l=0 c=0x12b23180).fault with nothing to send, going to standby 2014-09-18 18:23:46.876177 7faa34ecf700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.168:6804/17298 pipe(0x5602c00 sd=51 :37000 s=2 pgs=240 cs=1 l=0 c=0x12767440).fault with nothing to send, going to standby 2014-09-18 18:24:04.951091 7faa3b1fa700 0 log_channel(default) log [INF] : 2.763 scrub ok 2014-09-18 18:28:54.211711 7faa256df700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.178:6841/4511 pipe(0x139b1700 sd=214 :42305 s=2 pgs=309 cs=1 l=0 c=0x1366a000).fault with nothing to send, going to standby 2014-09-18 18:30:43.532508 7faa367e8700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.168:6809/18319 pipe(0x5601080 sd=40 :45522 s=2 pgs=167 cs=1 l=0 c=0x114c07e0).fault with nothing to send, going to standby 2014-09-18 18:31:24.507329 7faa338b9700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.178:6840/4508 pipe(0x12796940 sd=93 :48878 s=2 pgs=299 cs=1 l=0 c=0x127ecb00).fault with nothing to send, going to standby 2014-09-18 18:33:30.207180 7faa33dbe700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.168:6825/16848 pipe(0x127942c0 sd=76 :56313 s=2 pgs=222 cs=1 l=0 c=0x5851e40).fault with nothing to send, going to standby 2014-09-18 18:33:34.438276 7faa359da700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.168:6814/16237 pipe(0x5601b80 sd=35 :40163 s=2 pgs=223 cs=1 l=0 c=0x127602c0).fault with nothing to send, going to standby 2014-09-18 18:34:02.009048 7faa348c9700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6816/2660 pipe(0x5603700 sd=79 :56618 s=2 pgs=9570 cs=1 l=0 c=0x12798f20).fault with nothing to send, going to standby 2014-09-18 18:36:54.957962 7faa341c2700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.168:6845/1010695 pipe(0x12794580 sd=39 :56730 s=2 pgs=81887 cs=1 l=0 c=0x127a2100).fault with nothing to send, going to standby 2014-09-18 18:38:35.013083 7faa362e3700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.178:6817/7086 pipe(0x5601e40 sd=45 :36671 s=2 pgs=191 cs=1 l=0 c=0x114c0520).fault with nothing to send, going to standby 2014-09-18 18:40:46.983188 7faa31ea6700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6836/27583 pipe(0x12869340 sd=111 :6814 s=2 pgs=45 cs=1 l=0 c=0x127ef5a0).fault with nothing to send, going to standby 2014-09-18 18:41:07.562625 7faa32fb7700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.168:6802/15056 pipe(0x12796680 sd=83 :56067 s=2 pgs=312 cs=1 l=0 c=0x127ec9a0).fault with nothing to send, going to standby 2014-09-18 18:41:47.695522 7faa343c4700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.178:6819/4507 pipe(0x12795600 sd=84 :43986 s=2 pgs=296 cs=1 l=0 c=0x127a1080).fault with nothing to send, going to standby 2014-09-18 18:41:47.699293 7faa36ced700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6811/17421 pipe(0x5600dc0 sd=26 :59143 s=2 pgs=751 cs=1 l=0 c=0x114bedc0).fault with nothing to send, going to standby 2014-09-18 18:41:47.701021 7faa31da5700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.178:6803/2487 pipe(0x12869080 sd=112 :6814 s=2 pgs=8402 cs=1 l=0 c=0x127ef440).fault with nothing to send, going to standby 2014-09-18 18:41:47.701990 7faa345c6700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.170:6816/2318 pipe(0x12794840 sd=67 :41225 s=2 pgs=4022 cs=1 l=0 c=0x12799760).fault with nothing to send, going to standby 2014-09-18 18:41:47.701995 7faa35bdc700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6819/11463 pipe(0x56023c0 sd=56 :42033 s=2 pgs=13272 cs=1 l=0 c=0x12741b80).fault with nothing to send, going to standby 2014-09-18 18:41:47.704474 7faa349ca700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6815/37668 pipe(0x12794b00 sd=58 :33688 s=2 pgs=946 cs=1 l=0 c=0x12767c80).fault with nothing to send, going to standby 2014-09-18 18:41:47.705704 7faa34ccd700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6802/5612 pipe(0x56039c0 sd=62 :57887 s=2 pgs=10355 cs=1 l=0 c=0x127651e0).fault with nothing to send, going to standby 2014-09-18 18:41:47.708947 7faa2c951700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.166:6801/43534 pipe(0x186ca580 sd=255 :6814 s=2 pgs=23 cs=1 l=0 c=0x140eb2e0).fault with nothing to send, going to standby 2014-09-18 18:41:47.709030 7faa351d2700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6829/19404 pipe(0x5603c80 sd=46 :47912 s=2 pgs=12097 cs=1 l=0 c=0x12765080).fault with nothing to send, going to standby 2014-09-18 18:41:47.724677 7faa355d6700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6837/36458 pipe(0x5603180 sd=69 :59570 s=2 pgs=570 cs=1 l=0 c=0x12761080).fault with nothing to send, going to standby 2014-09-18 18:41:47.775702 7faa339ba700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6831/25184 pipe(0x12794000 sd=61 :39416 s=2 pgs=57 cs=1 l=0 c=0x127ed4a0).fault with nothing to send, going to standby 2014-09-18 18:51:54.222518 7faa3b1fa700 0 log_channel(default) log [INF] : 2.16c scrub ok 2014-09-18 18:51:58.701482 7faa3b1fa700 0 log_channel(default) log [INF] : 2.4e8 scrub ok 2014-09-18 18:52:27.782402 7faa23986700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6811/17421 pipe(0x186cd700 sd=75 :6814 s=0 pgs=0 cs=0 l=0 c=0x114bfa20).accept connect_seq 2 vs existing 1 state standby 2014-09-18 18:53:12.063414 7faa3b1fa700 0 log_channel(default) log [INF] : 2.22d scrub ok 2014-09-18 18:53:16.186671 7faa3b1fa700 0 log_channel(default) log [INF] : 2.1d8 scrub ok 2014-09-18 18:54:59.260455 7faa3b1fa700 0 log_channel(default) log [INF] : 2.26d scrub ok 2014-09-18 18:55:40.811776 7faa3b1fa700 0 log_channel(default) log [INF] : 2.2ab scrub ok 2014-09-18 18:58:17.401596 7faa36ced700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6831/25184 pipe(0x1383fe40 sd=72 :6814 s=0 pgs=0 cs=0 l=0 c=0x13e8e680).accept connect_seq 2 vs existing 1 state standby 2014-09-18 19:06:54.345782 7faa35bdc700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6819/11463 pipe(0x56023c0 sd=23 :42701 s=2 pgs=13292 cs=3 l=0 c=0x12741b80).fault with nothing to send, going to standby 2014-09-18 19:07:28.522598 7faa23986700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6811/17421 pipe(0x186cd700 sd=75 :6814 s=2 pgs=806 cs=3 l=0 c=0x114bedc0).fault with nothing to send, going to standby 2014-09-18 19:08:16.308062 7faa351d2700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6829/19404 pipe(0x5603c80 sd=46 :48584 s=2 pgs=12125 cs=3 l=0 c=0x12765080).fault with nothing to send, going to standby 2014-09-18 19:13:17.790762 7faa36ced700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.164:6831/25184 pipe(0x1383fe40 sd=72 :6814 s=2 pgs=73 cs=3 l=0 c=0x127ed4a0).fault with nothing to send, going to standby 2014-09-18 19:30:48.009005 7faa3b1fa700 0 log_channel(default) log [INF] : 2.3d5 scrub ok 2014-09-18 19:31:53.498580 7faa3b1fa700 0 log_channel(default) log [INF] : 2.376 scrub ok 2014-09-18 19:32:25.166458 7faa3b1fa700 0 log_channel(default) log [INF] : 2.6e2 scrub ok 2014-09-18 19:45:48.047501 7faa2c951700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.166:6801/43534 pipe(0x186ca580 sd=219 :58261 s=2 pgs=70 cs=3 l=0 c=0x140eb2e0).fault with nothing to send, going to standby 2014-09-18 19:46:53.620421 7faa34ccd700 0 -- 10.242.42.178:6814/10205 >> 10.242.42.172:6802/5612 pipe(0x56039c0 sd=18 :58577 s=2 pgs=10383 cs=3 l=0 c=0x127651e0).fault with nothing to send, going to standby