-749> 2014-04-25 09:18:54.601820 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.178:6817/5052082 0x128fa500 -748> 2014-04-25 09:18:54.601933 7f77c8dc4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.178:6817/5052082 pipe(0x128fa500 sd=26 :44320 s=4 pgs=2398 cs=1 l=0 c=0x13320dc0).reader couldn't read tag, Success -747> 2014-04-25 09:18:54.601952 7f77c8dc4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.178:6817/5052082 pipe(0x128fa500 sd=26 :44320 s=4 pgs=2398 cs=1 l=0 c=0x13320dc0).fault 0: Success -746> 2014-04-25 09:18:54.601974 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.189:6929/4062575 0x1d942280 -745> 2014-04-25 09:18:54.602054 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.177:6972/4035079 0x12476500 -744> 2014-04-25 09:18:54.602151 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.189:6801/3059065 0x11c41680 -743> 2014-04-25 09:18:54.602193 7f77ba4dc700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.185:6911/2011575 pipe(0x130a7900 sd=35 :41862 s=4 pgs=4398 cs=8 l=0 c=0x132bc580).connect read reply Success -742> 2014-04-25 09:18:54.602223 7f77ba4dc700 3 -- 10.193.207.183:6889/6044082 >> 10.193.207.185:6911/2011575 pipe(0x130a7900 sd=35 :41862 s=4 pgs=4398 cs=8 l=0 c=0x132bc580).connect fault, but state = closed != connecting, stopping -741> 2014-04-25 09:18:54.602254 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.186:6978/3012241 0x1189b180 -740> 2014-04-25 09:18:54.602354 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.188:6891/3098206 0x10839900 -739> 2014-04-25 09:18:54.602468 7f77c4172700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6891/3098206 pipe(0x10839900 sd=561 :6889 s=4 pgs=1921 cs=1 l=0 c=0x128491e0).reader couldn't read tag, Success -738> 2014-04-25 09:18:54.602487 7f77c4172700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6891/3098206 pipe(0x10839900 sd=561 :6889 s=4 pgs=1921 cs=1 l=0 c=0x128491e0).fault 0: Success -737> 2014-04-25 09:18:54.602511 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.177:6941/4035543 0x11e18c80 -736> 2014-04-25 09:18:54.602601 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.186:6845/4101251 0x11949900 -735> 2014-04-25 09:18:54.602704 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.178:6830/6044299 0x19ca9680 -734> 2014-04-25 09:18:54.602811 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.187:6877/3075929 0x12d1a500 -733> 2014-04-25 09:18:54.602899 7f779042c700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6877/3075929 pipe(0x12d1a500 sd=382 :6889 s=4 pgs=12147 cs=1 l=0 c=0x13288f20).reader couldn't read tag, Success -732> 2014-04-25 09:18:54.602921 7f779042c700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6877/3075929 pipe(0x12d1a500 sd=382 :6889 s=4 pgs=12147 cs=1 l=0 c=0x13288f20).fault 0: Success -731> 2014-04-25 09:18:54.602943 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.179:6889/7002729 0x19ca9b80 -730> 2014-04-25 09:18:54.675347 7f77d1147700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6981/5102110 pipe(0x5fb6f00 sd=288 :59278 s=4 pgs=5100 cs=1 l=0 c=0x1220a000).reader couldn't read tag, Success -729> 2014-04-25 09:18:54.675388 7f77d1147700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6981/5102110 pipe(0x5fb6f00 sd=288 :59278 s=4 pgs=5100 cs=1 l=0 c=0x1220a000).fault 0: Success -728> 2014-04-25 09:18:54.696467 7f778fb2f700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.178:0/45468 pipe(0x19e42500 sd=849 :6919 s=2 pgs=2597 cs=1 l=1 c=0x12282840).reader couldn't read tag, Success -727> 2014-04-25 09:18:54.696518 7f778fb2f700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.178:0/45468 pipe(0x19e42500 sd=849 :6919 s=2 pgs=2597 cs=1 l=1 c=0x12282840).fault 0: Success -726> 2014-04-25 09:18:54.697968 7f77f68f0700 1 -- 10.193.207.183:6919/6044082 <== osd.105 10.193.207.179:0/97831 54 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:50.303195) v2 ==== 47+0+0 (1309489787 0 0) 0x12461180 con 0x12426000 -725> 2014-04-25 09:18:54.698009 7f77f68f0700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -724> 2014-04-25 09:18:54.698014 7f77f68f0700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -723> 2014-04-25 09:18:54.698025 7f77f68f0700 1 -- 10.193.207.183:6919/6044082 <== osd.105 10.193.207.179:0/97831 55 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:51.416582) v2 ==== 47+0+0 (766771467 0 0) 0x124616c0 con 0x12426000 -722> 2014-04-25 09:18:54.698038 7f77f68f0700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -721> 2014-04-25 09:18:54.698041 7f77f68f0700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -720> 2014-04-25 09:18:54.698046 7f77f68f0700 1 -- 10.193.207.183:6919/6044082 <== osd.105 10.193.207.179:0/97831 56 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:52.533197) v2 ==== 47+0+0 (1496554607 0 0) 0x18e93340 con 0x12426000 -719> 2014-04-25 09:18:54.698058 7f77f68f0700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -718> 2014-04-25 09:18:54.698061 7f77f68f0700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -717> 2014-04-25 09:18:54.700512 7f77a873d700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.186:0/98764 pipe(0x1d626780 sd=655 :6890 s=2 pgs=2763 cs=1 l=1 c=0x12849600).reader couldn't read tag, Success -716> 2014-04-25 09:18:54.700558 7f77a873d700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.186:0/98764 pipe(0x1d626780 sd=655 :6890 s=2 pgs=2763 cs=1 l=1 c=0x12849600).fault 0: Success -715> 2014-04-25 09:18:54.701015 7f77e2459700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.179:6889/7002729 pipe(0x19ca9b80 sd=590 :6889 s=4 pgs=3426 cs=1 l=0 c=0x114e5080).reader couldn't read tag, Success -714> 2014-04-25 09:18:54.701039 7f77e2459700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.179:6889/7002729 pipe(0x19ca9b80 sd=590 :6889 s=4 pgs=3426 cs=1 l=0 c=0x114e5080).fault 0: Success -713> 2014-04-25 09:18:54.702809 7f77c79b0700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.188:0/70327 pipe(0x3db9680 sd=398 :6919 s=2 pgs=1438 cs=1 l=1 c=0x120f0420).reader couldn't read tag, Success -712> 2014-04-25 09:18:54.702849 7f77c79b0700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.188:0/70327 pipe(0x3db9680 sd=398 :6919 s=2 pgs=1438 cs=1 l=1 c=0x120f0420).fault 0: Success -711> 2014-04-25 09:18:54.703281 7f77b6098700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6806/5096720 pipe(0x10838c80 sd=459 :6889 s=4 pgs=3297 cs=1 l=0 c=0x132e5600).reader couldn't read tag, Success -710> 2014-04-25 09:18:54.703304 7f77b6098700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6806/5096720 pipe(0x10838c80 sd=459 :6889 s=4 pgs=3297 cs=1 l=0 c=0x132e5600).fault 0: Success -709> 2014-04-25 09:18:54.703423 7f77d9011700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.189:0/76441 pipe(0x12dcea00 sd=40 :6919 s=2 pgs=7800 cs=1 l=1 c=0x133afa20).reader couldn't read tag, Success -708> 2014-04-25 09:18:54.703458 7f77d9011700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.189:0/76441 pipe(0x12dcea00 sd=40 :6919 s=2 pgs=7800 cs=1 l=1 c=0x133afa20).fault 0: Success -707> 2014-04-25 09:18:54.703731 7f77cca00700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.185:0/13515 pipe(0x12a4a280 sd=358 :6919 s=2 pgs=2832 cs=1 l=1 c=0x120b0160).reader couldn't read tag, Success -706> 2014-04-25 09:18:54.703779 7f77cca00700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.185:0/13515 pipe(0x12a4a280 sd=358 :6919 s=2 pgs=2832 cs=1 l=1 c=0x120b0160).fault 0: Success -705> 2014-04-25 09:18:54.703950 7f77e7cb8700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6825/3074806 pipe(0x11f7c780 sd=579 :6889 s=4 pgs=2073 cs=1 l=0 c=0x1220edc0).reader couldn't read tag, Success -704> 2014-04-25 09:18:54.703980 7f77e7cb8700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6825/3074806 pipe(0x11f7c780 sd=579 :6889 s=4 pgs=2073 cs=1 l=0 c=0x1220edc0).fault 0: Success -703> 2014-04-25 09:18:54.705399 7f77e3267700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6972/4035079 pipe(0x12476500 sd=130 :36078 s=4 pgs=5327 cs=1 l=0 c=0x123b3e40).reader couldn't read tag, Success -702> 2014-04-25 09:18:54.705422 7f77e3267700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6972/4035079 pipe(0x12476500 sd=130 :36078 s=4 pgs=5327 cs=1 l=0 c=0x123b3e40).fault 0: Success -701> 2014-04-25 09:18:54.706083 7f77e67a3700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.189:7025/1063733 pipe(0x11948f00 sd=117 :50581 s=4 pgs=6581 cs=1 l=0 c=0x1100ab00).reader couldn't read tag, Success -700> 2014-04-25 09:18:54.706108 7f77e67a3700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.189:7025/1063733 pipe(0x11948f00 sd=117 :50581 s=4 pgs=6581 cs=1 l=0 c=0x1100ab00).fault 0: Success -699> 2014-04-25 09:18:54.707403 7f77d6fa5700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6941/4035543 pipe(0x11e18c80 sd=252 :55578 s=4 pgs=4955 cs=1 l=0 c=0x12a39b80).reader couldn't read tag, Success -698> 2014-04-25 09:18:54.707427 7f77d6fa5700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6941/4035543 pipe(0x11e18c80 sd=252 :55578 s=4 pgs=4955 cs=1 l=0 c=0x12a39b80).fault 0: Success -697> 2014-04-25 09:18:54.710599 7f77c8fc6700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.180:6818/6070764 pipe(0x14279400 sd=385 :6889 s=4 pgs=3698 cs=1 l=0 c=0x1339ec60).reader couldn't read tag, Success -696> 2014-04-25 09:18:54.710629 7f77c8fc6700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.180:6818/6070764 pipe(0x14279400 sd=385 :6889 s=4 pgs=3698 cs=1 l=0 c=0x1339ec60).fault 0: Success -695> 2014-04-25 09:18:54.711411 7f77d356b700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.185:0/11295 pipe(0x126be780 sd=274 :6890 s=2 pgs=2021 cs=1 l=1 c=0x133af4a0).reader couldn't read tag, Success -694> 2014-04-25 09:18:54.711453 7f77d356b700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.185:0/11295 pipe(0x126be780 sd=274 :6890 s=2 pgs=2021 cs=1 l=1 c=0x133af4a0).fault 0: Success -693> 2014-04-25 09:18:54.712763 7f77dec6d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6802/5097877 pipe(0x10839680 sd=695 :6889 s=4 pgs=3426 cs=1 l=0 c=0x124271e0).reader couldn't read tag, Success -692> 2014-04-25 09:18:54.712787 7f77dec6d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6802/5097877 pipe(0x10839680 sd=695 :6889 s=4 pgs=3426 cs=1 l=0 c=0x124271e0).fault 0: Success -691> 2014-04-25 09:18:54.713613 7f77c1a4b700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6846/4059228 pipe(0x12404280 sd=404 :56732 s=4 pgs=1954 cs=1 l=0 c=0x132789a0).reader couldn't read tag, Success -690> 2014-04-25 09:18:54.713639 7f77c1a4b700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6846/4059228 pipe(0x12404280 sd=404 :56732 s=4 pgs=1954 cs=1 l=0 c=0x132789a0).fault 0: Success -689> 2014-04-25 09:18:54.714277 7f77dd354700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.189:6933/4059314 pipe(0x1d41d180 sd=448 :6889 s=4 pgs=9842 cs=1 l=0 c=0x123b5ce0).reader couldn't read tag, Success -688> 2014-04-25 09:18:54.714300 7f77dd354700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.189:6933/4059314 pipe(0x1d41d180 sd=448 :6889 s=4 pgs=9842 cs=1 l=0 c=0x123b5ce0).fault 0: Success -687> 2014-04-25 09:18:54.721459 7f77d174d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:7010/3024320 pipe(0x13994500 sd=91 :48174 s=4 pgs=2624 cs=1 l=0 c=0x133a0b00).reader couldn't read tag, Success -686> 2014-04-25 09:18:54.721488 7f77d174d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:7010/3024320 pipe(0x13994500 sd=91 :48174 s=4 pgs=2624 cs=1 l=0 c=0x133a0b00).fault 0: Success -685> 2014-04-25 09:18:54.722228 7f77e396e700 2 -- 10.193.207.183:0/44082 >> 10.193.207.186:6931/3098032 pipe(0x12207400 sd=120 :47795 s=4 pgs=792 cs=1 l=1 c=0x1329cc60).reader couldn't read tag, Success -684> 2014-04-25 09:18:54.722250 7f77e396e700 2 -- 10.193.207.183:0/44082 >> 10.193.207.186:6931/3098032 pipe(0x12207400 sd=120 :47795 s=4 pgs=792 cs=1 l=1 c=0x1329cc60).fault 0: Success -683> 2014-04-25 09:18:54.724921 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.187:6950/1075529 0x1195bb80 -682> 2014-04-25 09:18:54.725067 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.179:6817/6097831 0x13995400 -681> 2014-04-25 09:18:54.725209 7f77ce91f700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.179:6817/6097831 pipe(0x13995400 sd=338 :6889 s=4 pgs=3366 cs=1 l=0 c=0x132bb600).reader couldn't read tag, Success -680> 2014-04-25 09:18:54.725232 7f77ce91f700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.179:6817/6097831 pipe(0x13995400 sd=338 :6889 s=4 pgs=3366 cs=1 l=0 c=0x132bb600).fault 0: Success -679> 2014-04-25 09:18:54.725255 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.176:7015/4056013 0x11ecc000 -678> 2014-04-25 09:18:54.725380 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.178:6875/6047055 0x14020a00 -677> 2014-04-25 09:18:54.725509 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.176:7005/6056853 0x12404000 -676> 2014-04-25 09:18:54.725622 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.178:6862/6046495 0x11c41900 -675> 2014-04-25 09:18:54.725741 7f77d5f95700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.178:6862/6046495 pipe(0x11c41900 sd=136 :39179 s=4 pgs=3502 cs=3 l=0 c=0x17f246e0).reader couldn't read tag, Success -674> 2014-04-25 09:18:54.725866 7f77d5f95700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.178:6862/6046495 pipe(0x11c41900 sd=136 :39179 s=4 pgs=3502 cs=3 l=0 c=0x17f246e0).fault 0: Success -673> 2014-04-25 09:18:54.725888 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.188:7004/2097122 0x11e19900 -672> 2014-04-25 09:18:54.726013 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.187:6899/2041222 0x126be500 -671> 2014-04-25 09:18:54.726109 7f7799dd7700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6899/2041222 pipe(0x126be500 sd=426 :6889 s=4 pgs=6101 cs=1 l=0 c=0x132d7340).reader couldn't read tag, Success -670> 2014-04-25 09:18:54.726128 7f7799dd7700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6899/2041222 pipe(0x126be500 sd=426 :6889 s=4 pgs=6101 cs=1 l=0 c=0x132d7340).fault 0: Success -669> 2014-04-25 09:18:54.726149 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.177:6809/3034909 0x14a94c80 -668> 2014-04-25 09:18:54.726276 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.176:6950/5057883 0x1095e500 -667> 2014-04-25 09:18:54.726380 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.187:6935/1078188 0x11ef6c80 -666> 2014-04-25 09:18:54.726492 7f77dd657700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6935/1078188 pipe(0x11ef6c80 sd=199 :50665 s=4 pgs=5630 cs=3 l=0 c=0x122729a0).reader couldn't read tag, Success -665> 2014-04-25 09:18:54.726537 7f77dd657700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6935/1078188 pipe(0x11ef6c80 sd=199 :50665 s=4 pgs=5630 cs=3 l=0 c=0x122729a0).fault 0: Success -664> 2014-04-25 09:18:54.726560 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.187:6960/1043859 0x11948c80 -663> 2014-04-25 09:18:54.726667 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.184:7072/6064447 0x11e18a00 -662> 2014-04-25 09:18:54.726814 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.188:6833/2092997 0x14020c80 -661> 2014-04-25 09:18:54.726918 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.179:6862/2099876 0x11948a00 -660> 2014-04-25 09:18:54.727096 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.178:7037/5044455 0x12d1a780 -659> 2014-04-25 09:18:54.727225 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.176:6947/7059458 0x1195a280 -658> 2014-04-25 09:18:54.727341 7f77f9af5700 5 -- 10.193.207.183:6889/6044082 mark_down_all 10.193.207.189:6962/3065233 0x1d942000 -657> 2014-04-25 09:18:54.727432 7f77af6ac700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.189:6962/3065233 pipe(0x1d942000 sd=356 :6889 s=4 pgs=12115 cs=1 l=0 c=0x123b5600).reader couldn't read tag, Success -656> 2014-04-25 09:18:54.727453 7f77af6ac700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.189:6962/3065233 pipe(0x1d942000 sd=356 :6889 s=4 pgs=12115 cs=1 l=0 c=0x123b5600).fault 0: Success -655> 2014-04-25 09:18:54.730948 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,171,448]/[210,171,448] -654> 2014-04-25 09:18:54.730993 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -653> 2014-04-25 09:18:54.731008 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -652> 2014-04-25 09:18:54.731021 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -651> 2014-04-25 09:18:54.731032 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -650> 2014-04-25 09:18:54.731044 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -649> 2014-04-25 09:18:54.731055 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -648> 2014-04-25 09:18:54.731065 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -647> 2014-04-25 09:18:54.731078 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] take_waiters -646> 2014-04-25 09:18:54.731090 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.25d1( v 1156'1524 (0'0,1156'1524] local-les=3782 n=1524 ec=638 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -645> 2014-04-25 09:18:54.731105 7f77ef0e4700 10 log is not dirty -644> 2014-04-25 09:18:54.731147 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,171,448]/[210,171,448] -643> 2014-04-25 09:18:54.731183 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -642> 2014-04-25 09:18:54.731195 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -641> 2014-04-25 09:18:54.731206 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -640> 2014-04-25 09:18:54.731217 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -639> 2014-04-25 09:18:54.731228 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -638> 2014-04-25 09:18:54.731238 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -637> 2014-04-25 09:18:54.731249 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -636> 2014-04-25 09:18:54.731261 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] take_waiters -635> 2014-04-25 09:18:54.731272 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.25d2( v 1110'672 (0'0,1110'672] local-les=3782 n=672 ec=633 les/c 3782/3783 3768/3768/3753) [210,171,448] r=0 lpr=3768 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -634> 2014-04-25 09:18:54.731285 7f77ef0e4700 10 log is not dirty -633> 2014-04-25 09:18:54.731303 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.4cdf( v 1110'1450 (0'0,1110'1450] local-les=3755 n=1450 ec=638 les/c 3755/3145 3753/3753/3753) [105,210,169] r=1 lpr=3753 pi=3144-3752/10 luod=0'0 lcod 0'0 active] handle_advance_map [105,210,169]/[105,210,169] -632> 2014-04-25 09:18:54.731322 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4cdf( v 1110'1450 (0'0,1110'1450] local-les=3755 n=1450 ec=638 les/c 3755/3145 3753/3753/3753) [105,210,169] r=1 lpr=3753 pi=3144-3752/10 luod=0'0 lcod 0'0 active] state: Started advmap -631> 2014-04-25 09:18:54.731334 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4cdf( v 1110'1450 (0'0,1110'1450] local-les=3755 n=1450 ec=638 les/c 3755/3145 3753/3753/3753) [105,210,169] r=1 lpr=3753 pi=3144-3752/10 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -630> 2014-04-25 09:18:54.731347 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4cdf( v 1110'1450 (0'0,1110'1450] local-les=3755 n=1450 ec=638 les/c 3755/3145 3753/3753/3753) [105,210,169] r=1 lpr=3753 pi=3144-3752/10 luod=0'0 lcod 0'0 active] handle_activate_map -629> 2014-04-25 09:18:54.731358 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4cdf( v 1110'1450 (0'0,1110'1450] local-les=3755 n=1450 ec=638 les/c 3755/3145 3753/3753/3753) [105,210,169] r=1 lpr=3753 pi=3144-3752/10 luod=0'0 lcod 0'0 active] take_waiters -628> 2014-04-25 09:18:54.731369 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4cdf( v 1110'1450 (0'0,1110'1450] local-les=3755 n=1450 ec=638 les/c 3755/3145 3753/3753/3753) [105,210,169] r=1 lpr=3753 pi=3144-3752/10 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -627> 2014-04-25 09:18:54.731382 7f77ef0e4700 10 log is not dirty -626> 2014-04-25 09:18:54.731399 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.3ace( v 1110'744 (0'0,1110'744] local-les=3755 n=744 ec=633 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] handle_advance_map [235,210]/[235,210] -625> 2014-04-25 09:18:54.731415 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3ace( v 1110'744 (0'0,1110'744] local-les=3755 n=744 ec=633 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] state: Started advmap -624> 2014-04-25 09:18:54.731426 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3ace( v 1110'744 (0'0,1110'744] local-les=3755 n=744 ec=633 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] check_recovery_sources no source osds () went down -623> 2014-04-25 09:18:54.731437 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3ace( v 1110'744 (0'0,1110'744] local-les=3755 n=744 ec=633 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] handle_activate_map -622> 2014-04-25 09:18:54.731466 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3ace( v 1110'744 (0'0,1110'744] local-les=3755 n=744 ec=633 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] take_waiters -621> 2014-04-25 09:18:54.731478 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3ace( v 1110'744 (0'0,1110'744] local-les=3755 n=744 ec=633 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -620> 2014-04-25 09:18:54.731491 7f77ef0e4700 10 log is not dirty -619> 2014-04-25 09:18:54.731531 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.4fd0( v 1110'689 (0'0,1110'689] local-les=3785 n=689 ec=633 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] handle_advance_map [86,210,1]/[86,210,1] -618> 2014-04-25 09:18:54.731551 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4fd0( v 1110'689 (0'0,1110'689] local-les=3785 n=689 ec=633 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] state: Started advmap -617> 2014-04-25 09:18:54.731563 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4fd0( v 1110'689 (0'0,1110'689] local-les=3785 n=689 ec=633 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -616> 2014-04-25 09:18:54.731575 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4fd0( v 1110'689 (0'0,1110'689] local-les=3785 n=689 ec=633 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] handle_activate_map -615> 2014-04-25 09:18:54.731585 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4fd0( v 1110'689 (0'0,1110'689] local-les=3785 n=689 ec=633 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] take_waiters -614> 2014-04-25 09:18:54.731597 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4fd0( v 1110'689 (0'0,1110'689] local-les=3785 n=689 ec=633 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -613> 2014-04-25 09:18:54.731612 7f77ef0e4700 10 log is not dirty -612> 2014-04-25 09:18:54.731645 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.3acd( v 1110'1380 (0'0,1110'1380] local-les=3755 n=1380 ec=638 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] handle_advance_map [235,210]/[235,210] -611> 2014-04-25 09:18:54.731663 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3acd( v 1110'1380 (0'0,1110'1380] local-les=3755 n=1380 ec=638 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] state: Started advmap -610> 2014-04-25 09:18:54.731675 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3acd( v 1110'1380 (0'0,1110'1380] local-les=3755 n=1380 ec=638 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] check_recovery_sources no source osds () went down -609> 2014-04-25 09:18:54.731686 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3acd( v 1110'1380 (0'0,1110'1380] local-les=3755 n=1380 ec=638 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] handle_activate_map -608> 2014-04-25 09:18:54.731710 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3acd( v 1110'1380 (0'0,1110'1380] local-les=3755 n=1380 ec=638 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] take_waiters -607> 2014-04-25 09:18:54.731721 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3acd( v 1110'1380 (0'0,1110'1380] local-les=3755 n=1380 ec=638 les/c 3755/3718 3798/3798/3798) [235,210] r=1 lpr=3798 pi=3119-3797/11 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -606> 2014-04-25 09:18:54.731735 7f77ef0e4700 10 log is not dirty -605> 2014-04-25 09:18:54.731752 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,271,241]/[210,271,241] -604> 2014-04-25 09:18:54.731768 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -603> 2014-04-25 09:18:54.731780 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -602> 2014-04-25 09:18:54.731791 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -601> 2014-04-25 09:18:54.731800 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -600> 2014-04-25 09:18:54.731810 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -599> 2014-04-25 09:18:54.731820 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -598> 2014-04-25 09:18:54.731830 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -597> 2014-04-25 09:18:54.731842 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -596> 2014-04-25 09:18:54.731853 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.34c2( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -595> 2014-04-25 09:18:54.731865 7f77ef0e4700 10 log is not dirty -594> 2014-04-25 09:18:54.731878 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.4fcf( v 1238'1513 (0'0,1238'1513] local-les=3785 n=1513 ec=638 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] handle_advance_map [86,210,1]/[86,210,1] -593> 2014-04-25 09:18:54.731895 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4fcf( v 1238'1513 (0'0,1238'1513] local-les=3785 n=1513 ec=638 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] state: Started advmap -592> 2014-04-25 09:18:54.731906 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4fcf( v 1238'1513 (0'0,1238'1513] local-les=3785 n=1513 ec=638 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -591> 2014-04-25 09:18:54.731918 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4fcf( v 1238'1513 (0'0,1238'1513] local-les=3785 n=1513 ec=638 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] handle_activate_map -590> 2014-04-25 09:18:54.731929 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4fcf( v 1238'1513 (0'0,1238'1513] local-les=3785 n=1513 ec=638 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] take_waiters -589> 2014-04-25 09:18:54.731940 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4fcf( v 1238'1513 (0'0,1238'1513] local-les=3785 n=1513 ec=638 les/c 3785/3786 3782/3782/3782) [86,210,1] r=1 lpr=3782 pi=3764-3781/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -588> 2014-04-25 09:18:54.731954 7f77ef0e4700 10 log is not dirty -587> 2014-04-25 09:18:54.731970 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,43,441]/[210,43,441] -586> 2014-04-25 09:18:54.731986 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -585> 2014-04-25 09:18:54.731997 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -584> 2014-04-25 09:18:54.732007 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -583> 2014-04-25 09:18:54.732017 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -582> 2014-04-25 09:18:54.732027 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -581> 2014-04-25 09:18:54.732037 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -580> 2014-04-25 09:18:54.732047 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -579> 2014-04-25 09:18:54.732059 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -578> 2014-04-25 09:18:54.732070 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.52c8( v 1176'688 (0'0,1176'688] local-les=3754 n=688 ec=633 les/c 3754/3757 3753/3753/3753) [210,43,441] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -577> 2014-04-25 09:18:54.732084 7f77ef0e4700 10 log is not dirty -576> 2014-04-25 09:18:54.732122 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.2bba( v 1110'1370 (0'0,1110'1370] local-les=3777 n=1370 ec=638 les/c 3777/3782 3756/3756/3756) [292,56,210] r=2 lpr=3756 pi=638-3755/61 luod=0'0 lcod 0'0 active] handle_advance_map [292,56,210]/[292,56,210] -575> 2014-04-25 09:18:54.732142 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2bba( v 1110'1370 (0'0,1110'1370] local-les=3777 n=1370 ec=638 les/c 3777/3782 3756/3756/3756) [292,56,210] r=2 lpr=3756 pi=638-3755/61 luod=0'0 lcod 0'0 active] state: Started advmap -574> 2014-04-25 09:18:54.732155 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2bba( v 1110'1370 (0'0,1110'1370] local-les=3777 n=1370 ec=638 les/c 3777/3782 3756/3756/3756) [292,56,210] r=2 lpr=3756 pi=638-3755/61 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -573> 2014-04-25 09:18:54.732173 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2bba( v 1110'1370 (0'0,1110'1370] local-les=3777 n=1370 ec=638 les/c 3777/3782 3756/3756/3756) [292,56,210] r=2 lpr=3756 pi=638-3755/61 luod=0'0 lcod 0'0 active] handle_activate_map -572> 2014-04-25 09:18:54.732184 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2bba( v 1110'1370 (0'0,1110'1370] local-les=3777 n=1370 ec=638 les/c 3777/3782 3756/3756/3756) [292,56,210] r=2 lpr=3756 pi=638-3755/61 luod=0'0 lcod 0'0 active] take_waiters -571> 2014-04-25 09:18:54.732197 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2bba( v 1110'1370 (0'0,1110'1370] local-les=3777 n=1370 ec=638 les/c 3777/3782 3756/3756/3756) [292,56,210] r=2 lpr=3756 pi=638-3755/61 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -570> 2014-04-25 09:18:54.732210 7f77ef0e4700 10 log is not dirty -569> 2014-04-25 09:18:54.732248 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.46c5( v 1204'707 (0'0,1204'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3391) [32,251,210] r=2 lpr=3753 pi=633-3752/58 luod=0'0 lcod 0'0 active] handle_advance_map [32,251,210]/[32,251,210] -568> 2014-04-25 09:18:54.732268 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.46c5( v 1204'707 (0'0,1204'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3391) [32,251,210] r=2 lpr=3753 pi=633-3752/58 luod=0'0 lcod 0'0 active] state: Started advmap -567> 2014-04-25 09:18:54.732281 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.46c5( v 1204'707 (0'0,1204'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3391) [32,251,210] r=2 lpr=3753 pi=633-3752/58 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -566> 2014-04-25 09:18:54.732293 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.46c5( v 1204'707 (0'0,1204'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3391) [32,251,210] r=2 lpr=3753 pi=633-3752/58 luod=0'0 lcod 0'0 active] handle_activate_map -565> 2014-04-25 09:18:54.732304 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.46c5( v 1204'707 (0'0,1204'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3391) [32,251,210] r=2 lpr=3753 pi=633-3752/58 luod=0'0 lcod 0'0 active] take_waiters -564> 2014-04-25 09:18:54.732315 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.46c5( v 1204'707 (0'0,1204'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3391) [32,251,210] r=2 lpr=3753 pi=633-3752/58 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -563> 2014-04-25 09:18:54.732328 7f77ef0e4700 10 log is not dirty -562> 2014-04-25 09:18:54.732361 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,271,241]/[210,271,241] -561> 2014-04-25 09:18:54.732379 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -560> 2014-04-25 09:18:54.732391 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -559> 2014-04-25 09:18:54.732402 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -558> 2014-04-25 09:18:54.732412 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -557> 2014-04-25 09:18:54.732422 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -556> 2014-04-25 09:18:54.732432 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -555> 2014-04-25 09:18:54.732441 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -554> 2014-04-25 09:18:54.732453 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -553> 2014-04-25 09:18:54.732464 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.34c3( v 1110'753 (0'0,1110'753] local-les=3754 n=753 ec=633 les/c 3754/3754 3753/3753/3753) [210,271,241] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -552> 2014-04-25 09:18:54.732477 7f77ef0e4700 10 log is not dirty -551> 2014-04-25 09:18:54.732492 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3753 pi=1354-3752/28 luod=0'0 lcod 0'0 active] handle_advance_map [115,210]/[115,210] -550> 2014-04-25 09:18:54.732508 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3753 pi=1354-3752/28 luod=0'0 lcod 0'0 active] state: Started advmap -549> 2014-04-25 09:18:54.732518 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3753 pi=1354-3752/28 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -548> 2014-04-25 09:18:54.732532 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3753 pi=1354-3752/28 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 170.236589 129 0.002294 -547> 2014-04-25 09:18:54.732547 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3753 pi=1354-3752/28 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 170.236616 0 0.000000 -546> 2014-04-25 09:18:54.732561 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3753 pi=1354-3752/28 luod=0'0 lcod 0'0 active] exit Started 177.412084 0 0.000000 -545> 2014-04-25 09:18:54.732574 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3753 pi=1354-3752/28 luod=0'0 lcod 0'0 active] enter Reset -544> 2014-04-25 09:18:54.732586 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3799 pi=1354-3752/28 luod=0'0 lcod 0'0 active] state: Reset advmap -543> 2014-04-25 09:18:54.732596 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3799 pi=1354-3752/28 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 -542> 2014-04-25 09:18:54.732609 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3799 pi=1354-3752/28 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -541> 2014-04-25 09:18:54.732620 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [277,115,210] r=2 lpr=3799 pi=1354-3752/28 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -540> 2014-04-25 09:18:54.732642 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [115,210] r=1 lpr=3799 pi=1354-3798/29 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [277,115,210]/[277,115,210]): not rw, up_thru 3796 up_from 2916 last_epoch_clean 3715 generate_past_intervals interval(3753-3798 [277,115,210]/[277,115,210] maybe_went_rw) : primary up 2916-3796 includes interval -539> 2014-04-25 09:18:54.732655 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3753/3753/2916) [115,210] r=1 lpr=3799 pi=1354-3798/29 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [277,115,210]/[277,115,210] maybe_went_rw) -538> 2014-04-25 09:18:54.732666 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 luod=0'0 lcod 0'0 active] up [277,115,210] -> [115,210], acting [277,115,210] -> [115,210], role 2 -> 1 -537> 2014-04-25 09:18:54.732679 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive] on_change -536> 2014-04-25 09:18:54.732695 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive] on_role_change -535> 2014-04-25 09:18:54.732705 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] cancel_recovery -534> 2014-04-25 09:18:54.732715 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] clear_recovery_state -533> 2014-04-25 09:18:54.732728 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] handle_activate_map -532> 2014-04-25 09:18:54.732787 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] update_heartbeat_peers unchanged -531> 2014-04-25 09:18:54.732798 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] take_waiters -530> 2014-04-25 09:18:54.732808 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] exit Reset 0.000235 1 0.000221 -529> 2014-04-25 09:18:54.732821 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] enter Started -528> 2014-04-25 09:18:54.732831 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] enter Start -527> 2014-04-25 09:18:54.732842 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] state: transitioning to Stray -526> 2014-04-25 09:18:54.732853 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] exit Start 0.000022 0 0.000000 -525> 2014-04-25 09:18:54.732866 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] enter Started/Stray -524> 2014-04-25 09:18:54.732878 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.19b2( v 1110'1432 (0'0,1110'1432] local-les=3756 n=1432 ec=638 les/c 3756/3715 3799/3799/3799) [115,210] r=1 lpr=3799 pi=1354-3798/29 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -523> 2014-04-25 09:18:54.732935 7f77ef0e4700 10 log is not dirty -522> 2014-04-25 09:18:54.733060 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] handle_advance_map [210]/[210] -521> 2014-04-25 09:18:54.733079 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: Started advmap -520> 2014-04-25 09:18:54.733091 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: up or acting affected, transitioning to Reset -519> 2014-04-25 09:18:54.733102 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] exit Started/Stray 12.644425 3 0.000206 -518> 2014-04-25 09:18:54.733115 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] exit Started 12.644483 0 0.000000 -517> 2014-04-25 09:18:54.733127 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] enter Reset -516> 2014-04-25 09:18:54.733137 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: Reset advmap -515> 2014-04-25 09:18:54.733147 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] _calc_past_interval_range: already have past intervals back to 3715 -514> 2014-04-25 09:18:54.733171 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] dropping down osd.435 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -513> 2014-04-25 09:18:54.733186 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] update_heartbeat_peers unchanged -512> 2014-04-25 09:18:54.733196 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] check_recovery_sources no source osds () went down -511> 2014-04-25 09:18:54.733206 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: up or acting affected, calling start_peering_interval again -510> 2014-04-25 09:18:54.733229 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] start_peering_interval: check_new_interval output: generate_past_intervals interval(3798-3798 [435,210]/[435,210]): not rw, up_thru 3796 up_from 3764 last_epoch_clean 3715 generate_past_intervals interval(3798-3798 [435,210]/[435,210]) : primary up 3764-3796 does not include interval -509> 2014-04-25 09:18:54.733241 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3798/3798/3798) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] noting past interval(3798-3798 [435,210]/[435,210]) -508> 2014-04-25 09:18:54.733254 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] up [435,210] -> [210], acting [435,210] -> [210], role 1 -> 0 -507> 2014-04-25 09:18:54.733265 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] clear_primary_state -506> 2014-04-25 09:18:54.733276 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] on_change -505> 2014-04-25 09:18:54.733293 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] on_role_change -504> 2014-04-25 09:18:54.733304 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -503> 2014-04-25 09:18:54.733313 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -502> 2014-04-25 09:18:54.733325 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -501> 2014-04-25 09:18:54.733336 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 -500> 2014-04-25 09:18:54.733347 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] take_waiters -499> 2014-04-25 09:18:54.733357 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000230 1 0.000246 -498> 2014-04-25 09:18:54.733369 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started -497> 2014-04-25 09:18:54.733379 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Start -496> 2014-04-25 09:18:54.733390 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -495> 2014-04-25 09:18:54.733401 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -494> 2014-04-25 09:18:54.733412 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -493> 2014-04-25 09:18:54.733423 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -492> 2014-04-25 09:18:54.733434 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -491> 2014-04-25 09:18:54.733444 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 -490> 2014-04-25 09:18:54.733456 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3798-3798 [435,210]/[435,210]) -489> 2014-04-25 09:18:54.733467 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3797 [306,435,210]/[306,435,210] maybe_went_rw) -488> 2014-04-25 09:18:54.733479 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.306 is down -487> 2014-04-25 09:18:54.733489 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down -486> 2014-04-25 09:18:54.733498 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3761-3763 [306,210]/[306,210]) -485> 2014-04-25 09:18:54.733510 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 306,435 blocked_by {} -484> 2014-04-25 09:18:54.733521 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -483> 2014-04-25 09:18:54.733536 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000103 0 0.000000 -482> 2014-04-25 09:18:54.733550 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -481> 2014-04-25 09:18:54.733565 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) -480> 2014-04-25 09:18:54.733579 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) -479> 2014-04-25 09:18:54.733593 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary -478> 2014-04-25 09:18:54.733603 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) -477> 2014-04-25 09:18:54.733617 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) -476> 2014-04-25 09:18:54.733629 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog -475> 2014-04-25 09:18:54.733644 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000093 0 0.000000 -474> 2014-04-25 09:18:54.733658 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -473> 2014-04-25 09:18:54.733668 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active -472> 2014-04-25 09:18:54.733683 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000026 0 0.000000 -471> 2014-04-25 09:18:54.733697 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -470> 2014-04-25 09:18:54.733708 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -469> 2014-04-25 09:18:54.733771 7f77ef0e4700 10 log is not dirty -468> 2014-04-25 09:18:54.733796 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.2801( v 1110'1446 (0'0,1110'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] handle_advance_map [5,11,210]/[5,11,210] -467> 2014-04-25 09:18:54.733815 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2801( v 1110'1446 (0'0,1110'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] state: Started advmap -466> 2014-04-25 09:18:54.733827 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2801( v 1110'1446 (0'0,1110'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -465> 2014-04-25 09:18:54.733839 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2801( v 1110'1446 (0'0,1110'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] handle_activate_map -464> 2014-04-25 09:18:54.733850 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2801( v 1110'1446 (0'0,1110'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] take_waiters -463> 2014-04-25 09:18:54.733861 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.2801( v 1110'1446 (0'0,1110'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -462> 2014-04-25 09:18:54.733874 7f77ef0e4700 10 log is not dirty -461> 2014-04-25 09:18:54.733914 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.1f01( v 1110'719 (0'0,1110'719] local-les=3754 n=719 ec=633 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_advance_map [426,210,432]/[426,210,432] -460> 2014-04-25 09:18:54.733934 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1f01( v 1110'719 (0'0,1110'719] local-les=3754 n=719 ec=633 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] state: Started advmap -459> 2014-04-25 09:18:54.733946 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1f01( v 1110'719 (0'0,1110'719] local-les=3754 n=719 ec=633 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -458> 2014-04-25 09:18:54.733958 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1f01( v 1110'719 (0'0,1110'719] local-les=3754 n=719 ec=633 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_activate_map -457> 2014-04-25 09:18:54.733969 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1f01( v 1110'719 (0'0,1110'719] local-les=3754 n=719 ec=633 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] take_waiters -456> 2014-04-25 09:18:54.733980 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1f01( v 1110'719 (0'0,1110'719] local-les=3754 n=719 ec=633 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -455> 2014-04-25 09:18:54.733993 7f77ef0e4700 10 log is not dirty -454> 2014-04-25 09:18:54.734028 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.2802( v 1110'707 (0'0,1110'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] handle_advance_map [5,11,210]/[5,11,210] -453> 2014-04-25 09:18:54.734047 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.2802( v 1110'707 (0'0,1110'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] state: Started advmap -452> 2014-04-25 09:18:54.734059 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.2802( v 1110'707 (0'0,1110'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -451> 2014-04-25 09:18:54.734070 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.2802( v 1110'707 (0'0,1110'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] handle_activate_map -450> 2014-04-25 09:18:54.734082 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.2802( v 1110'707 (0'0,1110'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] take_waiters -449> 2014-04-25 09:18:54.734092 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.2802( v 1110'707 (0'0,1110'707] local-les=3754 n=707 ec=633 les/c 3754/3754 3753/3753/3719) [5,11,210] r=2 lpr=3753 pi=3714-3752/6 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -448> 2014-04-25 09:18:54.734106 7f77ef0e4700 10 log is not dirty -447> 2014-04-25 09:18:54.734120 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.9f1( v 1110'1387 (0'0,1110'1387] local-les=3754 n=1387 ec=638 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] handle_advance_map [330,195,210]/[330,195,210] -446> 2014-04-25 09:18:54.734139 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.9f1( v 1110'1387 (0'0,1110'1387] local-les=3754 n=1387 ec=638 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] state: Started advmap -445> 2014-04-25 09:18:54.734150 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.9f1( v 1110'1387 (0'0,1110'1387] local-les=3754 n=1387 ec=638 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -444> 2014-04-25 09:18:54.734169 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.9f1( v 1110'1387 (0'0,1110'1387] local-les=3754 n=1387 ec=638 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] handle_activate_map -443> 2014-04-25 09:18:54.734180 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.9f1( v 1110'1387 (0'0,1110'1387] local-les=3754 n=1387 ec=638 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] take_waiters -442> 2014-04-25 09:18:54.734191 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.9f1( v 1110'1387 (0'0,1110'1387] local-les=3754 n=1387 ec=638 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -441> 2014-04-25 09:18:54.734205 7f77ef0e4700 10 log is not dirty -440> 2014-04-25 09:18:54.734240 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,67]/[210,67] -439> 2014-04-25 09:18:54.734259 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] state: Peering advmap -438> 2014-04-25 09:18:54.734273 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -437> 2014-04-25 09:18:54.734284 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] state: Started advmap -436> 2014-04-25 09:18:54.734295 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -435> 2014-04-25 09:18:54.734306 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] handle_activate_map -434> 2014-04-25 09:18:54.734317 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary -433> 2014-04-25 09:18:54.734330 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] take_waiters -432> 2014-04-25 09:18:54.734342 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.3d00( v 1196'1382 (0'0,1196'1382] local-les=3771 n=1382 ec=638 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -431> 2014-04-25 09:18:54.734356 7f77ef0e4700 10 log is not dirty -430> 2014-04-25 09:18:54.734390 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,67]/[210,67] -429> 2014-04-25 09:18:54.734409 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] state: Peering advmap -428> 2014-04-25 09:18:54.734421 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -427> 2014-04-25 09:18:54.734433 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] state: Started advmap -426> 2014-04-25 09:18:54.734443 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -425> 2014-04-25 09:18:54.734454 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] handle_activate_map -424> 2014-04-25 09:18:54.734465 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary -423> 2014-04-25 09:18:54.734478 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] take_waiters -422> 2014-04-25 09:18:54.734489 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.3d01( v 1110'710 (0'0,1110'710] local-les=3771 n=710 ec=633 les/c 3771/3715 3798/3798/3753) [210,67] r=0 lpr=3798 pi=3714-3797/9 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -421> 2014-04-25 09:18:54.734502 7f77ef0e4700 10 log is not dirty -420> 2014-04-25 09:18:54.734517 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.9f2( v 1110'715 (0'0,1110'715] local-les=3754 n=715 ec=633 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] handle_advance_map [330,195,210]/[330,195,210] -419> 2014-04-25 09:18:54.734534 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.9f2( v 1110'715 (0'0,1110'715] local-les=3754 n=715 ec=633 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] state: Started advmap -418> 2014-04-25 09:18:54.734546 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.9f2( v 1110'715 (0'0,1110'715] local-les=3754 n=715 ec=633 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -417> 2014-04-25 09:18:54.734557 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.9f2( v 1110'715 (0'0,1110'715] local-les=3754 n=715 ec=633 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] handle_activate_map -416> 2014-04-25 09:18:54.734568 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.9f2( v 1110'715 (0'0,1110'715] local-les=3754 n=715 ec=633 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] take_waiters -415> 2014-04-25 09:18:54.734579 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.9f2( v 1110'715 (0'0,1110'715] local-les=3754 n=715 ec=633 les/c 3754/3754 3753/3753/3725) [330,195,210] r=2 lpr=3753 pi=3119-3752/14 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -414> 2014-04-25 09:18:54.734593 7f77ef0e4700 10 log is not dirty -413> 2014-04-25 09:18:54.734607 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,309,198]/[210,309,198] -412> 2014-04-25 09:18:54.734625 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -411> 2014-04-25 09:18:54.734638 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -410> 2014-04-25 09:18:54.734649 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -409> 2014-04-25 09:18:54.734660 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -408> 2014-04-25 09:18:54.734672 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -407> 2014-04-25 09:18:54.734683 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -406> 2014-04-25 09:18:54.734709 7f77ef0e4700 10 osd.210 3799 queue_for_recovery queued pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] -405> 2014-04-25 09:18:54.734720 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -404> 2014-04-25 09:18:54.734735 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -403> 2014-04-25 09:18:54.734745 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -402> 2014-04-25 09:18:54.734759 7f77ef0e4700 10 log is not dirty -401> 2014-04-25 09:18:54.734793 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,7]/[210,7] -400> 2014-04-25 09:18:54.734813 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -399> 2014-04-25 09:18:54.734824 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.14 info 4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3731 n=1376 ec=638 les/c 3731/3742 3753/3753/3753) -398> 2014-04-25 09:18:54.734842 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 7,14,210 unchanged -397> 2014-04-25 09:18:54.734853 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -396> 2014-04-25 09:18:54.734864 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -395> 2014-04-25 09:18:54.734873 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset -394> 2014-04-25 09:18:54.734885 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 173.010679 133 0.004790 -393> 2014-04-25 09:18:54.734901 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 173.595813 0 0.000000 -392> 2014-04-25 09:18:54.734918 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started/Primary 175.546194 0 0.000000 -391> 2014-04-25 09:18:54.734935 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started 175.546337 0 0.000000 -390> 2014-04-25 09:18:54.734947 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] enter Reset -389> 2014-04-25 09:18:54.734957 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap -388> 2014-04-25 09:18:54.734973 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3754 >= end epoch 3753, nothing to do -387> 2014-04-25 09:18:54.734983 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -386> 2014-04-25 09:18:54.734993 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -385> 2014-04-25 09:18:54.735014 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [210,14,7]/[210,14,7]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) : primary up 3753-3797 includes interval -384> 2014-04-25 09:18:54.735027 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3753/3753/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) -383> 2014-04-25 09:18:54.735039 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] up [210,14,7] -> [210,7], acting [210,14,7] -> [210,7], role 0 -> 0 -382> 2014-04-25 09:18:54.735052 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -381> 2014-04-25 09:18:54.735065 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change -380> 2014-04-25 09:18:54.735082 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,14,7] -> [210,7], replicas changed -379> 2014-04-25 09:18:54.735093 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -378> 2014-04-25 09:18:54.735104 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -377> 2014-04-25 09:18:54.735116 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -376> 2014-04-25 09:18:54.735128 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 7,14,210 -> 7,210 -375> 2014-04-25 09:18:54.735140 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters -374> 2014-04-25 09:18:54.735420 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] handle_advance_map [70,210]/[70,210] -373> 2014-04-25 09:18:54.735444 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Started advmap -372> 2014-04-25 09:18:54.735455 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -371> 2014-04-25 09:18:54.735468 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 167.427665 123 0.002269 -370> 2014-04-25 09:18:54.735483 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 167.427693 0 0.000000 -369> 2014-04-25 09:18:54.735495 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started 177.660998 0 0.000000 -368> 2014-04-25 09:18:54.735507 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] enter Reset -367> 2014-04-25 09:18:54.735519 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Reset advmap -366> 2014-04-25 09:18:54.735530 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3718 -365> 2014-04-25 09:18:54.735542 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -364> 2014-04-25 09:18:54.735552 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -363> 2014-04-25 09:18:54.735572 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [70,210] r=1 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [380,70,210]/[380,70,210]): not rw, up_thru 3796 up_from 3723 last_epoch_clean 3718 generate_past_intervals interval(3753-3798 [380,70,210]/[380,70,210] maybe_went_rw) : primary up 3723-3796 includes interval -362> 2014-04-25 09:18:54.735585 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3753/3753/3723) [70,210] r=1 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [380,70,210]/[380,70,210] maybe_went_rw) -361> 2014-04-25 09:18:54.735596 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 active] up [380,70,210] -> [70,210], acting [380,70,210] -> [70,210], role 2 -> 1 -360> 2014-04-25 09:18:54.735608 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive] on_change -359> 2014-04-25 09:18:54.735624 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive] on_role_change -358> 2014-04-25 09:18:54.735634 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] cancel_recovery -357> 2014-04-25 09:18:54.735643 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] clear_recovery_state -356> 2014-04-25 09:18:54.735656 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] handle_activate_map -355> 2014-04-25 09:18:54.735677 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] update_heartbeat_peers unchanged -354> 2014-04-25 09:18:54.735687 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] take_waiters -353> 2014-04-25 09:18:54.735697 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] exit Reset 0.000190 1 0.000213 -352> 2014-04-25 09:18:54.735709 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] enter Started -351> 2014-04-25 09:18:54.735719 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] enter Start -350> 2014-04-25 09:18:54.735729 7f77efae5700 1 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] state: transitioning to Stray -349> 2014-04-25 09:18:54.735740 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] exit Start 0.000021 0 0.000000 -348> 2014-04-25 09:18:54.735753 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] enter Started/Stray -347> 2014-04-25 09:18:54.735764 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.5cb0( v 1110'1389 (0'0,1110'1389] local-les=3758 n=1389 ec=638 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -346> 2014-04-25 09:18:54.735804 7f77efae5700 10 log is not dirty -345> 2014-04-25 09:18:54.735858 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] handle_advance_map [70,210]/[70,210] -344> 2014-04-25 09:18:54.735879 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Started advmap -343> 2014-04-25 09:18:54.735889 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -342> 2014-04-25 09:18:54.735902 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 167.428434 123 0.002206 -341> 2014-04-25 09:18:54.735917 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 167.428472 0 0.000000 -340> 2014-04-25 09:18:54.735931 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started 177.660423 0 0.000000 -339> 2014-04-25 09:18:54.735945 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] enter Reset -338> 2014-04-25 09:18:54.735957 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Reset advmap -337> 2014-04-25 09:18:54.735968 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3718 -336> 2014-04-25 09:18:54.735980 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -335> 2014-04-25 09:18:54.735990 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [380,70,210] r=2 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -334> 2014-04-25 09:18:54.736011 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [70,210] r=1 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [380,70,210]/[380,70,210]): not rw, up_thru 3796 up_from 3723 last_epoch_clean 3718 generate_past_intervals interval(3753-3798 [380,70,210]/[380,70,210] maybe_went_rw) : primary up 3723-3796 includes interval -333> 2014-04-25 09:18:54.736023 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3753/3753/3723) [70,210] r=1 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [380,70,210]/[380,70,210] maybe_went_rw) -332> 2014-04-25 09:18:54.736035 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 active] up [380,70,210] -> [70,210], acting [380,70,210] -> [70,210], role 2 -> 1 -331> 2014-04-25 09:18:54.736048 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive] on_change -330> 2014-04-25 09:18:54.736062 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive] on_role_change -329> 2014-04-25 09:18:54.736072 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] cancel_recovery -328> 2014-04-25 09:18:54.736082 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] clear_recovery_state -327> 2014-04-25 09:18:54.736095 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] handle_activate_map -326> 2014-04-25 09:18:54.736114 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] update_heartbeat_peers unchanged -325> 2014-04-25 09:18:54.736125 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] take_waiters -324> 2014-04-25 09:18:54.736136 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] exit Reset 0.000192 1 0.000217 -323> 2014-04-25 09:18:54.736148 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] enter Started -322> 2014-04-25 09:18:54.736165 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] enter Start -321> 2014-04-25 09:18:54.736175 7f77efae5700 1 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] state: transitioning to Stray -320> 2014-04-25 09:18:54.736186 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] exit Start 0.000021 0 0.000000 -319> 2014-04-25 09:18:54.736198 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] enter Started/Stray -318> 2014-04-25 09:18:54.736210 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.5cb1( v 1178'697 (0'0,1178'697] local-les=3758 n=697 ec=633 les/c 3758/3718 3799/3799/3799) [70,210] r=1 lpr=3799 pi=3716-3798/6 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -317> 2014-04-25 09:18:54.736246 7f77efae5700 10 log is not dirty -316> 2014-04-25 09:18:54.736277 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,149,388]/[210,149,388] -315> 2014-04-25 09:18:54.736296 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -314> 2014-04-25 09:18:54.736308 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -313> 2014-04-25 09:18:54.736319 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -312> 2014-04-25 09:18:54.736329 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -311> 2014-04-25 09:18:54.736339 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -310> 2014-04-25 09:18:54.736350 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -309> 2014-04-25 09:18:54.736360 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -308> 2014-04-25 09:18:54.736371 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -307> 2014-04-25 09:18:54.736382 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.74ba( v 1110'734 (0'0,1110'734] local-les=3754 n=734 ec=633 les/c 3754/3754 3753/3753/3753) [210,149,388] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -306> 2014-04-25 09:18:54.736395 7f77efae5700 10 log is not dirty -305> 2014-04-25 09:18:54.736433 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.71bb( v 1110'695 (0'0,1110'695] local-les=3755 n=695 ec=633 les/c 3755/3757 3753/3753/3718) [66,57,210] r=2 lpr=3753 pi=2929-3752/22 luod=0'0 lcod 0'0 active] handle_advance_map [66,57,210]/[66,57,210] -304> 2014-04-25 09:18:54.736453 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.71bb( v 1110'695 (0'0,1110'695] local-les=3755 n=695 ec=633 les/c 3755/3757 3753/3753/3718) [66,57,210] r=2 lpr=3753 pi=2929-3752/22 luod=0'0 lcod 0'0 active] state: Started advmap -303> 2014-04-25 09:18:54.736465 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.71bb( v 1110'695 (0'0,1110'695] local-les=3755 n=695 ec=633 les/c 3755/3757 3753/3753/3718) [66,57,210] r=2 lpr=3753 pi=2929-3752/22 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -302> 2014-04-25 09:18:54.736477 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.71bb( v 1110'695 (0'0,1110'695] local-les=3755 n=695 ec=633 les/c 3755/3757 3753/3753/3718) [66,57,210] r=2 lpr=3753 pi=2929-3752/22 luod=0'0 lcod 0'0 active] handle_activate_map -301> 2014-04-25 09:18:54.736488 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.71bb( v 1110'695 (0'0,1110'695] local-les=3755 n=695 ec=633 les/c 3755/3757 3753/3753/3718) [66,57,210] r=2 lpr=3753 pi=2929-3752/22 luod=0'0 lcod 0'0 active] take_waiters -300> 2014-04-25 09:18:54.736499 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.71bb( v 1110'695 (0'0,1110'695] local-les=3755 n=695 ec=633 les/c 3755/3757 3753/3753/3718) [66,57,210] r=2 lpr=3753 pi=2929-3752/22 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -299> 2014-04-25 09:18:54.736512 7f77efae5700 10 log is not dirty -298> 2014-04-25 09:18:54.736548 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] handle_advance_map [210,409]/[210,409] -297> 2014-04-25 09:18:54.736567 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] state: Active advmap -296> 2014-04-25 09:18:54.736578 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] check_recovery_sources no source osds () went down -295> 2014-04-25 09:18:54.736589 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] state: Started advmap -294> 2014-04-25 09:18:54.736598 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] check_recovery_sources no source osds () went down -293> 2014-04-25 09:18:54.736609 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] handle_activate_map -292> 2014-04-25 09:18:54.736620 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] state: Active: handling ActMap -291> 2014-04-25 09:18:54.736630 7f77efae5700 10 osd.210 3799 queue_for_recovery queued pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] -290> 2014-04-25 09:18:54.736638 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] state: handle ActMap primary -289> 2014-04-25 09:18:54.736652 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] take_waiters -288> 2014-04-25 09:18:54.736662 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -287> 2014-04-25 09:18:54.736675 7f77efae5700 10 log is not dirty -286> 2014-04-25 09:18:54.736690 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.56af( v 1110'1458 (0'0,1110'1458] local-les=3754 n=1458 ec=638 les/c 3754/3715 3798/3798/3749) [72,210] r=1 lpr=3798 pi=2264-3797/24 lcod 0'0 inactive NOTIFY] handle_advance_map [72,210]/[72,210] -285> 2014-04-25 09:18:54.736706 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.56af( v 1110'1458 (0'0,1110'1458] local-les=3754 n=1458 ec=638 les/c 3754/3715 3798/3798/3749) [72,210] r=1 lpr=3798 pi=2264-3797/24 lcod 0'0 inactive NOTIFY] state: Started advmap -284> 2014-04-25 09:18:54.736717 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.56af( v 1110'1458 (0'0,1110'1458] local-les=3754 n=1458 ec=638 les/c 3754/3715 3798/3798/3749) [72,210] r=1 lpr=3798 pi=2264-3797/24 lcod 0'0 inactive NOTIFY] check_recovery_sources no source osds () went down -283> 2014-04-25 09:18:54.736728 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.56af( v 1110'1458 (0'0,1110'1458] local-les=3754 n=1458 ec=638 les/c 3754/3715 3798/3798/3749) [72,210] r=1 lpr=3798 pi=2264-3797/24 lcod 0'0 inactive NOTIFY] handle_activate_map -282> 2014-04-25 09:18:54.736769 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.56af( v 1110'1458 (0'0,1110'1458] local-les=3754 n=1458 ec=638 les/c 3754/3715 3798/3798/3749) [72,210] r=1 lpr=3798 pi=2264-3797/24 lcod 0'0 inactive NOTIFY] take_waiters -281> 2014-04-25 09:18:54.736781 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.56af( v 1110'1458 (0'0,1110'1458] local-les=3754 n=1458 ec=638 les/c 3754/3715 3798/3798/3749) [72,210] r=1 lpr=3798 pi=2264-3797/24 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -280> 2014-04-25 09:18:54.736794 7f77efae5700 10 log is not dirty -279> 2014-04-25 09:18:54.736836 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] handle_advance_map [210,409]/[210,409] -278> 2014-04-25 09:18:54.736857 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] state: Active advmap -277> 2014-04-25 09:18:54.736868 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] check_recovery_sources no source osds () went down -276> 2014-04-25 09:18:54.736878 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] state: Started advmap -275> 2014-04-25 09:18:54.736888 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] check_recovery_sources no source osds () went down -274> 2014-04-25 09:18:54.736898 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] handle_activate_map -273> 2014-04-25 09:18:54.736910 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] state: Active: handling ActMap -272> 2014-04-25 09:18:54.736920 7f77efae5700 10 osd.210 3799 queue_for_recovery queued pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] -271> 2014-04-25 09:18:54.736929 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] state: handle ActMap primary -270> 2014-04-25 09:18:54.736941 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] take_waiters -269> 2014-04-25 09:18:54.736952 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -268> 2014-04-25 09:18:54.736969 7f77efae5700 10 log is not dirty -267> 2014-04-25 09:18:54.737021 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,44,365]/[210,44,365] -266> 2014-04-25 09:18:54.737039 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -265> 2014-04-25 09:18:54.737050 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -264> 2014-04-25 09:18:54.737060 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -263> 2014-04-25 09:18:54.737071 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -262> 2014-04-25 09:18:54.737082 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -261> 2014-04-25 09:18:54.737092 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -260> 2014-04-25 09:18:54.737103 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -259> 2014-04-25 09:18:54.737115 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] take_waiters -258> 2014-04-25 09:18:54.737125 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.54f9( v 1110'702 (0'0,1110'702] local-les=3767 n=702 ec=633 les/c 3767/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -257> 2014-04-25 09:18:54.737138 7f77efae5700 10 log is not dirty -256> 2014-04-25 09:18:54.737185 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.7307( v 1110'1464 (0'0,1110'1464] local-les=3777 n=1464 ec=638 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_advance_map [67,88,210]/[67,88,210] -255> 2014-04-25 09:18:54.737207 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.7307( v 1110'1464 (0'0,1110'1464] local-les=3777 n=1464 ec=638 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] state: Started advmap -254> 2014-04-25 09:18:54.737218 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.7307( v 1110'1464 (0'0,1110'1464] local-les=3777 n=1464 ec=638 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -253> 2014-04-25 09:18:54.737230 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.7307( v 1110'1464 (0'0,1110'1464] local-les=3777 n=1464 ec=638 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_activate_map -252> 2014-04-25 09:18:54.737241 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.7307( v 1110'1464 (0'0,1110'1464] local-les=3777 n=1464 ec=638 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] take_waiters -251> 2014-04-25 09:18:54.737252 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.7307( v 1110'1464 (0'0,1110'1464] local-les=3777 n=1464 ec=638 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -250> 2014-04-25 09:18:54.737265 7f77efae5700 10 log is not dirty -249> 2014-04-25 09:18:54.737281 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,347,456]/[210,347,456] -248> 2014-04-25 09:18:54.737297 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -247> 2014-04-25 09:18:54.737309 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -246> 2014-04-25 09:18:54.737320 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -245> 2014-04-25 09:18:54.737330 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -244> 2014-04-25 09:18:54.737340 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -243> 2014-04-25 09:18:54.737351 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -242> 2014-04-25 09:18:54.737361 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -241> 2014-04-25 09:18:54.737372 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -240> 2014-04-25 09:18:54.737383 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.6ffd( v 1156'671 (0'0,1156'671] local-les=3754 n=671 ec=633 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -239> 2014-04-25 09:18:54.737396 7f77efae5700 10 log is not dirty -238> 2014-04-25 09:18:54.737412 7f77cda10700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.180:0/75847 pipe(0x1d07a500 sd=349 :6919 s=2 pgs=2003 cs=1 l=1 c=0x120b0c60).reader couldn't read tag, Success -237> 2014-04-25 09:18:54.737434 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.30e2( v 1139'1497 (0'0,1139'1497] local-les=3783 n=1497 ec=638 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=638-3779/57 luod=0'0 lcod 0'0 active] handle_advance_map [422,329,210]/[422,329,210] -236> 2014-04-25 09:18:54.737444 7f77cda10700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.180:0/75847 pipe(0x1d07a500 sd=349 :6919 s=2 pgs=2003 cs=1 l=1 c=0x120b0c60).fault 0: Success -235> 2014-04-25 09:18:54.737455 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.30e2( v 1139'1497 (0'0,1139'1497] local-les=3783 n=1497 ec=638 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=638-3779/57 luod=0'0 lcod 0'0 active] state: Started advmap -234> 2014-04-25 09:18:54.737466 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.30e2( v 1139'1497 (0'0,1139'1497] local-les=3783 n=1497 ec=638 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=638-3779/57 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -233> 2014-04-25 09:18:54.737479 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.30e2( v 1139'1497 (0'0,1139'1497] local-les=3783 n=1497 ec=638 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=638-3779/57 luod=0'0 lcod 0'0 active] handle_activate_map -232> 2014-04-25 09:18:54.737490 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.30e2( v 1139'1497 (0'0,1139'1497] local-les=3783 n=1497 ec=638 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=638-3779/57 luod=0'0 lcod 0'0 active] take_waiters -231> 2014-04-25 09:18:54.737501 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.30e2( v 1139'1497 (0'0,1139'1497] local-les=3783 n=1497 ec=638 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=638-3779/57 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -230> 2014-04-25 09:18:54.737514 7f77efae5700 10 log is not dirty -229> 2014-04-25 09:18:54.737547 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.1edc( v 1110'718 (0'0,1110'718] local-les=3755 n=718 ec=633 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] handle_advance_map [118,210]/[118,210] -228> 2014-04-25 09:18:54.737566 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.1edc( v 1110'718 (0'0,1110'718] local-les=3755 n=718 ec=633 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] state: Started advmap -227> 2014-04-25 09:18:54.737579 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.1edc( v 1110'718 (0'0,1110'718] local-les=3755 n=718 ec=633 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] check_recovery_sources no source osds () went down -226> 2014-04-25 09:18:54.737589 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.1edc( v 1110'718 (0'0,1110'718] local-les=3755 n=718 ec=633 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] handle_activate_map -225> 2014-04-25 09:18:54.737600 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.1edc( v 1110'718 (0'0,1110'718] local-les=3755 n=718 ec=633 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] take_waiters -224> 2014-04-25 09:18:54.737611 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.1edc( v 1110'718 (0'0,1110'718] local-les=3755 n=718 ec=633 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -223> 2014-04-25 09:18:54.737623 7f77efae5700 10 log is not dirty -222> 2014-04-25 09:18:54.737635 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.1edb( v 1133'1357 (0'0,1133'1357] local-les=3755 n=1357 ec=638 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] handle_advance_map [118,210]/[118,210] -221> 2014-04-25 09:18:54.737650 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.1edb( v 1133'1357 (0'0,1133'1357] local-les=3755 n=1357 ec=638 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] state: Started advmap -220> 2014-04-25 09:18:54.737661 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.1edb( v 1133'1357 (0'0,1133'1357] local-les=3755 n=1357 ec=638 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] check_recovery_sources no source osds () went down -219> 2014-04-25 09:18:54.737671 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.1edb( v 1133'1357 (0'0,1133'1357] local-les=3755 n=1357 ec=638 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] handle_activate_map -218> 2014-04-25 09:18:54.737682 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.1edb( v 1133'1357 (0'0,1133'1357] local-les=3755 n=1357 ec=638 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] take_waiters -217> 2014-04-25 09:18:54.737692 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.1edb( v 1133'1357 (0'0,1133'1357] local-les=3755 n=1357 ec=638 les/c 3755/3715 3798/3798/3719) [118,210] r=1 lpr=3798 pi=3714-3797/7 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -216> 2014-04-25 09:18:54.737704 7f77efae5700 10 log is not dirty -215> 2014-04-25 09:18:54.737716 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.24dc( v 1157'697 (0'0,1157'697] local-les=3755 n=697 ec=633 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] handle_advance_map [449,401,210]/[449,401,210] -214> 2014-04-25 09:18:54.737732 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.24dc( v 1157'697 (0'0,1157'697] local-les=3755 n=697 ec=633 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] state: Started advmap -213> 2014-04-25 09:18:54.737745 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.24dc( v 1157'697 (0'0,1157'697] local-les=3755 n=697 ec=633 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -212> 2014-04-25 09:18:54.737756 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.24dc( v 1157'697 (0'0,1157'697] local-les=3755 n=697 ec=633 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] handle_activate_map -211> 2014-04-25 09:18:54.737767 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.24dc( v 1157'697 (0'0,1157'697] local-les=3755 n=697 ec=633 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] take_waiters -210> 2014-04-25 09:18:54.737778 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.24dc( v 1157'697 (0'0,1157'697] local-les=3755 n=697 ec=633 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -209> 2014-04-25 09:18:54.737791 7f77efae5700 10 log is not dirty -208> 2014-04-25 09:18:54.737825 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.24db( v 1156'1394 (0'0,1156'1394] local-les=3755 n=1394 ec=638 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] handle_advance_map [449,401,210]/[449,401,210] -207> 2014-04-25 09:18:54.737844 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.24db( v 1156'1394 (0'0,1156'1394] local-les=3755 n=1394 ec=638 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] state: Started advmap -206> 2014-04-25 09:18:54.737857 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.24db( v 1156'1394 (0'0,1156'1394] local-les=3755 n=1394 ec=638 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -205> 2014-04-25 09:18:54.737869 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.24db( v 1156'1394 (0'0,1156'1394] local-les=3755 n=1394 ec=638 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] handle_activate_map -204> 2014-04-25 09:18:54.737880 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.24db( v 1156'1394 (0'0,1156'1394] local-les=3755 n=1394 ec=638 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] take_waiters -203> 2014-04-25 09:18:54.737892 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.24db( v 1156'1394 (0'0,1156'1394] local-les=3755 n=1394 ec=638 les/c 3755/3719 3753/3753/3723) [449,401,210] r=2 lpr=3753 pi=3718-3752/5 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -202> 2014-04-25 09:18:54.737905 7f77efae5700 10 log is not dirty -201> 2014-04-25 09:18:54.737921 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,347,456]/[210,347,456] -200> 2014-04-25 09:18:54.737938 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -199> 2014-04-25 09:18:54.737949 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -198> 2014-04-25 09:18:54.737960 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -197> 2014-04-25 09:18:54.737970 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -196> 2014-04-25 09:18:54.737980 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -195> 2014-04-25 09:18:54.737991 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -194> 2014-04-25 09:18:54.738001 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -193> 2014-04-25 09:18:54.738013 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -192> 2014-04-25 09:18:54.738024 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.6ffc( v 1110'1375 (0'0,1110'1375] local-les=3754 n=1375 ec=638 les/c 3754/3754 3753/3753/3753) [210,347,456] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -191> 2014-04-25 09:18:54.738037 7f77efae5700 10 log is not dirty -190> 2014-04-25 09:18:54.738051 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.30e3( v 1139'722 (0'0,1139'722] local-les=3783 n=722 ec=633 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=633-3779/57 luod=0'0 lcod 0'0 active] handle_advance_map [422,329,210]/[422,329,210] -189> 2014-04-25 09:18:54.738068 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.30e3( v 1139'722 (0'0,1139'722] local-les=3783 n=722 ec=633 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=633-3779/57 luod=0'0 lcod 0'0 active] state: Started advmap -188> 2014-04-25 09:18:54.738080 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.30e3( v 1139'722 (0'0,1139'722] local-les=3783 n=722 ec=633 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=633-3779/57 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -187> 2014-04-25 09:18:54.738092 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.30e3( v 1139'722 (0'0,1139'722] local-les=3783 n=722 ec=633 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=633-3779/57 luod=0'0 lcod 0'0 active] handle_activate_map -186> 2014-04-25 09:18:54.738103 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.30e3( v 1139'722 (0'0,1139'722] local-les=3783 n=722 ec=633 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=633-3779/57 luod=0'0 lcod 0'0 active] take_waiters -185> 2014-04-25 09:18:54.738115 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.30e3( v 1139'722 (0'0,1139'722] local-les=3783 n=722 ec=633 les/c 3783/3783 3780/3780/3780) [422,329,210] r=2 lpr=3780 pi=633-3779/57 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -184> 2014-04-25 09:18:54.738128 7f77efae5700 10 log is not dirty -183> 2014-04-25 09:18:54.738142 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,438]/[210,438] -182> 2014-04-25 09:18:54.738170 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -181> 2014-04-25 09:18:54.738183 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.206 info 3.fcf( v 1110'711 (0'0,1110'711] local-les=3754 n=711 ec=633 les/c 3754/3754 3756/3756/3753) -180> 2014-04-25 09:18:54.738201 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 206,210,438 unchanged -179> 2014-04-25 09:18:54.738212 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -178> 2014-04-25 09:18:54.738222 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -177> 2014-04-25 09:18:54.738231 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset -176> 2014-04-25 09:18:54.738243 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 154.516950 94 0.003452 -175> 2014-04-25 09:18:54.738256 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 156.581439 0 0.000000 -174> 2014-04-25 09:18:54.738270 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active] exit Started/Primary 170.632799 0 0.000000 -173> 2014-04-25 09:18:54.738281 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active] exit Started 170.632855 0 0.000000 -172> 2014-04-25 09:18:54.738293 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active] enter Reset -171> 2014-04-25 09:18:54.738304 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap -170> 2014-04-25 09:18:54.738316 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3767 >= end epoch 3756, nothing to do -169> 2014-04-25 09:18:54.738327 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -168> 2014-04-25 09:18:54.738337 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -167> 2014-04-25 09:18:54.738358 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3756-3798 [210,206,438]/[210,206,438]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3767 generate_past_intervals interval(3756-3798 [210,206,438]/[210,206,438] maybe_went_rw) : primary up 3753-3797 includes interval -166> 2014-04-25 09:18:54.738370 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3756/3756/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3756-3798 [210,206,438]/[210,206,438] maybe_went_rw) -165> 2014-04-25 09:18:54.738381 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 active] up [210,206,438] -> [210,438], acting [210,206,438] -> [210,438], role 0 -> 0 -164> 2014-04-25 09:18:54.738393 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -163> 2014-04-25 09:18:54.738405 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change -162> 2014-04-25 09:18:54.738422 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,206,438] -> [210,438], replicas changed -161> 2014-04-25 09:18:54.738433 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -160> 2014-04-25 09:18:54.738443 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -159> 2014-04-25 09:18:54.738455 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -158> 2014-04-25 09:18:54.738466 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 206,210,438 -> 210,438 -157> 2014-04-25 09:18:54.738478 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters -156> 2014-04-25 09:18:54.738488 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000195 1 0.000286 -155> 2014-04-25 09:18:54.738500 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started -154> 2014-04-25 09:18:54.738511 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start -153> 2014-04-25 09:18:54.738521 7f77efae5700 1 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -152> 2014-04-25 09:18:54.738532 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -151> 2014-04-25 09:18:54.738545 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -150> 2014-04-25 09:18:54.738556 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -149> 2014-04-25 09:18:54.738566 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -148> 2014-04-25 09:18:54.738577 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3767 -147> 2014-04-25 09:18:54.738589 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3756-3798 [210,206,438]/[210,206,438] maybe_went_rw) -146> 2014-04-25 09:18:54.738602 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.206 is down -145> 2014-04-25 09:18:54.738612 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,438 down 206 blocked_by {} -144> 2014-04-25 09:18:54.738624 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -143> 2014-04-25 09:18:54.738636 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.438 -142> 2014-04-25 09:18:54.738649 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.fcf( v 1110'711 (0'0,1110'711] local-les=3765 n=711 ec=633 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -141> 2014-04-25 09:18:54.738683 7f77efae5700 10 log is not dirty -140> 2014-04-25 09:18:54.738706 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,109,249]/[210,109,249] -139> 2014-04-25 09:18:54.738726 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -138> 2014-04-25 09:18:54.738740 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -137> 2014-04-25 09:18:54.738751 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -136> 2014-04-25 09:18:54.738761 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -135> 2014-04-25 09:18:54.738773 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -134> 2014-04-25 09:18:54.738785 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -133> 2014-04-25 09:18:54.738796 7f77efae5700 10 osd.210 3799 queue_for_recovery queued pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] -132> 2014-04-25 09:18:54.738805 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -131> 2014-04-25 09:18:54.738819 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -130> 2014-04-25 09:18:54.738831 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -129> 2014-04-25 09:18:54.738845 7f77efae5700 10 log is not dirty -128> 2014-04-25 09:18:54.738884 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,438]/[210,438] -127> 2014-04-25 09:18:54.738904 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -126> 2014-04-25 09:18:54.738916 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.206 info 4.fce( v 1110'1489 (0'0,1110'1489] local-les=3754 n=1488 ec=638 les/c 3754/3754 3756/3756/3753) -125> 2014-04-25 09:18:54.738934 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 206,210,438 unchanged -124> 2014-04-25 09:18:54.738944 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -123> 2014-04-25 09:18:54.738955 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -122> 2014-04-25 09:18:54.738964 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset -121> 2014-04-25 09:18:54.738976 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 154.516156 94 0.003566 -120> 2014-04-25 09:18:54.738988 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 156.581596 0 0.000000 -119> 2014-04-25 09:18:54.739001 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active] exit Started/Primary 170.631998 0 0.000000 -118> 2014-04-25 09:18:54.739013 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active] exit Started 170.632054 0 0.000000 -117> 2014-04-25 09:18:54.739024 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active] enter Reset -116> 2014-04-25 09:18:54.739035 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap -115> 2014-04-25 09:18:54.739047 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3767 >= end epoch 3756, nothing to do -114> 2014-04-25 09:18:54.739058 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -113> 2014-04-25 09:18:54.739067 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,206,438] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -112> 2014-04-25 09:18:54.739088 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3756-3798 [210,206,438]/[210,206,438]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3767 generate_past_intervals interval(3756-3798 [210,206,438]/[210,206,438] maybe_went_rw) : primary up 3753-3797 includes interval -111> 2014-04-25 09:18:54.739099 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3756/3756/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3756-3798 [210,206,438]/[210,206,438] maybe_went_rw) -110> 2014-04-25 09:18:54.739111 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 active] up [210,206,438] -> [210,438], acting [210,206,438] -> [210,438], role 0 -> 0 -109> 2014-04-25 09:18:54.739124 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -108> 2014-04-25 09:18:54.739136 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change -107> 2014-04-25 09:18:54.739153 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,206,438] -> [210,438], replicas changed -106> 2014-04-25 09:18:54.739171 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -105> 2014-04-25 09:18:54.739180 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -104> 2014-04-25 09:18:54.739193 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -103> 2014-04-25 09:18:54.739204 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 206,210,438 -> 210,438 -102> 2014-04-25 09:18:54.739215 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters -101> 2014-04-25 09:18:54.739225 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000201 1 0.000290 -100> 2014-04-25 09:18:54.739238 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started -99> 2014-04-25 09:18:54.739248 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start -98> 2014-04-25 09:18:54.739260 7f77efae5700 1 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -97> 2014-04-25 09:18:54.739271 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000023 0 0.000000 -96> 2014-04-25 09:18:54.739283 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -95> 2014-04-25 09:18:54.739294 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -94> 2014-04-25 09:18:54.739304 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -93> 2014-04-25 09:18:54.739314 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3767 -92> 2014-04-25 09:18:54.739326 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3756-3798 [210,206,438]/[210,206,438] maybe_went_rw) -91> 2014-04-25 09:18:54.739337 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.206 is down -90> 2014-04-25 09:18:54.739348 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,438 down 206 blocked_by {} -89> 2014-04-25 09:18:54.739359 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -88> 2014-04-25 09:18:54.739370 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.438 -87> 2014-04-25 09:18:54.739384 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.fce( v 1110'1489 (0'0,1110'1489] local-les=3765 n=1488 ec=638 les/c 3765/3767 3799/3799/3753) [210,438] r=0 lpr=3799 pi=3756-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -86> 2014-04-25 09:18:54.739419 7f77efae5700 10 log is not dirty -85> 2014-04-25 09:18:54.739442 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,109,249]/[210,109,249] -84> 2014-04-25 09:18:54.739461 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -83> 2014-04-25 09:18:54.739474 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -82> 2014-04-25 09:18:54.739486 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -81> 2014-04-25 09:18:54.739496 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -80> 2014-04-25 09:18:54.739508 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -79> 2014-04-25 09:18:54.739519 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -78> 2014-04-25 09:18:54.739531 7f77efae5700 10 osd.210 3799 queue_for_recovery queued pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] -77> 2014-04-25 09:18:54.739541 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -76> 2014-04-25 09:18:54.739556 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -75> 2014-04-25 09:18:54.739567 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -74> 2014-04-25 09:18:54.739581 7f77efae5700 10 log is not dirty -73> 2014-04-25 09:18:54.739596 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.12c4( v 1289'735 (0'0,1289'735] local-les=3755 n=735 ec=633 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] handle_advance_map [43,210]/[43,210] -72> 2014-04-25 09:18:54.739612 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.12c4( v 1289'735 (0'0,1289'735] local-les=3755 n=735 ec=633 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] state: Started advmap -71> 2014-04-25 09:18:54.739623 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.12c4( v 1289'735 (0'0,1289'735] local-les=3755 n=735 ec=633 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] check_recovery_sources no source osds () went down -70> 2014-04-25 09:18:54.739634 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.12c4( v 1289'735 (0'0,1289'735] local-les=3755 n=735 ec=633 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] handle_activate_map -69> 2014-04-25 09:18:54.739655 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.12c4( v 1289'735 (0'0,1289'735] local-les=3755 n=735 ec=633 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] take_waiters -68> 2014-04-25 09:18:54.739666 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.12c4( v 1289'735 (0'0,1289'735] local-les=3755 n=735 ec=633 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -67> 2014-04-25 09:18:54.739679 7f77efae5700 10 log is not dirty -66> 2014-04-25 09:18:54.739694 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.12c3( v 1110'1443 (0'0,1110'1443] local-les=3755 n=1443 ec=638 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] handle_advance_map [43,210]/[43,210] -65> 2014-04-25 09:18:54.739710 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.12c3( v 1110'1443 (0'0,1110'1443] local-les=3755 n=1443 ec=638 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] state: Started advmap -64> 2014-04-25 09:18:54.739722 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.12c3( v 1110'1443 (0'0,1110'1443] local-les=3755 n=1443 ec=638 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] check_recovery_sources no source osds () went down -63> 2014-04-25 09:18:54.739732 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.12c3( v 1110'1443 (0'0,1110'1443] local-les=3755 n=1443 ec=638 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] handle_activate_map -62> 2014-04-25 09:18:54.739752 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.12c3( v 1110'1443 (0'0,1110'1443] local-les=3755 n=1443 ec=638 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] take_waiters -61> 2014-04-25 09:18:54.739763 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.12c3( v 1110'1443 (0'0,1110'1443] local-les=3755 n=1443 ec=638 les/c 3755/3715 3798/3798/3798) [43,210] r=1 lpr=3798 pi=3714-3797/8 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -60> 2014-04-25 09:18:54.739775 7f77efae5700 10 log is not dirty -59> 2014-04-25 09:18:54.739790 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,382,246]/[210,382,246] -58> 2014-04-25 09:18:54.739808 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -57> 2014-04-25 09:18:54.739820 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -56> 2014-04-25 09:18:54.739830 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -55> 2014-04-25 09:18:54.739840 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -54> 2014-04-25 09:18:54.739851 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -53> 2014-04-25 09:18:54.739861 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -52> 2014-04-25 09:18:54.739872 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -51> 2014-04-25 09:18:54.739883 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -50> 2014-04-25 09:18:54.739894 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.3ccf( v 1110'682 (0'0,1110'682] local-les=3754 n=682 ec=633 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -49> 2014-04-25 09:18:54.739908 7f77efae5700 10 log is not dirty -48> 2014-04-25 09:18:54.739946 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,382,246]/[210,382,246] -47> 2014-04-25 09:18:54.739965 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -46> 2014-04-25 09:18:54.739977 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -45> 2014-04-25 09:18:54.739988 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -44> 2014-04-25 09:18:54.739999 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -43> 2014-04-25 09:18:54.740010 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -42> 2014-04-25 09:18:54.740021 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -41> 2014-04-25 09:18:54.740031 7f77efae5700 7 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -40> 2014-04-25 09:18:54.740042 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -39> 2014-04-25 09:18:54.740053 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[4.3cce( v 1110'1363 (0'0,1110'1363] local-les=3754 n=1363 ec=638 les/c 3754/3755 3753/3753/3753) [210,382,246] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -38> 2014-04-25 09:18:54.740066 7f77efae5700 10 log is not dirty -37> 2014-04-25 09:18:54.740082 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.18be( v 1139'737 (0'0,1139'737] local-les=3757 n=737 ec=633 les/c 3757/3121 3798/3798/3749) [171,210] r=1 lpr=3798 pi=2287-3797/29 lcod 0'0 inactive] handle_advance_map [171,210]/[171,210] -36> 2014-04-25 09:18:54.740098 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.18be( v 1139'737 (0'0,1139'737] local-les=3757 n=737 ec=633 les/c 3757/3121 3798/3798/3749) [171,210] r=1 lpr=3798 pi=2287-3797/29 lcod 0'0 inactive] state: Started advmap -35> 2014-04-25 09:18:54.740109 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.18be( v 1139'737 (0'0,1139'737] local-les=3757 n=737 ec=633 les/c 3757/3121 3798/3798/3749) [171,210] r=1 lpr=3798 pi=2287-3797/29 lcod 0'0 inactive] check_recovery_sources no source osds () went down -34> 2014-04-25 09:18:54.740119 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.18be( v 1139'737 (0'0,1139'737] local-les=3757 n=737 ec=633 les/c 3757/3121 3798/3798/3749) [171,210] r=1 lpr=3798 pi=2287-3797/29 lcod 0'0 inactive] handle_activate_map -33> 2014-04-25 09:18:54.740130 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.18be( v 1139'737 (0'0,1139'737] local-les=3757 n=737 ec=633 les/c 3757/3121 3798/3798/3749) [171,210] r=1 lpr=3798 pi=2287-3797/29 lcod 0'0 inactive] take_waiters -32> 2014-04-25 09:18:54.740141 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.18be( v 1139'737 (0'0,1139'737] local-les=3757 n=737 ec=633 les/c 3757/3121 3798/3798/3749) [171,210] r=1 lpr=3798 pi=2287-3797/29 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -31> 2014-04-25 09:18:54.740153 7f77efae5700 10 log is not dirty -30> 2014-04-25 09:18:54.740940 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) -29> 2014-04-25 09:18:54.740948 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] -28> 2014-04-25 09:18:54.740963 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] recovery raced and were queued twice, ignoring! -27> 2014-04-25 09:18:54.740978 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[4.6108( v 1110'1502 (0'0,1110'1502] local-les=3754 n=1502 ec=638 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] -26> 2014-04-25 09:18:54.740987 7f77edce2700 10 log is not dirty -25> 2014-04-25 09:18:54.740992 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) -24> 2014-04-25 09:18:54.740995 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] -23> 2014-04-25 09:18:54.741005 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] recovery raced and were queued twice, ignoring! -22> 2014-04-25 09:18:54.741015 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[3.53ae( v 1110'698 (0'0,1110'698] local-les=3754 n=698 ec=633 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] -21> 2014-04-25 09:18:54.741023 7f77edce2700 10 log is not dirty -20> 2014-04-25 09:18:54.741027 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) -19> 2014-04-25 09:18:54.741030 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] -18> 2014-04-25 09:18:54.741038 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] recovery raced and were queued twice, ignoring! -17> 2014-04-25 09:18:54.741048 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[4.53ad( v 1173'1411 (0'0,1173'1411] local-les=3754 n=1411 ec=638 les/c 3754/3754 3753/3753/3753) [210,409] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+degraded] -16> 2014-04-25 09:18:54.741057 7f77edce2700 10 log is not dirty -15> 2014-04-25 09:18:54.741060 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) -14> 2014-04-25 09:18:54.741063 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] -13> 2014-04-25 09:18:54.741073 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] recovery raced and were queued twice, ignoring! -12> 2014-04-25 09:18:54.741083 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[4.42d9( v 1201'1389 (0'0,1201'1389] local-les=3764 n=1388 ec=638 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] -11> 2014-04-25 09:18:54.741091 7f77edce2700 10 log is not dirty -10> 2014-04-25 09:18:54.741094 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) -9> 2014-04-25 09:18:54.741097 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] -8> 2014-04-25 09:18:54.741107 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] recovery raced and were queued twice, ignoring! -7> 2014-04-25 09:18:54.741118 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[3.42da( v 1110'672 (0'0,1110'672] local-les=3764 n=672 ec=633 les/c 3764/3427 3753/3753/3753) [210,109,249] r=0 lpr=3753 pi=3426-3752/11 lcod 0'0 mlcod 0'0 active+recovery_wait] -6> 2014-04-25 09:18:54.741127 7f77edce2700 10 log is not dirty -5> 2014-04-25 09:18:54.742944 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[4.5e1c( v 1110'1412 (0'0,1110'1412] local-les=3769 n=1412 ec=638 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] flushed -4> 2014-04-25 09:18:54.742828 7f77c2455700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -3> 2014-04-25 09:18:54.743077 7f77c1546700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -2> 2014-04-25 09:18:54.743213 7f77df778700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1> 2014-04-25 09:18:54.744672 7f77c1546700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.179:6817/6097831 pipe(0x1d26bb80 sd=62 :53522 s=1 pgs=0 cs=0 l=0 c=0x13275e40). got newly_acked_seq 35 vs out_seq 0 0> 2014-04-25 09:18:54.745908 7f77c1546700 -1 msg/Pipe.cc: In function 'int Pipe::connect()' thread 7f77c1546700 time 2014-04-25 09:18:54.744693 msg/Pipe.cc: 1045: FAILED assert(m) ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73) 1: (Pipe::connect()+0x3cb5) [0x992435] 2: (Pipe::writer()+0x5f3) [0x992d83] 3: (Pipe::Writer::entry()+0xd) [0x99ddfd] 4: /lib64/libpthread.so.0() [0x3a6e407851] 5: (clone()+0x6d) [0x3a6e0e890d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 10/10 osd 0/ 5 optracker 0/ 5 objclass 3/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 0/ 0 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 3/ 3 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /home/y/logs/ceph/ceph-osd.210.log --- end dump of recent events --- 2014-04-25 09:18:54.750078 7f77d0f45700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.750180 7f77db738700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.752180 7f77e7ab6700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.752284 7f77ca5dc700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.752451 7f77d9112700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.752600 7f779a5df700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.752704 7f77cb3ea700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.735150 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000204 1 0.000304 2014-04-25 09:18:54.752831 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.752845 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.752855 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.752869 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000025 0 0.000000 2014-04-25 09:18:54.752881 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.752892 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.752903 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.752915 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:54.752928 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) 2014-04-25 09:18:54.752941 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.14 is down 2014-04-25 09:18:54.752951 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 7,210 down 14 blocked_by {} 2014-04-25 09:18:54.752963 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.752976 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.7 2014-04-25 09:18:54.752991 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.753031 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:54.753492 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_advance_map [341,210]/[341,210] 2014-04-25 09:18:54.753520 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.753532 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.753545 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 171.868237 132 0.002767 2014-04-25 09:18:54.753560 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 171.868270 0 0.000000 2014-04-25 09:18:54.753573 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started 175.595095 0 0.000000 2014-04-25 09:18:54.753585 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.753597 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.753607 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3120 2014-04-25 09:18:54.753620 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] dropping down osd.204 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) 2014-04-25 09:18:54.753634 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged 2014-04-25 09:18:54.753645 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.753656 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.753678 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [341,210,204]/[341,210,204]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3120 generate_past_intervals interval(3753-3798 [341,210,204]/[341,210,204] maybe_went_rw) : primary up 3720-3796 includes interval 2014-04-25 09:18:54.753691 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [341,210,204]/[341,210,204] maybe_went_rw) 2014-04-25 09:18:54.753704 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] up [341,210,204] -> [341,210], acting [341,210,204] -> [341,210], role 1 -> 1 2014-04-25 09:18:54.753716 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] on_change 2014-04-25 09:18:54.753733 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.753744 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.753757 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.753769 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] update_heartbeat_peers unchanged 2014-04-25 09:18:54.753779 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.753790 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] exit Reset 0.000205 1 0.000239 2014-04-25 09:18:54.753802 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Started 2014-04-25 09:18:54.753813 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Start 2014-04-25 09:18:54.753823 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] state: transitioning to Stray 2014-04-25 09:18:54.753834 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:54.753847 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Started/Stray 2014-04-25 09:18:54.753858 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.753919 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.753990 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_advance_map [341,210]/[341,210] 2014-04-25 09:18:54.754011 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.754022 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.754034 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 171.857275 132 0.002394 2014-04-25 09:18:54.754048 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 171.857306 0 0.000000 2014-04-25 09:18:54.754061 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started 175.593812 0 0.000000 2014-04-25 09:18:54.754074 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.754086 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.754096 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3120 2014-04-25 09:18:54.754107 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] dropping down osd.204 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) 2014-04-25 09:18:54.754122 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged 2014-04-25 09:18:54.754133 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.754143 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.754172 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [341,210,204]/[341,210,204]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3120 generate_past_intervals interval(3753-3798 [341,210,204]/[341,210,204] maybe_went_rw) : primary up 3720-3796 includes interval 2014-04-25 09:18:54.754183 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [341,210,204]/[341,210,204] maybe_went_rw) 2014-04-25 09:18:54.754197 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] up [341,210,204] -> [341,210], acting [341,210,204] -> [341,210], role 1 -> 1 2014-04-25 09:18:54.754209 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] on_change 2014-04-25 09:18:54.754224 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.754234 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.754246 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.754258 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] update_heartbeat_peers unchanged 2014-04-25 09:18:54.754267 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.754278 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] exit Reset 0.000204 1 0.000236 2014-04-25 09:18:54.754292 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Started 2014-04-25 09:18:54.754303 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Start 2014-04-25 09:18:54.754313 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] state: transitioning to Stray 2014-04-25 09:18:54.754324 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] exit Start 0.000022 0 0.000000 2014-04-25 09:18:54.754337 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Started/Stray 2014-04-25 09:18:54.754348 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.754407 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.754435 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] handle_advance_map [210]/[210] 2014-04-25 09:18:54.754454 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.754466 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.754479 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 137.660129 67 0.001276 2014-04-25 09:18:54.754491 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 137.660160 0 0.000000 2014-04-25 09:18:54.754503 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started 169.026816 0 0.000000 2014-04-25 09:18:54.754515 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.754527 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.754537 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3777 2014-04-25 09:18:54.754549 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] dropping down osd.33 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) 2014-04-25 09:18:54.754563 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged 2014-04-25 09:18:54.754573 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.754583 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.754605 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3757-3798 [359,33,210]/[359,33,210]): not rw, up_thru 3796 up_from 3757 last_epoch_clean 3777 generate_past_intervals interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) : primary up 3757-3796 includes interval 2014-04-25 09:18:54.754617 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) 2014-04-25 09:18:54.754629 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] up [359,33,210] -> [210], acting [359,33,210] -> [210], role 2 -> 0 2014-04-25 09:18:54.754641 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.754652 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.754668 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.754679 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.754689 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.754702 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.754713 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 2014-04-25 09:18:54.754723 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.754734 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000219 1 0.000249 2014-04-25 09:18:54.754746 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.754757 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.754767 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.754778 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:54.754790 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.754801 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.754812 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.754822 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3777 2014-04-25 09:18:54.754833 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) 2014-04-25 09:18:54.754845 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.359 is down 2014-04-25 09:18:54.754855 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.33 is down 2014-04-25 09:18:54.754865 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3756 [33,210]/[33,210] maybe_went_rw) 2014-04-25 09:18:54.754877 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 33,359 blocked_by {} 2014-04-25 09:18:54.754889 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.754903 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000090 0 0.000000 2014-04-25 09:18:54.754918 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog 2014-04-25 09:18:54.754933 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) 2014-04-25 09:18:54.754946 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) 2014-04-25 09:18:54.754960 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary 2014-04-25 09:18:54.754969 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) 2014-04-25 09:18:54.754984 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) 2014-04-25 09:18:54.754996 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-04-25 09:18:54.755010 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000091 0 0.000000 2014-04-25 09:18:54.755023 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing 2014-04-25 09:18:54.755033 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active 2014-04-25 09:18:54.755048 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000025 0 0.000000 2014-04-25 09:18:54.755061 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru 2014-04-25 09:18:54.755073 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.755140 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.755174 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_advance_map [444,210,301]/[444,210,301] 2014-04-25 09:18:54.755193 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.755206 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.755218 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.755230 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.755241 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.755255 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.755297 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_advance_map [444,210,301]/[444,210,301] 2014-04-25 09:18:54.755316 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.755328 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.755341 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.755351 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.755363 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.755376 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.755391 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] handle_advance_map [210,171]/[210,171] 2014-04-25 09:18:54.755408 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.755419 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.755431 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 128.236795 58 0.001323 2014-04-25 09:18:54.755444 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 128.236827 0 0.000000 2014-04-25 09:18:54.755456 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started 158.124719 0 0.000000 2014-04-25 09:18:54.755468 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.755481 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.755492 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3780 2014-04-25 09:18:54.755503 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.755513 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.755534 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3764-3798 [435,210,171]/[435,210,171]): not rw, up_thru 3796 up_from 3764 last_epoch_clean 3780 generate_past_intervals interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) : primary up 3764-3796 includes interval 2014-04-25 09:18:54.755545 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) 2014-04-25 09:18:54.755557 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] up [435,210,171] -> [210,171], acting [435,210,171] -> [210,171], role 1 -> 0 2014-04-25 09:18:54.755569 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.755582 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.755598 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.755608 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.755618 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.755630 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.755642 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 171,210 2014-04-25 09:18:54.755652 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.755663 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000195 1 0.000222 2014-04-25 09:18:54.755675 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.755685 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.755696 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.755708 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000023 0 0.000000 2014-04-25 09:18:54.755720 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.755731 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.755742 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.755752 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3780 2014-04-25 09:18:54.755764 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) 2014-04-25 09:18:54.755775 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down 2014-04-25 09:18:54.755785 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3761-3763 [210,171]/[210,171] maybe_went_rw) 2014-04-25 09:18:54.755797 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 171,210 down 435 blocked_by {} 2014-04-25 09:18:54.755809 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.755822 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.171 2014-04-25 09:18:54.755836 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.755870 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.755894 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] handle_advance_map [210]/[210] 2014-04-25 09:18:54.755913 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.755925 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.755937 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 137.656590 67 0.001228 2014-04-25 09:18:54.755950 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 137.656619 0 0.000000 2014-04-25 09:18:54.755962 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started 169.025859 0 0.000000 2014-04-25 09:18:54.755975 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.755986 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.755997 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3777 2014-04-25 09:18:54.756009 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] dropping down osd.33 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) 2014-04-25 09:18:54.756023 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged 2014-04-25 09:18:54.756032 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.756043 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.756067 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3757-3798 [359,33,210]/[359,33,210]): not rw, up_thru 3796 up_from 3757 last_epoch_clean 3777 generate_past_intervals interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) : primary up 3757-3796 includes interval 2014-04-25 09:18:54.756079 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) 2014-04-25 09:18:54.756091 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] up [359,33,210] -> [210], acting [359,33,210] -> [210], role 2 -> 0 2014-04-25 09:18:54.756103 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.756115 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.756131 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.756143 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.756152 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.756171 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.756183 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 2014-04-25 09:18:54.756194 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.756204 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000230 1 0.000259 2014-04-25 09:18:54.756217 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.756228 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.756238 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.756248 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 2014-04-25 09:18:54.756260 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.756271 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.756282 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.756292 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3777 2014-04-25 09:18:54.756303 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) 2014-04-25 09:18:54.756315 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.359 is down 2014-04-25 09:18:54.756325 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.33 is down 2014-04-25 09:18:54.756336 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3756 [33,210]/[33,210] maybe_went_rw) 2014-04-25 09:18:54.756348 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 33,359 blocked_by {} 2014-04-25 09:18:54.756359 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.756372 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000090 0 0.000000 2014-04-25 09:18:54.756386 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog 2014-04-25 09:18:54.756398 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) 2014-04-25 09:18:54.756412 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) 2014-04-25 09:18:54.756426 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary 2014-04-25 09:18:54.756435 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) 2014-04-25 09:18:54.756449 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) 2014-04-25 09:18:54.756461 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-04-25 09:18:54.756474 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000089 0 0.000000 2014-04-25 09:18:54.756487 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing 2014-04-25 09:18:54.756498 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active 2014-04-25 09:18:54.756511 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000025 0 0.000000 2014-04-25 09:18:54.756524 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru 2014-04-25 09:18:54.756535 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.756602 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.756634 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] handle_advance_map [210]/[210] 2014-04-25 09:18:54.756653 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: Started advmap 2014-04-25 09:18:54.756664 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.756676 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] exit Started/Stray 12.671522 3 0.000234 2014-04-25 09:18:54.756688 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] exit Started 12.671578 0 0.000000 2014-04-25 09:18:54.756701 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] enter Reset 2014-04-25 09:18:54.756712 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: Reset advmap 2014-04-25 09:18:54.756721 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:54.756733 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] dropping down osd.435 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) 2014-04-25 09:18:54.756747 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] update_heartbeat_peers unchanged 2014-04-25 09:18:54.756757 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.756767 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.756789 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] start_peering_interval: check_new_interval output: generate_past_intervals interval(3798-3798 [435,210]/[435,210]): not rw, up_thru 3796 up_from 3764 last_epoch_clean 3715 generate_past_intervals interval(3798-3798 [435,210]/[435,210]) : primary up 3764-3796 does not include interval 2014-04-25 09:18:54.756801 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] noting past interval(3798-3798 [435,210]/[435,210]) 2014-04-25 09:18:54.756813 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] up [435,210] -> [210], acting [435,210] -> [210], role 1 -> 0 2014-04-25 09:18:54.756824 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] clear_primary_state 2014-04-25 09:18:54.756835 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] on_change 2014-04-25 09:18:54.756851 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] on_role_change 2014-04-25 09:18:54.756861 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.756871 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.756882 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.756894 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 2014-04-25 09:18:54.756904 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.756916 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000215 1 0.000231 2014-04-25 09:18:54.756928 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.756938 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.756951 7f77d8a0b700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.756948 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.756959 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:54.756971 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.756981 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.756991 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.757001 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:54.757013 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3798-3798 [435,210]/[435,210]) 2014-04-25 09:18:54.757024 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3797 [306,435,210]/[306,435,210] maybe_went_rw) 2014-04-25 09:18:54.757036 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.306 is down 2014-04-25 09:18:54.757046 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down 2014-04-25 09:18:54.757056 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3761-3763 [306,210]/[306,210]) 2014-04-25 09:18:54.757067 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 306,435 blocked_by {} 2014-04-25 09:18:54.757078 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.757091 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000100 0 0.000000 2014-04-25 09:18:54.757103 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog 2014-04-25 09:18:54.757116 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) 2014-04-25 09:18:54.757130 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) 2014-04-25 09:18:54.757143 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary 2014-04-25 09:18:54.757153 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) 2014-04-25 09:18:54.757186 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) 2014-04-25 09:18:54.757199 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-04-25 09:18:54.757212 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000108 0 0.000000 2014-04-25 09:18:54.757224 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing 2014-04-25 09:18:54.757235 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active 2014-04-25 09:18:54.757247 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000023 0 0.000000 2014-04-25 09:18:54.757260 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru 2014-04-25 09:18:54.757270 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.757331 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.757358 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,80,105]/[210,80,105] 2014-04-25 09:18:54.757379 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:54.757393 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.757405 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:54.757416 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.757428 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map 2014-04-25 09:18:54.757439 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap 2014-04-25 09:18:54.757467 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:54.757476 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary 2014-04-25 09:18:54.757491 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters 2014-04-25 09:18:54.757502 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.757516 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.757560 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] handle_advance_map [210,171]/[210,171] 2014-04-25 09:18:54.757581 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.757592 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.757605 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 128.096077 58 0.001418 2014-04-25 09:18:54.757618 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 128.096111 0 0.000000 2014-04-25 09:18:54.757630 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started 158.123644 0 0.000000 2014-04-25 09:18:54.757642 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.757654 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.757664 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3780 2014-04-25 09:18:54.757677 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.757687 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.757707 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3764-3798 [435,210,171]/[435,210,171]): not rw, up_thru 3796 up_from 3764 last_epoch_clean 3780 generate_past_intervals interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) : primary up 3764-3796 includes interval 2014-04-25 09:18:54.757720 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) 2014-04-25 09:18:54.757732 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] up [435,210,171] -> [210,171], acting [435,210,171] -> [210,171], role 1 -> 0 2014-04-25 09:18:54.757745 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.757756 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.757773 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.757783 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.757793 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.757805 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.757817 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 171,210 2014-04-25 09:18:54.757828 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.757839 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000197 1 0.000225 2014-04-25 09:18:54.757851 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.757862 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.757872 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.757882 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 2014-04-25 09:18:54.757895 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.757905 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.757917 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.757928 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3780 2014-04-25 09:18:54.757940 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) 2014-04-25 09:18:54.757952 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down 2014-04-25 09:18:54.757962 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3761-3763 [210,171]/[210,171] maybe_went_rw) 2014-04-25 09:18:54.757974 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 171,210 down 435 blocked_by {} 2014-04-25 09:18:54.757986 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.757998 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.171 2014-04-25 09:18:54.758012 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.758047 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.758073 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,452,445]/[210,452,445] 2014-04-25 09:18:54.758092 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.758104 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758114 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.758125 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758135 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:54.758246 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_advance_map [365,447,210]/[365,447,210] 2014-04-25 09:18:54.758271 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.758283 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758297 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.758308 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.758319 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.758332 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.758373 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_advance_map [88,428,210]/[88,428,210] 2014-04-25 09:18:54.758394 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.758406 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758418 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.758429 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.758440 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.758454 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.758489 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_advance_map [53,101,210]/[53,101,210] 2014-04-25 09:18:54.758508 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.758520 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758532 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.758543 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.758555 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.758568 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.758583 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_advance_map [365,447,210]/[365,447,210] 2014-04-25 09:18:54.758599 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.758611 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758623 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.758635 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.758645 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.758659 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.758673 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_advance_map [88,428,210]/[88,428,210] 2014-04-25 09:18:54.758690 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.758701 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758714 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.758724 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.758737 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.758750 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.758764 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,102,268]/[210,102,268] 2014-04-25 09:18:54.758781 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.758793 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758803 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.758813 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758823 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:54.758834 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:54.758844 7f77f18e8700 7 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:54.758856 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:54.758867 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.758879 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.758918 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,102,268]/[210,102,268] 2014-04-25 09:18:54.758938 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.758950 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758961 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.758971 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.758982 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:54.758992 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:54.759003 7f77f18e8700 7 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:54.759015 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:54.759026 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.759039 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.759054 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_advance_map [43,210]/[43,210] 2014-04-25 09:18:54.759071 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.759082 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.759094 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.759105 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.759116 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.759130 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.759171 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,200]/[210,200] 2014-04-25 09:18:54.759202 7f77af1a7700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.759195 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:54.759221 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.400 info 3.256a( v 1110'642 lc 0'0 (0'0,1110'642] local-les=3747 n=642 ec=633 les/c 3754/3754 3755/3755/3753) 2014-04-25 09:18:54.759294 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 200,210,400 unchanged 2014-04-25 09:18:54.759307 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.759319 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:54.759329 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.759342 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 161.951519 115 0.005363 2014-04-25 09:18:54.759356 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 161.951998 0 0.000000 2014-04-25 09:18:54.759371 7f77df576700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.759375 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started/Primary 172.069340 0 0.000000 2014-04-25 09:18:54.759387 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started 172.069416 0 0.000000 2014-04-25 09:18:54.759399 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] enter Reset 2014-04-25 09:18:54.759412 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.759422 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:54.759434 7f77e9fdb700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.759435 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.759445 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.759468 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3755-3798 [210,200,400]/[210,200,400]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3755-3798 [210,200,400]/[210,200,400] maybe_went_rw) : primary up 3753-3797 includes interval 2014-04-25 09:18:54.759480 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] noting past interval(3755-3798 [210,200,400]/[210,200,400] maybe_went_rw) 2014-04-25 09:18:54.759491 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] up [210,200,400] -> [210,200], acting [210,200,400] -> [210,200], role 0 -> 0 2014-04-25 09:18:54.759504 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.759516 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.759532 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] [210,200,400] -> [210,200], replicas changed 2014-04-25 09:18:54.759543 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.759553 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.759566 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.759577 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 200,210,400 -> 200,210 2014-04-25 09:18:54.759589 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.759600 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000201 1 0.000371 2014-04-25 09:18:54.759612 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.759622 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.759632 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.759643 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 2014-04-25 09:18:54.759654 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.759665 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.759676 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.759686 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:54.759698 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3755-3798 [210,200,400]/[210,200,400] maybe_went_rw) 2014-04-25 09:18:54.759710 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.400 is down 2014-04-25 09:18:54.759720 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3754 [210,200]/[210,200] maybe_went_rw) 2014-04-25 09:18:54.759731 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 200,210 down 400 blocked_by {} 2014-04-25 09:18:54.759742 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.759754 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.200 2014-04-25 09:18:54.759768 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.759805 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.759833 7f77f22e9700 10 osd.210 pg_epoch: 3798 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] handle_advance_map [196,210,282]/[196,210,282] 2014-04-25 09:18:54.759857 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.759869 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.759883 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.759894 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.759905 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.759919 7f77f22e9700 10 log is not dirty 2014-04-25 09:18:54.759938 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) 2014-04-25 09:18:54.759942 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:54.759955 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] recovery raced and were queued twice, ignoring! 2014-04-25 09:18:54.759969 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:54.759978 7f77edce2700 10 log is not dirty 2014-04-25 09:18:54.761102 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] flushed 2014-04-25 09:18:54.761131 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] flushed 2014-04-25 09:18:54.761153 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] flushed 2014-04-25 09:18:54.761288 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] handle_advance_map [7,210]/[7,210] 2014-04-25 09:18:54.761313 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.761325 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.761338 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 171.342455 133 0.002256 2014-04-25 09:18:54.761353 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 171.342482 0 0.000000 2014-04-25 09:18:54.761367 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] exit Started 175.621415 0 0.000000 2014-04-25 09:18:54.761380 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.761391 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.761402 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3755 2014-04-25 09:18:54.761415 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] dropping down osd.179 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) 2014-04-25 09:18:54.761430 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged 2014-04-25 09:18:54.761441 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.761452 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.761473 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [7,179,210]/[7,179,210]): not rw, up_thru 3796 up_from 3717 last_epoch_clean 3755 generate_past_intervals interval(3753-3798 [7,179,210]/[7,179,210] maybe_went_rw) : primary up 3717-3796 includes interval 2014-04-25 09:18:54.761487 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [7,179,210]/[7,179,210] maybe_went_rw) 2014-04-25 09:18:54.761498 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 luod=0'0 lcod 0'0 active] up [7,179,210] -> [7,210], acting [7,179,210] -> [7,210], role 2 -> 1 2014-04-25 09:18:54.761511 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive] on_change 2014-04-25 09:18:54.761526 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.761536 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] cancel_recovery 2014-04-25 09:18:54.761546 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] clear_recovery_state 2014-04-25 09:18:54.761559 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] handle_activate_map 2014-04-25 09:18:54.761645 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] update_heartbeat_peers unchanged 2014-04-25 09:18:54.761656 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] take_waiters 2014-04-25 09:18:54.761667 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] exit Reset 0.000287 1 0.000248 2014-04-25 09:18:54.761679 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] enter Started 2014-04-25 09:18:54.761689 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] enter Start 2014-04-25 09:18:54.761699 7f77efae5700 1 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] state: transitioning to Stray 2014-04-25 09:18:54.761710 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] exit Start 0.000021 0 0.000000 2014-04-25 09:18:54.761723 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] enter Started/Stray 2014-04-25 09:18:54.761735 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.761792 7f77efae5700 10 log is not dirty 2014-04-25 09:18:54.761835 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,309,198]/[210,309,198] 2014-04-25 09:18:54.761856 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:54.761870 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.761883 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:54.761893 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.761907 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map 2014-04-25 09:18:54.761918 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap 2014-04-25 09:18:54.761962 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) 2014-04-25 09:18:54.761968 7f77ef0e4700 10 osd.210 3799 queue_for_recovery queued pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:54.761979 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary 2014-04-25 09:18:54.761993 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters 2014-04-25 09:18:54.762005 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.762019 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:54.758147 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:54.762025 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:54.762040 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:54.762051 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.762064 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.762835 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] handle_advance_map [450,252,210]/[450,252,210] 2014-04-25 09:18:54.762859 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.762871 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.762883 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.762895 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.762906 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.762921 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.762960 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_advance_map [92,210]/[92,210] 2014-04-25 09:18:54.762980 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] state: Started advmap 2014-04-25 09:18:54.762991 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.763002 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.763013 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.763023 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.763037 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.763070 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_advance_map [43,210]/[43,210] 2014-04-25 09:18:54.763089 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.763101 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.763112 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.763124 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.763135 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.763148 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.763169 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_advance_map [92,210]/[92,210] 2014-04-25 09:18:54.763187 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] state: Started advmap 2014-04-25 09:18:54.763199 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.763210 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.763220 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.763231 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.763244 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.763258 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] handle_advance_map [450,252,210]/[450,252,210] 2014-04-25 09:18:54.763275 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.763287 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.763298 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.763309 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.763320 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.763333 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.763347 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] handle_advance_map [210]/[210] 2014-04-25 09:18:54.763363 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.763374 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.763387 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 172.545697 135 0.002914 2014-04-25 09:18:54.763401 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 172.545731 0 0.000000 2014-04-25 09:18:54.763414 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started 177.343139 0 0.000000 2014-04-25 09:18:54.763426 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.763438 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.763448 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:54.763459 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] dropping down osd.14 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) 2014-04-25 09:18:54.763473 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged 2014-04-25 09:18:54.763484 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.763495 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.763518 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [184,14,210]/[184,14,210]): not rw, up_thru 3796 up_from 3717 last_epoch_clean 3715 generate_past_intervals interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) : primary up 3717-3796 includes interval 2014-04-25 09:18:54.763532 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) 2014-04-25 09:18:54.763544 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] up [184,14,210] -> [210], acting [184,14,210] -> [210], role 2 -> 0 2014-04-25 09:18:54.763556 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.763567 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.763583 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.763594 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.763604 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.763617 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.763628 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 2014-04-25 09:18:54.763639 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.763650 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000223 1 0.000254 2014-04-25 09:18:54.763661 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.763672 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.763682 7f77f18e8700 1 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.763694 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000022 0 0.000000 2014-04-25 09:18:54.763706 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.763717 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.763728 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.763738 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:54.763750 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) 2014-04-25 09:18:54.763762 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.184 is down 2014-04-25 09:18:54.763772 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.14 is down 2014-04-25 09:18:54.763782 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [184,14]/[184,14]) 2014-04-25 09:18:54.763794 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 14,184 blocked_by {} 2014-04-25 09:18:54.763805 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.763818 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000091 0 0.000000 2014-04-25 09:18:54.763833 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog 2014-04-25 09:18:54.763846 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) 2014-04-25 09:18:54.763860 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) 2014-04-25 09:18:54.763875 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary 2014-04-25 09:18:54.763884 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) 2014-04-25 09:18:54.763898 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) 2014-04-25 09:18:54.763910 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-04-25 09:18:54.763924 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000091 0 0.000000 2014-04-25 09:18:54.763938 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing 2014-04-25 09:18:54.763948 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active 2014-04-25 09:18:54.763961 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000023 0 0.000000 2014-04-25 09:18:54.763973 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru 2014-04-25 09:18:54.763984 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.764033 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.764072 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] handle_advance_map [210]/[210] 2014-04-25 09:18:54.764091 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.764102 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.764114 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 173.178391 135 0.002388 2014-04-25 09:18:54.764127 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 173.178420 0 0.000000 2014-04-25 09:18:54.764139 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started 177.340467 0 0.000000 2014-04-25 09:18:54.764151 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.764170 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.764181 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:54.764192 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] dropping down osd.14 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) 2014-04-25 09:18:54.764207 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged 2014-04-25 09:18:54.764217 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.764227 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.764250 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [184,14,210]/[184,14,210]): not rw, up_thru 3796 up_from 3717 last_epoch_clean 3715 generate_past_intervals interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) : primary up 3717-3796 includes interval 2014-04-25 09:18:54.764263 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) 2014-04-25 09:18:54.764275 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] up [184,14,210] -> [210], acting [184,14,210] -> [210], role 2 -> 0 2014-04-25 09:18:54.764287 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.764298 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.764314 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.764324 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.764334 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.764353 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.764364 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 2014-04-25 09:18:54.764374 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.764384 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000233 1 0.000262 2014-04-25 09:18:54.764396 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.764406 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.764416 7f77f18e8700 1 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.764427 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 2014-04-25 09:18:54.764438 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.764448 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.764458 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.764469 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:54.764480 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) 2014-04-25 09:18:54.764492 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.184 is down 2014-04-25 09:18:54.764502 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.14 is down 2014-04-25 09:18:54.764512 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [184,14]/[184,14]) 2014-04-25 09:18:54.764524 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 14,184 blocked_by {} 2014-04-25 09:18:54.764535 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.764547 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000089 0 0.000000 2014-04-25 09:18:54.764560 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog 2014-04-25 09:18:54.764573 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) 2014-04-25 09:18:54.764587 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) 2014-04-25 09:18:54.764599 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary 2014-04-25 09:18:54.764609 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) 2014-04-25 09:18:54.764623 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) 2014-04-25 09:18:54.764634 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-04-25 09:18:54.764647 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000087 0 0.000000 2014-04-25 09:18:54.764660 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing 2014-04-25 09:18:54.764670 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active 2014-04-25 09:18:54.764684 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000024 0 0.000000 2014-04-25 09:18:54.764696 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru 2014-04-25 09:18:54.764707 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.764757 7f77f18e8700 10 log is not dirty 2014-04-25 09:18:54.766893 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] handle_advance_map [210,434]/[210,434] 2014-04-25 09:18:54.766919 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.766930 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.766944 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 170.066336 129 0.002302 2014-04-25 09:18:54.766957 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 170.066366 0 0.000000 2014-04-25 09:18:54.766970 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started 177.339327 0 0.000000 2014-04-25 09:18:54.766981 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.766993 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.767008 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3718 2014-04-25 09:18:54.767020 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.767030 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.767052 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [129,210,434]/[129,210,434]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3718 generate_past_intervals interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) : primary up 3720-3796 includes interval 2014-04-25 09:18:54.767064 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) 2014-04-25 09:18:54.767076 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] up [129,210,434] -> [210,434], acting [129,210,434] -> [210,434], role 1 -> 0 2014-04-25 09:18:54.767089 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.767101 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.767119 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.767129 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.767138 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.767150 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.767169 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210,434 2014-04-25 09:18:54.767180 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.767191 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000209 1 0.000233 2014-04-25 09:18:54.767203 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.767214 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.767224 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.767235 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:54.767249 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.767259 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.767270 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.767282 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3718 2014-04-25 09:18:54.767294 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) 2014-04-25 09:18:54.767306 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.129 is down 2014-04-25 09:18:54.767316 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [129,434]/[129,434]) 2014-04-25 09:18:54.767327 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,434 down 129 blocked_by {} 2014-04-25 09:18:54.767339 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.767351 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.434 2014-04-25 09:18:54.767365 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.767406 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.767459 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] handle_advance_map [210,434]/[210,434] 2014-04-25 09:18:54.767479 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.767490 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.767502 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 170.066516 129 0.002459 2014-04-25 09:18:54.767515 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 170.066545 0 0.000000 2014-04-25 09:18:54.767527 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started 177.338797 0 0.000000 2014-04-25 09:18:54.767540 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.767551 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.767561 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3718 2014-04-25 09:18:54.767573 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.767583 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.767605 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [129,210,434]/[129,210,434]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3718 generate_past_intervals interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) : primary up 3720-3796 includes interval 2014-04-25 09:18:54.767617 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) 2014-04-25 09:18:54.767629 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] up [129,210,434] -> [210,434], acting [129,210,434] -> [210,434], role 1 -> 0 2014-04-25 09:18:54.767641 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.767652 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.767668 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.767681 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.767690 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.767702 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.767720 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210,434 2014-04-25 09:18:54.767731 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.767742 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000202 1 0.000224 2014-04-25 09:18:54.767754 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.767763 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.767773 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.767785 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000022 0 0.000000 2014-04-25 09:18:54.767798 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.767808 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.767818 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.767828 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3718 2014-04-25 09:18:54.767840 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) 2014-04-25 09:18:54.767852 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.129 is down 2014-04-25 09:18:54.767861 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [129,434]/[129,434]) 2014-04-25 09:18:54.767874 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,434 down 129 blocked_by {} 2014-04-25 09:18:54.767885 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.767896 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.434 2014-04-25 09:18:54.767910 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.767951 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.767981 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,62,7]/[210,62,7] 2014-04-25 09:18:54.768000 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.768012 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.768023 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.768033 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.768043 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:54.768054 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:54.768064 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:54.768075 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:54.768086 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.768099 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.768116 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210]/[210] 2014-04-25 09:18:54.768134 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.768146 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.243 info 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3749 n=1426 ec=638 les/c 3718/3718 3764/3764/3753) 2014-04-25 09:18:54.768167 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.435 info 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3754 n=1426 ec=638 les/c 3754/3754 3764/3764/3753) 2014-04-25 09:18:54.768184 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 210,243,435 unchanged 2014-04-25 09:18:54.768195 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.768205 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.768214 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.768226 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 125.984272 55 0.002052 2014-04-25 09:18:54.768238 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 156.737850 0 0.000000 2014-04-25 09:18:54.768252 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started/Primary 158.279280 0 0.000000 2014-04-25 09:18:54.768264 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started 158.279344 0 0.000000 2014-04-25 09:18:54.768276 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] enter Reset 2014-04-25 09:18:54.768292 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.768307 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3780 >= end epoch 3764, nothing to do 2014-04-25 09:18:54.768318 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.768327 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.768347 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3764-3798 [210,243,435]/[210,243,435]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3780 generate_past_intervals interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) : primary up 3753-3797 includes interval 2014-04-25 09:18:54.768359 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) 2014-04-25 09:18:54.768370 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] up [210,243,435] -> [210], acting [210,243,435] -> [210], role 0 -> 0 2014-04-25 09:18:54.768382 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.768393 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.768409 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,243,435] -> [210], replicas changed 2014-04-25 09:18:54.768420 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.768430 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.768442 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.768453 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 210,243,435 -> 210 2014-04-25 09:18:54.768464 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.768474 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000199 1 0.000308 2014-04-25 09:18:54.768486 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.768496 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.768505 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.768516 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:54.768528 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.768538 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.768548 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.768558 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3780 2014-04-25 09:18:54.768570 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) 2014-04-25 09:18:54.768583 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.243 is down 2014-04-25 09:18:54.768592 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down 2014-04-25 09:18:54.768602 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 243,435 blocked_by {} 2014-04-25 09:18:54.768613 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.768626 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000079 0 0.000000 2014-04-25 09:18:54.768640 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog 2014-04-25 09:18:54.768652 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) 2014-04-25 09:18:54.768665 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) 2014-04-25 09:18:54.768679 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary 2014-04-25 09:18:54.768688 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) 2014-04-25 09:18:54.768702 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) 2014-04-25 09:18:54.768714 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-04-25 09:18:54.768727 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000088 0 0.000000 2014-04-25 09:18:54.768740 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing 2014-04-25 09:18:54.768750 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active 2014-04-25 09:18:54.768763 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000024 0 0.000000 2014-04-25 09:18:54.768775 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru 2014-04-25 09:18:54.768786 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.768818 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.768844 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210]/[210] 2014-04-25 09:18:54.768862 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.768873 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.243 info 3.2863( v 1110'677 (0'0,1110'677] local-les=3749 n=677 ec=633 les/c 3718/3718 3764/3764/3753) 2014-04-25 09:18:54.768894 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.435 info 3.2863( v 1110'677 (0'0,1110'677] local-les=3754 n=677 ec=633 les/c 3754/3754 3764/3764/3753) 2014-04-25 09:18:54.768910 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 210,243,435 unchanged 2014-04-25 09:18:54.768922 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.768932 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.768942 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.768953 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 125.984903 55 0.002070 2014-04-25 09:18:54.768966 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 156.737979 0 0.000000 2014-04-25 09:18:54.768979 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started/Primary 158.278308 0 0.000000 2014-04-25 09:18:54.768990 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started 158.278372 0 0.000000 2014-04-25 09:18:54.769002 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] enter Reset 2014-04-25 09:18:54.769012 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.769024 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3780 >= end epoch 3764, nothing to do 2014-04-25 09:18:54.769034 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.769044 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.769064 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3764-3798 [210,243,435]/[210,243,435]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3780 generate_past_intervals interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) : primary up 3753-3797 includes interval 2014-04-25 09:18:54.769075 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) 2014-04-25 09:18:54.769087 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] up [210,243,435] -> [210], acting [210,243,435] -> [210], role 0 -> 0 2014-04-25 09:18:54.769098 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.769110 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.769125 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,243,435] -> [210], replicas changed 2014-04-25 09:18:54.769138 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.769147 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.769171 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.769183 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 210,243,435 -> 210 2014-04-25 09:18:54.769194 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.769204 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000202 1 0.000310 2014-04-25 09:18:54.769216 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.769226 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.769235 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.769246 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:54.769258 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.769269 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.769280 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.769289 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3780 2014-04-25 09:18:54.769302 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) 2014-04-25 09:18:54.769313 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.243 is down 2014-04-25 09:18:54.769324 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down 2014-04-25 09:18:54.769333 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 243,435 blocked_by {} 2014-04-25 09:18:54.769345 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.769358 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000078 0 0.000000 2014-04-25 09:18:54.769370 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog 2014-04-25 09:18:54.769382 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) 2014-04-25 09:18:54.769396 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) 2014-04-25 09:18:54.769409 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary 2014-04-25 09:18:54.769419 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) 2014-04-25 09:18:54.769438 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) 2014-04-25 09:18:54.769449 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-04-25 09:18:54.769463 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000093 0 0.000000 2014-04-25 09:18:54.769474 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing 2014-04-25 09:18:54.769484 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active 2014-04-25 09:18:54.769496 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000022 0 0.000000 2014-04-25 09:18:54.769508 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru 2014-04-25 09:18:54.769519 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.769553 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.769579 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,430,74]/[210,430,74] 2014-04-25 09:18:54.769597 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.769609 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.769620 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.769630 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.769640 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:54.769651 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:54.769661 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:54.769673 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:54.769684 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.769696 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.769738 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,430,74]/[210,430,74] 2014-04-25 09:18:54.769758 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.769770 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.769780 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.769790 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.769800 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:54.769810 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:54.769820 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:54.769831 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:54.769842 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.769854 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.769870 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] handle_advance_map [210,195]/[210,195] 2014-04-25 09:18:54.769886 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.769897 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.769910 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 173.320337 136 0.002477 2014-04-25 09:18:54.769924 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 173.320371 0 0.000000 2014-04-25 09:18:54.769937 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started 177.276108 0 0.000000 2014-04-25 09:18:54.769952 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.769964 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.769975 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:54.769986 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.769996 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.770019 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [33,210,195]/[33,210,195]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3754 generate_past_intervals interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) : primary up 3720-3796 includes interval 2014-04-25 09:18:54.770031 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) 2014-04-25 09:18:54.770043 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] up [33,210,195] -> [210,195], acting [33,210,195] -> [210,195], role 1 -> 0 2014-04-25 09:18:54.770056 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.770068 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.770084 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.770099 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.770108 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.770120 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.770130 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 195,210 2014-04-25 09:18:54.770141 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.770151 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000199 1 0.000235 2014-04-25 09:18:54.770177 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.770188 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.770197 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.770209 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 2014-04-25 09:18:54.770221 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.770231 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.770241 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.770253 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:54.770264 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) 2014-04-25 09:18:54.770276 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.33 is down 2014-04-25 09:18:54.770286 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [33,195]/[33,195] maybe_went_rw) 2014-04-25 09:18:54.770298 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 195,210 down 33 blocked_by {} 2014-04-25 09:18:54.770309 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.770321 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.195 2014-04-25 09:18:54.770335 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.770407 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.770464 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_advance_map [344,183,210]/[344,183,210] 2014-04-25 09:18:54.770484 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.770496 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.770507 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:54.770518 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:54.770529 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.770543 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.770583 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] handle_advance_map [210,195]/[210,195] 2014-04-25 09:18:54.770602 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:54.770613 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:54.770625 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 173.272252 136 0.002332 2014-04-25 09:18:54.770639 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 173.272281 0 0.000000 2014-04-25 09:18:54.770651 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started 177.333309 0 0.000000 2014-04-25 09:18:54.770663 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] enter Reset 2014-04-25 09:18:54.770674 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: Reset advmap 2014-04-25 09:18:54.770685 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:54.770697 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.770707 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:54.770729 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [33,210,195]/[33,210,195]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3754 generate_past_intervals interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) : primary up 3720-3796 includes interval 2014-04-25 09:18:54.770743 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) 2014-04-25 09:18:54.770755 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] up [33,210,195] -> [210,195], acting [33,210,195] -> [210,195], role 1 -> 0 2014-04-25 09:18:54.770768 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:54.770779 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:54.770795 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] on_role_change 2014-04-25 09:18:54.770806 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:54.770815 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:54.770833 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:54.770845 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 195,210 2014-04-25 09:18:54.770855 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:54.770865 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000202 1 0.000230 2014-04-25 09:18:54.770876 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:54.770886 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:54.770896 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:54.770907 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:54.770920 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:54.770930 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:54.770941 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:54.770951 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:54.770964 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) 2014-04-25 09:18:54.770976 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.33 is down 2014-04-25 09:18:54.770982 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.770986 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [33,195]/[33,195] maybe_went_rw) 2014-04-25 09:18:54.770998 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 195,210 down 33 blocked_by {} 2014-04-25 09:18:54.771009 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:54.771022 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.195 2014-04-25 09:18:54.771035 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.771106 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.771133 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_advance_map [71,352,210]/[71,352,210] 2014-04-25 09:18:54.771324 7f77f22e9700 10 osd.210 pg_epoch: 3798 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,172,132]/[210,172,132] 2014-04-25 09:18:54.771347 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:54.771359 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.771370 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:54.771381 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.771397 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:54.771407 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:54.771417 7f77f22e9700 7 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:54.771429 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:54.771440 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:54.771453 7f77f22e9700 10 log is not dirty 2014-04-25 09:18:54.771984 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] flushed 2014-04-25 09:18:54.772013 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] flushed 2014-04-25 09:18:54.772811 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] flushed 2014-04-25 09:18:54.772836 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] flushed 2014-04-25 09:18:54.771152 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:55.052913 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.052948 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:55.052962 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:55.052974 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.052989 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:55.053069 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_advance_map [71,352,210]/[71,352,210] 2014-04-25 09:18:55.053091 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:55.053104 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.053115 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:55.053127 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:55.053138 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.053152 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:55.053179 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_advance_map [182,210,62]/[182,210,62] 2014-04-25 09:18:55.053198 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:55.053210 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.053222 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:55.053232 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:55.053243 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.053256 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:55.053300 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,384]/[210,384] 2014-04-25 09:18:55.053322 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: Peering advmap 2014-04-25 09:18:55.053336 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.053347 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: Started advmap 2014-04-25 09:18:55.053358 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.053370 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_activate_map 2014-04-25 09:18:55.053381 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary 2014-04-25 09:18:55.053394 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] take_waiters 2014-04-25 09:18:55.053405 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.053419 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:55.053455 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_advance_map [344,183,210]/[344,183,210] 2014-04-25 09:18:55.053477 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:55.053489 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.053501 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:55.053512 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:55.053522 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.053535 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:55.053551 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_advance_map [182,210,62]/[182,210,62] 2014-04-25 09:18:55.053568 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:55.053579 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.053590 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:55.053601 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:55.053611 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.053625 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:55.053639 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,384]/[210,384] 2014-04-25 09:18:55.053656 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: Peering advmap 2014-04-25 09:18:55.053668 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.053678 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: Started advmap 2014-04-25 09:18:55.053688 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.053698 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_activate_map 2014-04-25 09:18:55.053709 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary 2014-04-25 09:18:55.053720 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] take_waiters 2014-04-25 09:18:55.053731 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.053745 7f77f2cea700 10 log is not dirty 2014-04-25 09:18:54.773112 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:55.054072 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] recovery raced and were queued twice, ignoring! 2014-04-25 09:18:55.054095 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:55.054107 7f77edce2700 10 log is not dirty 2014-04-25 09:18:54.773146 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,434]/[210,434] 2014-04-25 09:18:55.054242 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: Peering advmap 2014-04-25 09:18:55.054265 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.054277 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: Started advmap 2014-04-25 09:18:55.054288 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.054299 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_activate_map 2014-04-25 09:18:55.054312 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary 2014-04-25 09:18:55.054324 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] take_waiters 2014-04-25 09:18:55.054336 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.054351 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:55.054374 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,7]/[210,7] 2014-04-25 09:18:55.054391 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:55.054403 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.14 info 3.1ef1( v 1139'748 (0'0,1139'748] local-les=3731 n=748 ec=633 les/c 3731/3742 3753/3753/3753) 2014-04-25 09:18:55.054422 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 7,14,210 unchanged 2014-04-25 09:18:55.054434 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.054444 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:55.054454 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:55.054467 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 173.435357 133 0.008101 2014-04-25 09:18:55.054483 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 173.913060 0 0.000000 2014-04-25 09:18:55.054498 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started/Primary 175.949633 0 0.000000 2014-04-25 09:18:55.054511 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started 175.949758 0 0.000000 2014-04-25 09:18:55.054522 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] enter Reset 2014-04-25 09:18:55.054534 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap 2014-04-25 09:18:55.054549 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3754 >= end epoch 3753, nothing to do 2014-04-25 09:18:55.054561 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.054570 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:55.054594 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [210,14,7]/[210,14,7]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) : primary up 3753-3797 includes interval 2014-04-25 09:18:55.054609 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) 2014-04-25 09:18:55.054621 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] up [210,14,7] -> [210,7], acting [210,14,7] -> [210,7], role 0 -> 0 2014-04-25 09:18:55.054635 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:55.054647 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:55.054663 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,14,7] -> [210,7], replicas changed 2014-04-25 09:18:55.054674 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:55.054684 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:55.054696 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:55.054707 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 7,14,210 -> 7,210 2014-04-25 09:18:55.054718 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:55.054729 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000207 1 0.000307 2014-04-25 09:18:55.054741 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:55.054751 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:55.054761 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:55.054772 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:55.054785 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:55.054796 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:55.054806 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:55.054818 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:55.054830 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) 2014-04-25 09:18:55.054844 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.14 is down 2014-04-25 09:18:55.054854 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 7,210 down 14 blocked_by {} 2014-04-25 09:18:55.054865 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:55.054878 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.7 2014-04-25 09:18:55.054894 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.054934 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:55.054981 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_advance_map [67,88,210]/[67,88,210] 2014-04-25 09:18:55.055001 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:55.055012 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.055023 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:55.055035 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:55.055046 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.055058 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:55.055075 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,57]/[210,57] 2014-04-25 09:18:55.055093 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:55.055108 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.201 info 3.4bfd( v 1110'721 lc 0'0 (0'0,1110'721] local-les=3748 n=721 ec=633 les/c 3748/3715 3756/3756/3753) 2014-04-25 09:18:55.055173 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 57,201,210 unchanged 2014-04-25 09:18:55.055187 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.055198 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:55.055207 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:55.055220 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 162.230589 118 0.006154 2014-04-25 09:18:55.055233 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 162.231052 0 0.000000 2014-04-25 09:18:55.055249 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] exit Started/Primary 170.954643 0 0.000000 2014-04-25 09:18:55.055261 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] exit Started 170.954700 0 0.000000 2014-04-25 09:18:55.055273 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] enter Reset 2014-04-25 09:18:55.055285 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] state: Reset advmap 2014-04-25 09:18:55.055295 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:55.055306 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.055316 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:55.055338 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3756-3798 [210,201,57]/[210,201,57]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3715 generate_past_intervals interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) : primary up 3753-3797 includes interval 2014-04-25 09:18:55.055349 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] noting past interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) 2014-04-25 09:18:55.055361 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] up [210,201,57] -> [210,57], acting [210,201,57] -> [210,57], role 0 -> 0 2014-04-25 09:18:55.055374 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:55.055394 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:55.055410 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] [210,201,57] -> [210,57], replicas changed 2014-04-25 09:18:55.055421 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:55.055430 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:55.055442 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:55.055454 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 57,201,210 -> 57,210 2014-04-25 09:18:55.055465 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:55.055475 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000202 1 0.000351 2014-04-25 09:18:55.055487 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:55.055498 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:55.055508 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:55.055519 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:55.055531 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:55.055542 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:55.055552 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:55.055562 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:55.055573 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) 2014-04-25 09:18:55.055584 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.201 is down 2014-04-25 09:18:55.055594 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3755 [210,57]/[210,57] maybe_went_rw) 2014-04-25 09:18:55.055606 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 57,210 down 201 blocked_by {} 2014-04-25 09:18:55.055617 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:55.055629 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.57 2014-04-25 09:18:55.055643 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.055684 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:55.055710 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,434]/[210,434] 2014-04-25 09:18:55.055729 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: Peering advmap 2014-04-25 09:18:55.055742 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.055753 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: Started advmap 2014-04-25 09:18:55.055764 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.055775 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_activate_map 2014-04-25 09:18:55.055786 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary 2014-04-25 09:18:55.055798 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] take_waiters 2014-04-25 09:18:55.055810 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.055823 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:55.055839 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,44,365]/[210,44,365] 2014-04-25 09:18:55.055857 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:55.055868 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.055878 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:55.055889 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.055899 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:55.055909 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:55.055921 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:55.055933 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:55.055944 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.055956 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:55.055976 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,335]/[210,335] 2014-04-25 09:18:55.055994 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:55.056005 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.289 info 3.18e4( v 1196'732 (0'0,1196'732] local-les=3754 n=732 ec=633 les/c 3754/3754 3755/3755/3753) 2014-04-25 09:18:55.056023 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 210,289,335 unchanged 2014-04-25 09:18:55.056034 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.056044 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:55.056054 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:55.056066 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 162.845289 115 0.005654 2014-04-25 09:18:55.056079 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 162.845809 0 0.000000 2014-04-25 09:18:55.056095 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started/Primary 172.293876 0 0.000000 2014-04-25 09:18:55.056106 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started 172.293946 0 0.000000 2014-04-25 09:18:55.056117 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] enter Reset 2014-04-25 09:18:55.056129 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: Reset advmap 2014-04-25 09:18:55.056139 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:55.056149 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.056166 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:55.056187 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3755-3798 [210,289,335]/[210,289,335]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) : primary up 3753-3797 includes interval 2014-04-25 09:18:55.056200 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] noting past interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) 2014-04-25 09:18:55.056211 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] up [210,289,335] -> [210,335], acting [210,289,335] -> [210,335], role 0 -> 0 2014-04-25 09:18:55.056264 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:55.056276 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:55.056292 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] [210,289,335] -> [210,335], replicas changed 2014-04-25 09:18:55.056305 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:55.056314 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:55.056326 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:55.056337 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 210,289,335 -> 210,335 2014-04-25 09:18:55.056348 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:55.056358 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000240 1 0.000333 2014-04-25 09:18:55.056369 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:55.056379 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:55.056389 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:55.056400 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 2014-04-25 09:18:55.056412 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:55.056422 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:55.056432 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:55.056442 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:55.056456 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) 2014-04-25 09:18:55.056468 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.289 is down 2014-04-25 09:18:55.056478 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3754 [210,289]/[210,289] maybe_went_rw) 2014-04-25 09:18:55.056490 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,335 down 289 blocked_by {} 2014-04-25 09:18:55.056500 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:55.056512 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.335 2014-04-25 09:18:55.056525 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.056559 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:55.056617 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,335]/[210,335] 2014-04-25 09:18:55.056638 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:55.056651 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.289 info 4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3754 n=1514 ec=638 les/c 3754/3754 3755/3755/3753) 2014-04-25 09:18:55.056668 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 210,289,335 unchanged 2014-04-25 09:18:55.056680 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.056691 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:55.056700 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:55.056712 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 162.865985 115 0.005636 2014-04-25 09:18:55.056725 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 162.866614 0 0.000000 2014-04-25 09:18:55.056740 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started/Primary 172.292423 0 0.000000 2014-04-25 09:18:55.056752 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started 172.292490 0 0.000000 2014-04-25 09:18:55.056764 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] enter Reset 2014-04-25 09:18:55.056775 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: Reset advmap 2014-04-25 09:18:55.056785 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:55.056796 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.056806 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:55.056826 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3755-3798 [210,289,335]/[210,289,335]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) : primary up 3753-3797 includes interval 2014-04-25 09:18:55.056838 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] noting past interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) 2014-04-25 09:18:55.056850 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] up [210,289,335] -> [210,335], acting [210,289,335] -> [210,335], role 0 -> 0 2014-04-25 09:18:55.056949 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:55.056962 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:55.056979 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] [210,289,335] -> [210,335], replicas changed 2014-04-25 09:18:55.056990 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:55.056999 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:55.057011 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:55.057022 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 210,289,335 -> 210,335 2014-04-25 09:18:55.057034 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:55.057045 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000280 1 0.000374 2014-04-25 09:18:55.057056 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:55.057066 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:55.057076 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:55.057086 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 2014-04-25 09:18:55.057098 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:55.057108 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:55.057119 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:55.057128 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 2014-04-25 09:18:55.057140 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) 2014-04-25 09:18:55.057151 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.289 is down 2014-04-25 09:18:55.057174 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3754 [210,289]/[210,289] maybe_went_rw) 2014-04-25 09:18:55.057186 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,335 down 289 blocked_by {} 2014-04-25 09:18:55.057197 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:55.057210 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.335 2014-04-25 09:18:55.057223 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.057259 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:55.057290 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,57]/[210,57] 2014-04-25 09:18:55.057309 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:55.057321 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.201 info 4.4bfc( v 1110'1416 lc 0'0 (0'0,1110'1416] local-les=3748 n=1416 ec=638 les/c 3748/3715 3756/3756/3753) 2014-04-25 09:18:55.057424 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 57,201,210 unchanged 2014-04-25 09:18:55.057436 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.057448 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:55.057457 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset 2014-04-25 09:18:55.057470 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 162.247034 117 0.161419 2014-04-25 09:18:55.057483 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 162.247613 0 0.000000 2014-04-25 09:18:55.057499 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] exit Started/Primary 170.954834 0 0.000000 2014-04-25 09:18:55.057510 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] exit Started 170.954891 0 0.000000 2014-04-25 09:18:55.057522 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] enter Reset 2014-04-25 09:18:55.057533 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] state: Reset advmap 2014-04-25 09:18:55.057544 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:55.057556 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.057566 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again 2014-04-25 09:18:55.057588 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3756-3798 [210,201,57]/[210,201,57]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3715 generate_past_intervals interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) : primary up 3753-3797 includes interval 2014-04-25 09:18:55.057600 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] noting past interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) 2014-04-25 09:18:55.057610 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] up [210,201,57] -> [210,57], acting [210,201,57] -> [210,57], role 0 -> 0 2014-04-25 09:18:55.057622 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] clear_primary_state 2014-04-25 09:18:55.057635 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] on_change 2014-04-25 09:18:55.057651 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] [210,201,57] -> [210,57], replicas changed 2014-04-25 09:18:55.057661 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] cancel_recovery 2014-04-25 09:18:55.057670 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state 2014-04-25 09:18:55.057682 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] handle_activate_map 2014-04-25 09:18:55.057694 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 57,201,210 -> 57,210 2014-04-25 09:18:55.057706 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] take_waiters 2014-04-25 09:18:55.057718 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000197 1 0.000374 2014-04-25 09:18:55.057732 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started 2014-04-25 09:18:55.057743 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Start 2014-04-25 09:18:55.057754 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary 2014-04-25 09:18:55.057766 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000023 0 0.000000 2014-04-25 09:18:55.057779 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary 2014-04-25 09:18:55.057789 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering 2014-04-25 09:18:55.057801 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo 2014-04-25 09:18:55.057811 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 2014-04-25 09:18:55.057823 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) 2014-04-25 09:18:55.057837 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.201 is down 2014-04-25 09:18:55.057848 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3755 [210,57]/[210,57] maybe_went_rw) 2014-04-25 09:18:55.057861 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 57,210 down 201 blocked_by {} 2014-04-25 09:18:55.057872 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor 2014-04-25 09:18:55.057885 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.57 2014-04-25 09:18:55.057898 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.057942 7f77ef0e4700 10 log is not dirty 2014-04-25 09:18:54.773389 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_advance_map [426,210,432]/[426,210,432] 2014-04-25 09:18:54.776072 7f77a69a2700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.057983 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] state: Started advmap 2014-04-25 09:18:55.058003 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058017 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_activate_map 2014-04-25 09:18:55.058028 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] take_waiters 2014-04-25 09:18:55.058040 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.058053 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:54.778173 7f77c1e4f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.058073 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,452,445]/[210,452,445] 2014-04-25 09:18:55.058091 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap 2014-04-25 09:18:55.058102 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:54.778224 7f77c8dc4700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.058112 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap 2014-04-25 09:18:55.058122 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058133 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map 2014-04-25 09:18:55.058144 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap 2014-04-25 09:18:55.058155 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary 2014-04-25 09:18:55.058173 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters 2014-04-25 09:18:55.058184 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt 2014-04-25 09:18:55.058196 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:55.058213 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,221,153]/[210,221,153] 2014-04-25 09:18:55.058232 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:55.058244 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058255 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:54.784810 7f77a659e700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.058267 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058280 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map 2014-04-25 09:18:55.058291 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap 2014-04-25 09:18:55.058315 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:55.058325 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary 2014-04-25 09:18:55.058340 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters 2014-04-25 09:18:55.058351 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3754 epoch_requested: 3754 LocalRecoveryReserved 2014-04-25 09:18:55.058365 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 173.487659 137 3.004374 2014-04-25 09:18:55.058378 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/WaitRemoteRecoveryReserved 2014-04-25 09:18:54.784865 7f77d063c700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.058443 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitRemoteRecoveryReserved 0.000065 0 0.000000 2014-04-25 09:18:55.058459 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/Recovering 2014-04-25 09:18:54.784922 7f77d4278700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.058472 7f77ee6e3700 10 osd.210 3799 queue_for_recovery already queued pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] 2014-04-25 09:18:55.058484 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:55.058501 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,193,171]/[210,193,171] 2014-04-25 09:18:54.784967 7f77a75ae700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.058520 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:55.058532 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058543 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:55.058554 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058565 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map 2014-04-25 09:18:55.058575 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap 2014-04-25 09:18:55.058586 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:55.058595 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary 2014-04-25 09:18:55.058608 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters 2014-04-25 09:18:55.058619 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3754 epoch_requested: 3754 LocalRecoveryReserved 2014-04-25 09:18:55.058630 7f77f72f1700 10 osd.210 3800 handle_osd_ping canceling queued failure report for osd.420 2014-04-25 09:18:55.058632 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 173.322826 138 0.006657 2014-04-25 09:18:55.058645 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/WaitRemoteRecoveryReserved 2014-04-25 09:18:55.058659 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:55.058679 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.309 has 3799 2014-04-25 09:18:55.058699 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,193,171]/[210,193,171] 2014-04-25 09:18:55.058721 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:55.058734 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058746 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:55.058757 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058770 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.172 has 3800 2014-04-25 09:18:55.058769 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map 2014-04-25 09:18:55.058780 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap 2014-04-25 09:18:55.058792 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:55.058800 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary 2014-04-25 09:18:55.058816 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters 2014-04-25 09:18:55.058828 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3754 epoch_requested: 3754 LocalRecoveryReserved 2014-04-25 09:18:55.058851 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.268 has 3800 2014-04-25 09:18:55.058842 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 173.322969 138 0.006648 2014-04-25 09:18:55.058856 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/WaitRemoteRecoveryReserved 2014-04-25 09:18:55.058870 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.163 has 3800 2014-04-25 09:18:55.058871 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:55.058887 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,80,105]/[210,80,105] 2014-04-25 09:18:55.058904 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap 2014-04-25 09:18:55.058918 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058930 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.288 has 3799 2014-04-25 09:18:55.058929 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap 2014-04-25 09:18:55.058940 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down 2014-04-25 09:18:55.058952 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map 2014-04-25 09:18:55.058963 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap 2014-04-25 09:18:55.058975 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] 2014-04-25 09:18:55.058984 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary 2014-04-25 09:18:55.058997 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters 2014-04-25 09:18:55.059008 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3754 epoch_requested: 3754 LocalRecoveryReserved 2014-04-25 09:18:55.059021 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 172.805408 138 0.161930 2014-04-25 09:18:55.059027 7f77f72f1700 10 osd.210 3800 handle_osd_ping osd.172 10.193.207.185:6802/2012373 says i am down in 3800 2014-04-25 09:18:55.059035 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/WaitRemoteRecoveryReserved 2014-04-25 09:18:55.059079 7f77ee6e3700 10 log is not dirty 2014-04-25 09:18:55.059084 7f77f72f1700 10 osd.210 3800 handle_osd_ping osd.268 10.193.207.188:6865/4084837 says i am down in 3800 2014-04-25 09:18:55.059103 7f77f72f1700 10 osd.210 3800 handle_osd_ping osd.163 10.193.207.184:7073/6064447 says i am down in 3800 2014-04-25 09:18:55.059146 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 2014-04-25 09:18:55.059152 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request 2014-04-25 09:18:55.059174 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 2014-04-25 09:18:55.059177 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request 2014-04-25 09:18:55.059193 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 2014-04-25 09:18:55.059195 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request 2014-04-25 09:18:54.790094 7f77a427b700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.791879 7f77d8304700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.791986 7f77c5c8d700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.793941 7f77cb6ed700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.059950 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=74 :47454 s=2 pgs=7812 cs=1 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:54.800254 7f77ca2d9700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.800314 7f77e91cd700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.811771 7f77d3c72700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.812322 7f77c3566700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.061252 7f77f68f0700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 2014-04-25 09:18:55.061257 7f77f68f0700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request 2014-04-25 09:18:55.061285 7f77f68f0700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 2014-04-25 09:18:55.061287 7f77f68f0700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request 2014-04-25 09:18:55.061500 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) 2014-04-25 09:18:55.061507 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] 2014-04-25 09:18:55.061522 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] recover_replicas(5) 2014-04-25 09:18:55.061536 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] peer osd.221 missing 0 objects. 2014-04-25 09:18:55.061547 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] peer osd.153 missing 65 objects. 2014-04-25 09:18:55.061559 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] recover_replicas: recover_object_replicas(7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3) 2014-04-25 09:18:55.061574 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] prep_object_replica_pushes: on 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 2014-04-25 09:18:55.061953 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] populate_obc_watchers 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 2014-04-25 09:18:55.061986 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] get_object_context 0x1c7d2f00 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 0 -> 1 read 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3(979'622 client.25998.0:1201815 wrlock_by=unknown.0.0:0) 2014-04-25 09:18:55.062003 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_lock for 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 2014-04-25 09:18:55.062017 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] start_recovery_op 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 2014-04-25 09:18:55.062029 7f77edce2700 10 osd.210 3800 start_recovery_op pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 (5/15 rops) 2014-04-25 09:18:55.062052 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] prep_push_to_replica7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 v979'622 size 2355200 to osd.153 2014-04-25 09:18:55.062247 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] get_snapset_context eb6ccdaaa67adb1b0ae4417d079fa03c 0 -> 1 2014-04-25 09:18:55.062299 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 clone_overlap {} 2014-04-25 09:18:55.062314 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 data_subset [0~2355200] clone_subsets {} 2014-04-25 09:18:55.062327 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] put_snapset_context eb6ccdaaa67adb1b0ae4417d079fa03c 1 -> 0 2014-04-25 09:18:55.062349 7f77edce2700 7 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] send_push_op 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 v 979'622 size 2355200 recovery_info: ObjectRecoveryInfo(7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3@979'622, copy_subset: [0~2355200], clone_subset: {}) 2014-04-25 09:18:54.846522 7f77ded6e700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.065865 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.070401 7f77c1546700 -1 *** Caught signal (Aborted) ** in thread 7f77c1546700 ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73) 1: /home/y/bin64/ceph-osd() [0x80db19] 2: /lib64/libpthread.so.0() [0x3a6e40f500] 3: (gsignal()+0x35) [0x3a6e0328a5] 4: (abort()+0x175) [0x3a6e034085] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x3a700bea5d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2014-04-25 09:18:55.078118 7f77a0d46700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.078241 7f77d356b700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.079527 7f77e95d1700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.080033 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48083 s=2 pgs=7816 cs=3 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:55.081437 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.082401 7f77a873d700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:54.851735 7f77c5081700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.084539 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 2014-04-25 09:18:55.084543 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request 2014-04-25 09:18:55.084574 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 2014-04-25 09:18:55.084576 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request 2014-04-25 09:18:55.084844 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 2014-04-25 09:18:55.084846 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request 2014-04-25 09:18:55.087211 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48146 s=2 pgs=7817 cs=5 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:55.090072 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.094441 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48189 s=2 pgs=7818 cs=7 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:55.095041 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd --- begin dump of recent events --- -1710> 2014-04-25 09:18:54.750078 7f77d0f45700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1709> 2014-04-25 09:18:54.750180 7f77db738700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1708> 2014-04-25 09:18:54.752180 7f77e7ab6700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1707> 2014-04-25 09:18:54.752284 7f77ca5dc700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1706> 2014-04-25 09:18:54.752451 7f77d9112700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1705> 2014-04-25 09:18:54.752600 7f779a5df700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1704> 2014-04-25 09:18:54.752704 7f77cb3ea700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1703> 2014-04-25 09:18:54.735150 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000204 1 0.000304 -1702> 2014-04-25 09:18:54.752831 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started -1701> 2014-04-25 09:18:54.752845 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start -1700> 2014-04-25 09:18:54.752855 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1699> 2014-04-25 09:18:54.752869 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000025 0 0.000000 -1698> 2014-04-25 09:18:54.752881 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1697> 2014-04-25 09:18:54.752892 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1696> 2014-04-25 09:18:54.752903 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1695> 2014-04-25 09:18:54.752915 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 -1694> 2014-04-25 09:18:54.752928 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) -1693> 2014-04-25 09:18:54.752941 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.14 is down -1692> 2014-04-25 09:18:54.752951 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 7,210 down 14 blocked_by {} -1691> 2014-04-25 09:18:54.752963 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1690> 2014-04-25 09:18:54.752976 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.7 -1689> 2014-04-25 09:18:54.752991 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.1ef0( v 1185'1376 (0'0,1185'1376] local-les=3754 n=1376 ec=638 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1688> 2014-04-25 09:18:54.753031 7f77ef0e4700 10 log is not dirty -1687> 2014-04-25 09:18:54.753492 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_advance_map [341,210]/[341,210] -1686> 2014-04-25 09:18:54.753520 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Started advmap -1685> 2014-04-25 09:18:54.753532 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1684> 2014-04-25 09:18:54.753545 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 171.868237 132 0.002767 -1683> 2014-04-25 09:18:54.753560 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 171.868270 0 0.000000 -1682> 2014-04-25 09:18:54.753573 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started 175.595095 0 0.000000 -1681> 2014-04-25 09:18:54.753585 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] enter Reset -1680> 2014-04-25 09:18:54.753597 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Reset advmap -1679> 2014-04-25 09:18:54.753607 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3120 -1678> 2014-04-25 09:18:54.753620 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] dropping down osd.204 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -1677> 2014-04-25 09:18:54.753634 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged -1676> 2014-04-25 09:18:54.753645 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1675> 2014-04-25 09:18:54.753656 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1674> 2014-04-25 09:18:54.753678 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [341,210,204]/[341,210,204]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3120 generate_past_intervals interval(3753-3798 [341,210,204]/[341,210,204] maybe_went_rw) : primary up 3720-3796 includes interval -1673> 2014-04-25 09:18:54.753691 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3753/3753/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [341,210,204]/[341,210,204] maybe_went_rw) -1672> 2014-04-25 09:18:54.753704 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] up [341,210,204] -> [341,210], acting [341,210,204] -> [341,210], role 1 -> 1 -1671> 2014-04-25 09:18:54.753716 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] on_change -1670> 2014-04-25 09:18:54.753733 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] cancel_recovery -1669> 2014-04-25 09:18:54.753744 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] clear_recovery_state -1668> 2014-04-25 09:18:54.753757 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] handle_activate_map -1667> 2014-04-25 09:18:54.753769 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] update_heartbeat_peers unchanged -1666> 2014-04-25 09:18:54.753779 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] take_waiters -1665> 2014-04-25 09:18:54.753790 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] exit Reset 0.000205 1 0.000239 -1664> 2014-04-25 09:18:54.753802 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Started -1663> 2014-04-25 09:18:54.753813 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Start -1662> 2014-04-25 09:18:54.753823 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] state: transitioning to Stray -1661> 2014-04-25 09:18:54.753834 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] exit Start 0.000021 0 0.000000 -1660> 2014-04-25 09:18:54.753847 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Started/Stray -1659> 2014-04-25 09:18:54.753858 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.409( v 1110'1432 (0'0,1110'1432] local-les=3754 n=1432 ec=638 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1658> 2014-04-25 09:18:54.753919 7f77ee6e3700 10 log is not dirty -1657> 2014-04-25 09:18:54.753990 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_advance_map [341,210]/[341,210] -1656> 2014-04-25 09:18:54.754011 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Started advmap -1655> 2014-04-25 09:18:54.754022 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1654> 2014-04-25 09:18:54.754034 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 171.857275 132 0.002394 -1653> 2014-04-25 09:18:54.754048 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 171.857306 0 0.000000 -1652> 2014-04-25 09:18:54.754061 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] exit Started 175.593812 0 0.000000 -1651> 2014-04-25 09:18:54.754074 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] enter Reset -1650> 2014-04-25 09:18:54.754086 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Reset advmap -1649> 2014-04-25 09:18:54.754096 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3120 -1648> 2014-04-25 09:18:54.754107 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] dropping down osd.204 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -1647> 2014-04-25 09:18:54.754122 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged -1646> 2014-04-25 09:18:54.754133 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1645> 2014-04-25 09:18:54.754143 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210,204] r=1 lpr=3799 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1644> 2014-04-25 09:18:54.754172 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [341,210,204]/[341,210,204]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3120 generate_past_intervals interval(3753-3798 [341,210,204]/[341,210,204] maybe_went_rw) : primary up 3720-3796 includes interval -1643> 2014-04-25 09:18:54.754183 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3753/3753/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [341,210,204]/[341,210,204] maybe_went_rw) -1642> 2014-04-25 09:18:54.754197 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 luod=0'0 lcod 0'0 active] up [341,210,204] -> [341,210], acting [341,210,204] -> [341,210], role 1 -> 1 -1641> 2014-04-25 09:18:54.754209 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] on_change -1640> 2014-04-25 09:18:54.754224 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] cancel_recovery -1639> 2014-04-25 09:18:54.754234 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] clear_recovery_state -1638> 2014-04-25 09:18:54.754246 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] handle_activate_map -1637> 2014-04-25 09:18:54.754258 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] update_heartbeat_peers unchanged -1636> 2014-04-25 09:18:54.754267 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] take_waiters -1635> 2014-04-25 09:18:54.754278 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] exit Reset 0.000204 1 0.000236 -1634> 2014-04-25 09:18:54.754292 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Started -1633> 2014-04-25 09:18:54.754303 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Start -1632> 2014-04-25 09:18:54.754313 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] state: transitioning to Stray -1631> 2014-04-25 09:18:54.754324 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] exit Start 0.000022 0 0.000000 -1630> 2014-04-25 09:18:54.754337 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] enter Started/Stray -1629> 2014-04-25 09:18:54.754348 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.40a( v 1220'710 (0'0,1220'710] local-les=3754 n=710 ec=633 les/c 3754/3120 3799/3799/3720) [341,210] r=1 lpr=3799 pi=3119-3798/22 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1628> 2014-04-25 09:18:54.754407 7f77ee6e3700 10 log is not dirty -1627> 2014-04-25 09:18:54.754435 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] handle_advance_map [210]/[210] -1626> 2014-04-25 09:18:54.754454 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: Started advmap -1625> 2014-04-25 09:18:54.754466 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1624> 2014-04-25 09:18:54.754479 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 137.660129 67 0.001276 -1623> 2014-04-25 09:18:54.754491 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 137.660160 0 0.000000 -1622> 2014-04-25 09:18:54.754503 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started 169.026816 0 0.000000 -1621> 2014-04-25 09:18:54.754515 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] enter Reset -1620> 2014-04-25 09:18:54.754527 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: Reset advmap -1619> 2014-04-25 09:18:54.754537 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3777 -1618> 2014-04-25 09:18:54.754549 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] dropping down osd.33 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -1617> 2014-04-25 09:18:54.754563 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged -1616> 2014-04-25 09:18:54.754573 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1615> 2014-04-25 09:18:54.754583 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1614> 2014-04-25 09:18:54.754605 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3757-3798 [359,33,210]/[359,33,210]): not rw, up_thru 3796 up_from 3757 last_epoch_clean 3777 generate_past_intervals interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) : primary up 3757-3796 includes interval -1613> 2014-04-25 09:18:54.754617 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3757/3757/3757) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) -1612> 2014-04-25 09:18:54.754629 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] up [359,33,210] -> [210], acting [359,33,210] -> [210], role 2 -> 0 -1611> 2014-04-25 09:18:54.754641 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -1610> 2014-04-25 09:18:54.754652 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] on_change -1609> 2014-04-25 09:18:54.754668 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] on_role_change -1608> 2014-04-25 09:18:54.754679 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1607> 2014-04-25 09:18:54.754689 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1606> 2014-04-25 09:18:54.754702 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1605> 2014-04-25 09:18:54.754713 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 -1604> 2014-04-25 09:18:54.754723 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] take_waiters -1603> 2014-04-25 09:18:54.754734 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000219 1 0.000249 -1602> 2014-04-25 09:18:54.754746 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started -1601> 2014-04-25 09:18:54.754757 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Start -1600> 2014-04-25 09:18:54.754767 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1599> 2014-04-25 09:18:54.754778 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -1598> 2014-04-25 09:18:54.754790 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1597> 2014-04-25 09:18:54.754801 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1596> 2014-04-25 09:18:54.754812 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1595> 2014-04-25 09:18:54.754822 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3777 -1594> 2014-04-25 09:18:54.754833 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) -1593> 2014-04-25 09:18:54.754845 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.359 is down -1592> 2014-04-25 09:18:54.754855 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.33 is down -1591> 2014-04-25 09:18:54.754865 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3756 [33,210]/[33,210] maybe_went_rw) -1590> 2014-04-25 09:18:54.754877 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 33,359 blocked_by {} -1589> 2014-04-25 09:18:54.754889 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1588> 2014-04-25 09:18:54.754903 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000090 0 0.000000 -1587> 2014-04-25 09:18:54.754918 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -1586> 2014-04-25 09:18:54.754933 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) -1585> 2014-04-25 09:18:54.754946 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) -1584> 2014-04-25 09:18:54.754960 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary -1583> 2014-04-25 09:18:54.754969 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) -1582> 2014-04-25 09:18:54.754984 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) -1581> 2014-04-25 09:18:54.754996 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog -1580> 2014-04-25 09:18:54.755010 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000091 0 0.000000 -1579> 2014-04-25 09:18:54.755023 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -1578> 2014-04-25 09:18:54.755033 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active -1577> 2014-04-25 09:18:54.755048 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000025 0 0.000000 -1576> 2014-04-25 09:18:54.755061 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -1575> 2014-04-25 09:18:54.755073 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1574> 2014-04-25 09:18:54.755140 7f77ee6e3700 10 log is not dirty -1573> 2014-04-25 09:18:54.755174 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_advance_map [444,210,301]/[444,210,301] -1572> 2014-04-25 09:18:54.755193 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] state: Started advmap -1571> 2014-04-25 09:18:54.755206 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1570> 2014-04-25 09:18:54.755218 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_activate_map -1569> 2014-04-25 09:18:54.755230 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] take_waiters -1568> 2014-04-25 09:18:54.755241 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.581c( v 1110'726 (0'0,1110'726] local-les=3767 n=726 ec=633 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1567> 2014-04-25 09:18:54.755255 7f77ee6e3700 10 log is not dirty -1566> 2014-04-25 09:18:54.755297 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_advance_map [444,210,301]/[444,210,301] -1565> 2014-04-25 09:18:54.755316 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] state: Started advmap -1564> 2014-04-25 09:18:54.755328 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1563> 2014-04-25 09:18:54.755341 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_activate_map -1562> 2014-04-25 09:18:54.755351 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] take_waiters -1561> 2014-04-25 09:18:54.755363 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.581b( v 1110'1432 (0'0,1110'1432] local-les=3767 n=1432 ec=638 les/c 3767/3768 3755/3755/3755) [444,210,301] r=1 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1560> 2014-04-25 09:18:54.755376 7f77ee6e3700 10 log is not dirty -1559> 2014-04-25 09:18:54.755391 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] handle_advance_map [210,171]/[210,171] -1558> 2014-04-25 09:18:54.755408 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: Started advmap -1557> 2014-04-25 09:18:54.755419 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1556> 2014-04-25 09:18:54.755431 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 128.236795 58 0.001323 -1555> 2014-04-25 09:18:54.755444 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 128.236827 0 0.000000 -1554> 2014-04-25 09:18:54.755456 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started 158.124719 0 0.000000 -1553> 2014-04-25 09:18:54.755468 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] enter Reset -1552> 2014-04-25 09:18:54.755481 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: Reset advmap -1551> 2014-04-25 09:18:54.755492 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3780 -1550> 2014-04-25 09:18:54.755503 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1549> 2014-04-25 09:18:54.755513 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1548> 2014-04-25 09:18:54.755534 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3764-3798 [435,210,171]/[435,210,171]): not rw, up_thru 3796 up_from 3764 last_epoch_clean 3780 generate_past_intervals interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) : primary up 3764-3796 includes interval -1547> 2014-04-25 09:18:54.755545 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3764/3764/3764) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) -1546> 2014-04-25 09:18:54.755557 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] up [435,210,171] -> [210,171], acting [435,210,171] -> [210,171], role 1 -> 0 -1545> 2014-04-25 09:18:54.755569 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -1544> 2014-04-25 09:18:54.755582 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change -1543> 2014-04-25 09:18:54.755598 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] on_role_change -1542> 2014-04-25 09:18:54.755608 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1541> 2014-04-25 09:18:54.755618 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1540> 2014-04-25 09:18:54.755630 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1539> 2014-04-25 09:18:54.755642 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 171,210 -1538> 2014-04-25 09:18:54.755652 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters -1537> 2014-04-25 09:18:54.755663 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000195 1 0.000222 -1536> 2014-04-25 09:18:54.755675 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started -1535> 2014-04-25 09:18:54.755685 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start -1534> 2014-04-25 09:18:54.755696 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1533> 2014-04-25 09:18:54.755708 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000023 0 0.000000 -1532> 2014-04-25 09:18:54.755720 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1531> 2014-04-25 09:18:54.755731 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1530> 2014-04-25 09:18:54.755742 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1529> 2014-04-25 09:18:54.755752 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3780 -1528> 2014-04-25 09:18:54.755764 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) -1527> 2014-04-25 09:18:54.755775 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down -1526> 2014-04-25 09:18:54.755785 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3761-3763 [210,171]/[210,171] maybe_went_rw) -1525> 2014-04-25 09:18:54.755797 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 171,210 down 435 blocked_by {} -1524> 2014-04-25 09:18:54.755809 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1523> 2014-04-25 09:18:54.755822 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.171 -1522> 2014-04-25 09:18:54.755836 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6d26( v 1110'707 (0'0,1110'707] local-les=3780 n=707 ec=633 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1521> 2014-04-25 09:18:54.755870 7f77ee6e3700 10 log is not dirty -1520> 2014-04-25 09:18:54.755894 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] handle_advance_map [210]/[210] -1519> 2014-04-25 09:18:54.755913 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: Started advmap -1518> 2014-04-25 09:18:54.755925 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1517> 2014-04-25 09:18:54.755937 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 137.656590 67 0.001228 -1516> 2014-04-25 09:18:54.755950 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 137.656619 0 0.000000 -1515> 2014-04-25 09:18:54.755962 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] exit Started 169.025859 0 0.000000 -1514> 2014-04-25 09:18:54.755975 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3757 pi=1918-3756/31 luod=0'0 lcod 0'0 active] enter Reset -1513> 2014-04-25 09:18:54.755986 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: Reset advmap -1512> 2014-04-25 09:18:54.755997 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3777 -1511> 2014-04-25 09:18:54.756009 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] dropping down osd.33 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -1510> 2014-04-25 09:18:54.756023 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged -1509> 2014-04-25 09:18:54.756032 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1508> 2014-04-25 09:18:54.756043 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [359,33,210] r=2 lpr=3799 pi=1918-3756/31 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1507> 2014-04-25 09:18:54.756067 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3757-3798 [359,33,210]/[359,33,210]): not rw, up_thru 3796 up_from 3757 last_epoch_clean 3777 generate_past_intervals interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) : primary up 3757-3796 includes interval -1506> 2014-04-25 09:18:54.756079 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3757/3757/3757) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) -1505> 2014-04-25 09:18:54.756091 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 luod=0'0 lcod 0'0 mlcod 0'0 active] up [359,33,210] -> [210], acting [359,33,210] -> [210], role 2 -> 0 -1504> 2014-04-25 09:18:54.756103 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -1503> 2014-04-25 09:18:54.756115 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] on_change -1502> 2014-04-25 09:18:54.756131 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] on_role_change -1501> 2014-04-25 09:18:54.756143 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1500> 2014-04-25 09:18:54.756152 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1499> 2014-04-25 09:18:54.756171 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1498> 2014-04-25 09:18:54.756183 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 -1497> 2014-04-25 09:18:54.756194 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] take_waiters -1496> 2014-04-25 09:18:54.756204 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000230 1 0.000259 -1495> 2014-04-25 09:18:54.756217 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started -1494> 2014-04-25 09:18:54.756228 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Start -1493> 2014-04-25 09:18:54.756238 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1492> 2014-04-25 09:18:54.756248 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 -1491> 2014-04-25 09:18:54.756260 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1490> 2014-04-25 09:18:54.756271 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1489> 2014-04-25 09:18:54.756282 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1488> 2014-04-25 09:18:54.756292 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3777 -1487> 2014-04-25 09:18:54.756303 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3757-3798 [359,33,210]/[359,33,210] maybe_went_rw) -1486> 2014-04-25 09:18:54.756315 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.359 is down -1485> 2014-04-25 09:18:54.756325 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.33 is down -1484> 2014-04-25 09:18:54.756336 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3756 [33,210]/[33,210] maybe_went_rw) -1483> 2014-04-25 09:18:54.756348 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 33,359 blocked_by {} -1482> 2014-04-25 09:18:54.756359 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1481> 2014-04-25 09:18:54.756372 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000090 0 0.000000 -1480> 2014-04-25 09:18:54.756386 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -1479> 2014-04-25 09:18:54.756398 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) -1478> 2014-04-25 09:18:54.756412 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) -1477> 2014-04-25 09:18:54.756426 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary -1476> 2014-04-25 09:18:54.756435 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) -1475> 2014-04-25 09:18:54.756449 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) -1474> 2014-04-25 09:18:54.756461 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog -1473> 2014-04-25 09:18:54.756474 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000089 0 0.000000 -1472> 2014-04-25 09:18:54.756487 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -1471> 2014-04-25 09:18:54.756498 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active -1470> 2014-04-25 09:18:54.756511 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000025 0 0.000000 -1469> 2014-04-25 09:18:54.756524 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -1468> 2014-04-25 09:18:54.756535 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1467> 2014-04-25 09:18:54.756602 7f77ee6e3700 10 log is not dirty -1466> 2014-04-25 09:18:54.756634 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] handle_advance_map [210]/[210] -1465> 2014-04-25 09:18:54.756653 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: Started advmap -1464> 2014-04-25 09:18:54.756664 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: up or acting affected, transitioning to Reset -1463> 2014-04-25 09:18:54.756676 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] exit Started/Stray 12.671522 3 0.000234 -1462> 2014-04-25 09:18:54.756688 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] exit Started 12.671578 0 0.000000 -1461> 2014-04-25 09:18:54.756701 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3798 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] enter Reset -1460> 2014-04-25 09:18:54.756712 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: Reset advmap -1459> 2014-04-25 09:18:54.756721 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] _calc_past_interval_range: already have past intervals back to 3715 -1458> 2014-04-25 09:18:54.756733 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] dropping down osd.435 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -1457> 2014-04-25 09:18:54.756747 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] update_heartbeat_peers unchanged -1456> 2014-04-25 09:18:54.756757 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] check_recovery_sources no source osds () went down -1455> 2014-04-25 09:18:54.756767 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [435,210] r=1 lpr=3799 pi=1915-3797/28 lcod 0'0 inactive NOTIFY] state: up or acting affected, calling start_peering_interval again -1454> 2014-04-25 09:18:54.756789 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] start_peering_interval: check_new_interval output: generate_past_intervals interval(3798-3798 [435,210]/[435,210]): not rw, up_thru 3796 up_from 3764 last_epoch_clean 3715 generate_past_intervals interval(3798-3798 [435,210]/[435,210]) : primary up 3764-3796 does not include interval -1453> 2014-04-25 09:18:54.756801 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3798/3798/3798) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] noting past interval(3798-3798 [435,210]/[435,210]) -1452> 2014-04-25 09:18:54.756813 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] up [435,210] -> [210], acting [435,210] -> [210], role 1 -> 0 -1451> 2014-04-25 09:18:54.756824 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] clear_primary_state -1450> 2014-04-25 09:18:54.756835 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] on_change -1449> 2014-04-25 09:18:54.756851 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive NOTIFY] on_role_change -1448> 2014-04-25 09:18:54.756861 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1447> 2014-04-25 09:18:54.756871 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1446> 2014-04-25 09:18:54.756882 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1445> 2014-04-25 09:18:54.756894 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 -1444> 2014-04-25 09:18:54.756904 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] take_waiters -1443> 2014-04-25 09:18:54.756916 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000215 1 0.000231 -1442> 2014-04-25 09:18:54.756928 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started -1441> 2014-04-25 09:18:54.756938 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Start -1440> 2014-04-25 09:18:54.756951 7f77d8a0b700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1439> 2014-04-25 09:18:54.756948 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1438> 2014-04-25 09:18:54.756959 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -1437> 2014-04-25 09:18:54.756971 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1436> 2014-04-25 09:18:54.756981 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1435> 2014-04-25 09:18:54.756991 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1434> 2014-04-25 09:18:54.757001 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 -1433> 2014-04-25 09:18:54.757013 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3798-3798 [435,210]/[435,210]) -1432> 2014-04-25 09:18:54.757024 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3797 [306,435,210]/[306,435,210] maybe_went_rw) -1431> 2014-04-25 09:18:54.757036 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.306 is down -1430> 2014-04-25 09:18:54.757046 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down -1429> 2014-04-25 09:18:54.757056 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3761-3763 [306,210]/[306,210]) -1428> 2014-04-25 09:18:54.757067 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 306,435 blocked_by {} -1427> 2014-04-25 09:18:54.757078 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1426> 2014-04-25 09:18:54.757091 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000100 0 0.000000 -1425> 2014-04-25 09:18:54.757103 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -1424> 2014-04-25 09:18:54.757116 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) -1423> 2014-04-25 09:18:54.757130 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) -1422> 2014-04-25 09:18:54.757143 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary -1421> 2014-04-25 09:18:54.757153 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) -1420> 2014-04-25 09:18:54.757186 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) -1419> 2014-04-25 09:18:54.757199 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog -1418> 2014-04-25 09:18:54.757212 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000108 0 0.000000 -1417> 2014-04-25 09:18:54.757224 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -1416> 2014-04-25 09:18:54.757235 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active -1415> 2014-04-25 09:18:54.757247 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000023 0 0.000000 -1414> 2014-04-25 09:18:54.757260 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -1413> 2014-04-25 09:18:54.757270 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1412> 2014-04-25 09:18:54.757331 7f77ee6e3700 10 log is not dirty -1411> 2014-04-25 09:18:54.757358 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,80,105]/[210,80,105] -1410> 2014-04-25 09:18:54.757379 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -1409> 2014-04-25 09:18:54.757393 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -1408> 2014-04-25 09:18:54.757405 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -1407> 2014-04-25 09:18:54.757416 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -1406> 2014-04-25 09:18:54.757428 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -1405> 2014-04-25 09:18:54.757439 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -1404> 2014-04-25 09:18:54.757467 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] -1403> 2014-04-25 09:18:54.757476 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -1402> 2014-04-25 09:18:54.757491 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -1401> 2014-04-25 09:18:54.757502 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1400> 2014-04-25 09:18:54.757516 7f77ee6e3700 10 log is not dirty -1399> 2014-04-25 09:18:54.757560 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] handle_advance_map [210,171]/[210,171] -1398> 2014-04-25 09:18:54.757581 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: Started advmap -1397> 2014-04-25 09:18:54.757592 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1396> 2014-04-25 09:18:54.757605 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 128.096077 58 0.001418 -1395> 2014-04-25 09:18:54.757618 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 128.096111 0 0.000000 -1394> 2014-04-25 09:18:54.757630 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] exit Started 158.123644 0 0.000000 -1393> 2014-04-25 09:18:54.757642 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3764 pi=3761-3763/1 luod=0'0 lcod 0'0 active] enter Reset -1392> 2014-04-25 09:18:54.757654 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: Reset advmap -1391> 2014-04-25 09:18:54.757664 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3780 -1390> 2014-04-25 09:18:54.757677 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1389> 2014-04-25 09:18:54.757687 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [435,210,171] r=1 lpr=3799 pi=3761-3763/1 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1388> 2014-04-25 09:18:54.757707 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3764-3798 [435,210,171]/[435,210,171]): not rw, up_thru 3796 up_from 3764 last_epoch_clean 3780 generate_past_intervals interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) : primary up 3764-3796 includes interval -1387> 2014-04-25 09:18:54.757720 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3764/3764/3764) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) -1386> 2014-04-25 09:18:54.757732 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 luod=0'0 lcod 0'0 mlcod 0'0 active] up [435,210,171] -> [210,171], acting [435,210,171] -> [210,171], role 1 -> 0 -1385> 2014-04-25 09:18:54.757745 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -1384> 2014-04-25 09:18:54.757756 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change -1383> 2014-04-25 09:18:54.757773 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] on_role_change -1382> 2014-04-25 09:18:54.757783 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1381> 2014-04-25 09:18:54.757793 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1380> 2014-04-25 09:18:54.757805 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1379> 2014-04-25 09:18:54.757817 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 171,210 -1378> 2014-04-25 09:18:54.757828 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters -1377> 2014-04-25 09:18:54.757839 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000197 1 0.000225 -1376> 2014-04-25 09:18:54.757851 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started -1375> 2014-04-25 09:18:54.757862 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start -1374> 2014-04-25 09:18:54.757872 7f77ee6e3700 1 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1373> 2014-04-25 09:18:54.757882 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 -1372> 2014-04-25 09:18:54.757895 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1371> 2014-04-25 09:18:54.757905 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1370> 2014-04-25 09:18:54.757917 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1369> 2014-04-25 09:18:54.757928 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3780 -1368> 2014-04-25 09:18:54.757940 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3798 [435,210,171]/[435,210,171] maybe_went_rw) -1367> 2014-04-25 09:18:54.757952 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down -1366> 2014-04-25 09:18:54.757962 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3761-3763 [210,171]/[210,171] maybe_went_rw) -1365> 2014-04-25 09:18:54.757974 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 171,210 down 435 blocked_by {} -1364> 2014-04-25 09:18:54.757986 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1363> 2014-04-25 09:18:54.757998 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.171 -1362> 2014-04-25 09:18:54.758012 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.6d25( v 1219'1402 (0'0,1219'1402] local-les=3780 n=1401 ec=638 les/c 3780/3780 3799/3799/3799) [210,171] r=0 lpr=3799 pi=3761-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1361> 2014-04-25 09:18:54.758047 7f77ee6e3700 10 log is not dirty -1360> 2014-04-25 09:18:54.758073 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,452,445]/[210,452,445] -1359> 2014-04-25 09:18:54.758092 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -1358> 2014-04-25 09:18:54.758104 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -1357> 2014-04-25 09:18:54.758114 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -1356> 2014-04-25 09:18:54.758125 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -1355> 2014-04-25 09:18:54.758135 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -1354> 2014-04-25 09:18:54.758246 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_advance_map [365,447,210]/[365,447,210] -1353> 2014-04-25 09:18:54.758271 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] state: Started advmap -1352> 2014-04-25 09:18:54.758283 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1351> 2014-04-25 09:18:54.758297 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_activate_map -1350> 2014-04-25 09:18:54.758308 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] take_waiters -1349> 2014-04-25 09:18:54.758319 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2b89( v 1110'1462 (0'0,1110'1462] local-les=3775 n=1462 ec=638 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1348> 2014-04-25 09:18:54.758332 7f77f18e8700 10 log is not dirty -1347> 2014-04-25 09:18:54.758373 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_advance_map [88,428,210]/[88,428,210] -1346> 2014-04-25 09:18:54.758394 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] state: Started advmap -1345> 2014-04-25 09:18:54.758406 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1344> 2014-04-25 09:18:54.758418 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_activate_map -1343> 2014-04-25 09:18:54.758429 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] take_waiters -1342> 2014-04-25 09:18:54.758440 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2588( v 1110'748 (0'0,1110'748] local-les=3775 n=748 ec=633 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1341> 2014-04-25 09:18:54.758454 7f77f18e8700 10 log is not dirty -1340> 2014-04-25 09:18:54.758489 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_advance_map [53,101,210]/[53,101,210] -1339> 2014-04-25 09:18:54.758508 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] state: Started advmap -1338> 2014-04-25 09:18:54.758520 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1337> 2014-04-25 09:18:54.758532 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_activate_map -1336> 2014-04-25 09:18:54.758543 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] take_waiters -1335> 2014-04-25 09:18:54.758555 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1f8c( v 1110'1359 (0'0,1110'1359] local-les=3777 n=1359 ec=638 les/c 3777/3779 3755/3755/3755) [53,101,210] r=2 lpr=3755 pi=3753-3754/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1334> 2014-04-25 09:18:54.758568 7f77f18e8700 10 log is not dirty -1333> 2014-04-25 09:18:54.758583 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_advance_map [365,447,210]/[365,447,210] -1332> 2014-04-25 09:18:54.758599 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] state: Started advmap -1331> 2014-04-25 09:18:54.758611 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1330> 2014-04-25 09:18:54.758623 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_activate_map -1329> 2014-04-25 09:18:54.758635 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] take_waiters -1328> 2014-04-25 09:18:54.758645 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.2b8a( v 1201'709 (0'0,1201'709] local-les=3775 n=709 ec=633 les/c 3775/3777 3765/3765/3765) [365,447,210] r=2 lpr=3765 pi=1701-3764/42 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1327> 2014-04-25 09:18:54.758659 7f77f18e8700 10 log is not dirty -1326> 2014-04-25 09:18:54.758673 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_advance_map [88,428,210]/[88,428,210] -1325> 2014-04-25 09:18:54.758690 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] state: Started advmap -1324> 2014-04-25 09:18:54.758701 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1323> 2014-04-25 09:18:54.758714 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_activate_map -1322> 2014-04-25 09:18:54.758724 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] take_waiters -1321> 2014-04-25 09:18:54.758737 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.2587( v 1110'1402 (0'0,1110'1402] local-les=3775 n=1402 ec=638 les/c 3775/3777 3766/3766/3720) [88,428,210] r=2 lpr=3766 pi=3119-3765/18 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1320> 2014-04-25 09:18:54.758750 7f77f18e8700 10 log is not dirty -1319> 2014-04-25 09:18:54.758764 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,102,268]/[210,102,268] -1318> 2014-04-25 09:18:54.758781 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -1317> 2014-04-25 09:18:54.758793 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -1316> 2014-04-25 09:18:54.758803 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -1315> 2014-04-25 09:18:54.758813 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -1314> 2014-04-25 09:18:54.758823 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -1313> 2014-04-25 09:18:54.758834 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -1312> 2014-04-25 09:18:54.758844 7f77f18e8700 7 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -1311> 2014-04-25 09:18:54.758856 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] take_waiters -1310> 2014-04-25 09:18:54.758867 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.6494( v 1110'719 (0'0,1110'719] local-les=3779 n=719 ec=633 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1309> 2014-04-25 09:18:54.758879 7f77f18e8700 10 log is not dirty -1308> 2014-04-25 09:18:54.758918 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,102,268]/[210,102,268] -1307> 2014-04-25 09:18:54.758938 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -1306> 2014-04-25 09:18:54.758950 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -1305> 2014-04-25 09:18:54.758961 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -1304> 2014-04-25 09:18:54.758971 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -1303> 2014-04-25 09:18:54.758982 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -1302> 2014-04-25 09:18:54.758992 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -1301> 2014-04-25 09:18:54.759003 7f77f18e8700 7 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -1300> 2014-04-25 09:18:54.759015 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] take_waiters -1299> 2014-04-25 09:18:54.759026 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.6493( v 1110'1359 (0'0,1110'1359] local-les=3779 n=1359 ec=638 les/c 3779/3781 3771/3771/3753) [210,102,268] r=0 lpr=3771 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1298> 2014-04-25 09:18:54.759039 7f77f18e8700 10 log is not dirty -1297> 2014-04-25 09:18:54.759054 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_advance_map [43,210]/[43,210] -1296> 2014-04-25 09:18:54.759071 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Started advmap -1295> 2014-04-25 09:18:54.759082 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1294> 2014-04-25 09:18:54.759094 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_activate_map -1293> 2014-04-25 09:18:54.759105 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] take_waiters -1292> 2014-04-25 09:18:54.759116 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.558a( v 1110'1394 (0'0,1110'1394] local-les=3755 n=1394 ec=638 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1291> 2014-04-25 09:18:54.759130 7f77f18e8700 10 log is not dirty -1290> 2014-04-25 09:18:54.759171 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,200]/[210,200] -1289> 2014-04-25 09:18:54.759202 7f77af1a7700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1288> 2014-04-25 09:18:54.759195 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -1287> 2014-04-25 09:18:54.759221 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.400 info 3.256a( v 1110'642 lc 0'0 (0'0,1110'642] local-les=3747 n=642 ec=633 les/c 3754/3754 3755/3755/3753) -1286> 2014-04-25 09:18:54.759294 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 200,210,400 unchanged -1285> 2014-04-25 09:18:54.759307 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -1284> 2014-04-25 09:18:54.759319 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -1283> 2014-04-25 09:18:54.759329 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset -1282> 2014-04-25 09:18:54.759342 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 161.951519 115 0.005363 -1281> 2014-04-25 09:18:54.759356 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 161.951998 0 0.000000 -1280> 2014-04-25 09:18:54.759371 7f77df576700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1279> 2014-04-25 09:18:54.759375 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started/Primary 172.069340 0 0.000000 -1278> 2014-04-25 09:18:54.759387 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started 172.069416 0 0.000000 -1277> 2014-04-25 09:18:54.759399 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] enter Reset -1276> 2014-04-25 09:18:54.759412 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: Reset advmap -1275> 2014-04-25 09:18:54.759422 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 -1274> 2014-04-25 09:18:54.759434 7f77e9fdb700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1273> 2014-04-25 09:18:54.759435 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -1272> 2014-04-25 09:18:54.759445 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200,400] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1271> 2014-04-25 09:18:54.759468 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3755-3798 [210,200,400]/[210,200,400]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3755-3798 [210,200,400]/[210,200,400] maybe_went_rw) : primary up 3753-3797 includes interval -1270> 2014-04-25 09:18:54.759480 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3755/3755/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] noting past interval(3755-3798 [210,200,400]/[210,200,400] maybe_went_rw) -1269> 2014-04-25 09:18:54.759491 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] up [210,200,400] -> [210,200], acting [210,200,400] -> [210,200], role 0 -> 0 -1268> 2014-04-25 09:18:54.759504 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -1267> 2014-04-25 09:18:54.759516 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change -1266> 2014-04-25 09:18:54.759532 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] [210,200,400] -> [210,200], replicas changed -1265> 2014-04-25 09:18:54.759543 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1264> 2014-04-25 09:18:54.759553 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1263> 2014-04-25 09:18:54.759566 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1262> 2014-04-25 09:18:54.759577 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 200,210,400 -> 200,210 -1261> 2014-04-25 09:18:54.759589 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters -1260> 2014-04-25 09:18:54.759600 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000201 1 0.000371 -1259> 2014-04-25 09:18:54.759612 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started -1258> 2014-04-25 09:18:54.759622 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start -1257> 2014-04-25 09:18:54.759632 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1256> 2014-04-25 09:18:54.759643 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 -1255> 2014-04-25 09:18:54.759654 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1254> 2014-04-25 09:18:54.759665 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1253> 2014-04-25 09:18:54.759676 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1252> 2014-04-25 09:18:54.759686 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 -1251> 2014-04-25 09:18:54.759698 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3755-3798 [210,200,400]/[210,200,400] maybe_went_rw) -1250> 2014-04-25 09:18:54.759710 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.400 is down -1249> 2014-04-25 09:18:54.759720 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3754 [210,200]/[210,200] maybe_went_rw) -1248> 2014-04-25 09:18:54.759731 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 200,210 down 400 blocked_by {} -1247> 2014-04-25 09:18:54.759742 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1246> 2014-04-25 09:18:54.759754 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.200 -1245> 2014-04-25 09:18:54.759768 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.256a( v 1110'642 (0'0,1110'642] local-les=3761 n=642 ec=633 les/c 3761/3754 3799/3799/3753) [210,200] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1244> 2014-04-25 09:18:54.759805 7f77f2cea700 10 log is not dirty -1243> 2014-04-25 09:18:54.759833 7f77f22e9700 10 osd.210 pg_epoch: 3798 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] handle_advance_map [196,210,282]/[196,210,282] -1242> 2014-04-25 09:18:54.759857 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] state: Started advmap -1241> 2014-04-25 09:18:54.759869 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1240> 2014-04-25 09:18:54.759883 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] handle_activate_map -1239> 2014-04-25 09:18:54.759894 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] take_waiters -1238> 2014-04-25 09:18:54.759905 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.522e( v 1110'1368 (0'0,1110'1368] local-les=3777 n=1367 ec=638 les/c 3777/3777 3774/3774/3719) [196,210,282] r=1 lpr=3774 pi=3715-3773/7 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1237> 2014-04-25 09:18:54.759919 7f77f22e9700 10 log is not dirty -1236> 2014-04-25 09:18:54.759938 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) -1235> 2014-04-25 09:18:54.759942 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] -1234> 2014-04-25 09:18:54.759955 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] recovery raced and were queued twice, ignoring! -1233> 2014-04-25 09:18:54.759969 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[3.5818( v 1200'677 (0'0,1200'677] local-les=3754 n=677 ec=633 les/c 3754/3148 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] -1232> 2014-04-25 09:18:54.759978 7f77edce2700 10 log is not dirty -1231> 2014-04-25 09:18:54.761102 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[4.6422( v 1110'1383 (0'0,1110'1383] local-les=3774 n=1382 ec=638 les/c 3774/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] flushed -1230> 2014-04-25 09:18:54.761131 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[3.6423( v 1110'738 (0'0,1110'738] local-les=3775 n=738 ec=633 les/c 3775/3777 3799/3799/3799) [210] r=0 lpr=3799 pi=1918-3798/32 lcod 0'0 mlcod 0'0 peering] flushed -1229> 2014-04-25 09:18:54.761153 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[3.5e1d( v 1110'702 (0'0,1110'702] local-les=3769 n=702 ec=633 les/c 3769/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=1915-3798/29 lcod 0'0 mlcod 0'0 peering] flushed -1228> 2014-04-25 09:18:54.761288 7f77efae5700 10 osd.210 pg_epoch: 3798 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] handle_advance_map [7,210]/[7,210] -1227> 2014-04-25 09:18:54.761313 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] state: Started advmap -1226> 2014-04-25 09:18:54.761325 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1225> 2014-04-25 09:18:54.761338 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 171.342455 133 0.002256 -1224> 2014-04-25 09:18:54.761353 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 171.342482 0 0.000000 -1223> 2014-04-25 09:18:54.761367 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] exit Started 175.621415 0 0.000000 -1222> 2014-04-25 09:18:54.761380 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3753 pi=2291-3752/33 luod=0'0 lcod 0'0 active] enter Reset -1221> 2014-04-25 09:18:54.761391 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] state: Reset advmap -1220> 2014-04-25 09:18:54.761402 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3755 -1219> 2014-04-25 09:18:54.761415 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] dropping down osd.179 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -1218> 2014-04-25 09:18:54.761430 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged -1217> 2014-04-25 09:18:54.761441 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1216> 2014-04-25 09:18:54.761452 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,179,210] r=2 lpr=3799 pi=2291-3752/33 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1215> 2014-04-25 09:18:54.761473 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 luod=0'0 lcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [7,179,210]/[7,179,210]): not rw, up_thru 3796 up_from 3717 last_epoch_clean 3755 generate_past_intervals interval(3753-3798 [7,179,210]/[7,179,210] maybe_went_rw) : primary up 3717-3796 includes interval -1214> 2014-04-25 09:18:54.761487 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3753/3753/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 luod=0'0 lcod 0'0 active] noting past interval(3753-3798 [7,179,210]/[7,179,210] maybe_went_rw) -1213> 2014-04-25 09:18:54.761498 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 luod=0'0 lcod 0'0 active] up [7,179,210] -> [7,210], acting [7,179,210] -> [7,210], role 2 -> 1 -1212> 2014-04-25 09:18:54.761511 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive] on_change -1211> 2014-04-25 09:18:54.761526 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive] on_role_change -1210> 2014-04-25 09:18:54.761536 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] cancel_recovery -1209> 2014-04-25 09:18:54.761546 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] clear_recovery_state -1208> 2014-04-25 09:18:54.761559 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] handle_activate_map -1207> 2014-04-25 09:18:54.761645 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] update_heartbeat_peers unchanged -1206> 2014-04-25 09:18:54.761656 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] take_waiters -1205> 2014-04-25 09:18:54.761667 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] exit Reset 0.000287 1 0.000248 -1204> 2014-04-25 09:18:54.761679 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] enter Started -1203> 2014-04-25 09:18:54.761689 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] enter Start -1202> 2014-04-25 09:18:54.761699 7f77efae5700 1 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] state: transitioning to Stray -1201> 2014-04-25 09:18:54.761710 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] exit Start 0.000021 0 0.000000 -1200> 2014-04-25 09:18:54.761723 7f77efae5700 5 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] enter Started/Stray -1199> 2014-04-25 09:18:54.761735 7f77efae5700 10 osd.210 pg_epoch: 3799 pg[3.cb5( v 1110'772 (0'0,1110'772] local-les=3755 n=772 ec=633 les/c 3755/3755 3799/3799/3717) [7,210] r=1 lpr=3799 pi=2291-3798/34 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1198> 2014-04-25 09:18:54.761792 7f77efae5700 10 log is not dirty -1197> 2014-04-25 09:18:54.761835 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,309,198]/[210,309,198] -1196> 2014-04-25 09:18:54.761856 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -1195> 2014-04-25 09:18:54.761870 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -1194> 2014-04-25 09:18:54.761883 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -1193> 2014-04-25 09:18:54.761893 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -1192> 2014-04-25 09:18:54.761907 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -1191> 2014-04-25 09:18:54.761918 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -1190> 2014-04-25 09:18:54.761962 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) -1189> 2014-04-25 09:18:54.761968 7f77ef0e4700 10 osd.210 3799 queue_for_recovery queued pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] -1188> 2014-04-25 09:18:54.761979 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -1187> 2014-04-25 09:18:54.761993 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -1186> 2014-04-25 09:18:54.762005 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1185> 2014-04-25 09:18:54.762019 7f77ef0e4700 10 log is not dirty -1184> 2014-04-25 09:18:54.758147 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -1183> 2014-04-25 09:18:54.762025 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -1182> 2014-04-25 09:18:54.762040 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -1181> 2014-04-25 09:18:54.762051 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.4c17( v 1110'747 (0'0,1110'747] local-les=3754 n=747 ec=633 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1180> 2014-04-25 09:18:54.762064 7f77ee6e3700 10 log is not dirty -1179> 2014-04-25 09:18:54.762835 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] handle_advance_map [450,252,210]/[450,252,210] -1178> 2014-04-25 09:18:54.762859 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] state: Started advmap -1177> 2014-04-25 09:18:54.762871 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1176> 2014-04-25 09:18:54.762883 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] handle_activate_map -1175> 2014-04-25 09:18:54.762895 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] take_waiters -1174> 2014-04-25 09:18:54.762906 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.5b8d( v 1110'683 (0'0,1110'683] local-les=3754 n=683 ec=633 les/c 3754/3754 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=633-3752/50 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1173> 2014-04-25 09:18:54.762921 7f77f18e8700 10 log is not dirty -1172> 2014-04-25 09:18:54.762960 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_advance_map [92,210]/[92,210] -1171> 2014-04-25 09:18:54.762980 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] state: Started advmap -1170> 2014-04-25 09:18:54.762991 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] check_recovery_sources no source osds () went down -1169> 2014-04-25 09:18:54.763002 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_activate_map -1168> 2014-04-25 09:18:54.763013 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] take_waiters -1167> 2014-04-25 09:18:54.763023 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.1076( v 1110'703 (0'0,1110'703] local-les=3770 n=703 ec=633 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1166> 2014-04-25 09:18:54.763037 7f77f18e8700 10 log is not dirty -1165> 2014-04-25 09:18:54.763070 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_advance_map [43,210]/[43,210] -1164> 2014-04-25 09:18:54.763089 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] state: Started advmap -1163> 2014-04-25 09:18:54.763101 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1162> 2014-04-25 09:18:54.763112 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_activate_map -1161> 2014-04-25 09:18:54.763124 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] take_waiters -1160> 2014-04-25 09:18:54.763135 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.558b( v 1156'681 (0'0,1156'681] local-les=3755 n=681 ec=633 les/c 3755/3757 3753/3753/3752) [43,210] r=1 lpr=3753 pi=3119-3752/21 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1159> 2014-04-25 09:18:54.763148 7f77f18e8700 10 log is not dirty -1158> 2014-04-25 09:18:54.763169 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_advance_map [92,210]/[92,210] -1157> 2014-04-25 09:18:54.763187 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] state: Started advmap -1156> 2014-04-25 09:18:54.763199 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] check_recovery_sources no source osds () went down -1155> 2014-04-25 09:18:54.763210 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_activate_map -1154> 2014-04-25 09:18:54.763220 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] take_waiters -1153> 2014-04-25 09:18:54.763231 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.1075( v 1110'1450 (0'0,1110'1450] local-les=3770 n=1450 ec=638 les/c 3770/3715 3798/3798/3726) [92,210] r=1 lpr=3798 pi=2271-3797/34 lcod 0'0 inactive] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1152> 2014-04-25 09:18:54.763244 7f77f18e8700 10 log is not dirty -1151> 2014-04-25 09:18:54.763258 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] handle_advance_map [450,252,210]/[450,252,210] -1150> 2014-04-25 09:18:54.763275 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] state: Started advmap -1149> 2014-04-25 09:18:54.763287 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1148> 2014-04-25 09:18:54.763298 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] handle_activate_map -1147> 2014-04-25 09:18:54.763309 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] take_waiters -1146> 2014-04-25 09:18:54.763320 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.5b8c( v 1138'1502 (0'0,1138'1502] local-les=3754 n=1502 ec=638 les/c 3754/3755 3753/3753/3753) [450,252,210] r=2 lpr=3753 pi=638-3752/50 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1145> 2014-04-25 09:18:54.763333 7f77f18e8700 10 log is not dirty -1144> 2014-04-25 09:18:54.763347 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] handle_advance_map [210]/[210] -1143> 2014-04-25 09:18:54.763363 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: Started advmap -1142> 2014-04-25 09:18:54.763374 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1141> 2014-04-25 09:18:54.763387 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 172.545697 135 0.002914 -1140> 2014-04-25 09:18:54.763401 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 172.545731 0 0.000000 -1139> 2014-04-25 09:18:54.763414 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started 177.343139 0 0.000000 -1138> 2014-04-25 09:18:54.763426 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] enter Reset -1137> 2014-04-25 09:18:54.763438 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: Reset advmap -1136> 2014-04-25 09:18:54.763448 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 -1135> 2014-04-25 09:18:54.763459 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] dropping down osd.14 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -1134> 2014-04-25 09:18:54.763473 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged -1133> 2014-04-25 09:18:54.763484 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1132> 2014-04-25 09:18:54.763495 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1131> 2014-04-25 09:18:54.763518 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [184,14,210]/[184,14,210]): not rw, up_thru 3796 up_from 3717 last_epoch_clean 3715 generate_past_intervals interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) : primary up 3717-3796 includes interval -1130> 2014-04-25 09:18:54.763532 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3753/3753/3717) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) -1129> 2014-04-25 09:18:54.763544 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] up [184,14,210] -> [210], acting [184,14,210] -> [210], role 2 -> 0 -1128> 2014-04-25 09:18:54.763556 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -1127> 2014-04-25 09:18:54.763567 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] on_change -1126> 2014-04-25 09:18:54.763583 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] on_role_change -1125> 2014-04-25 09:18:54.763594 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1124> 2014-04-25 09:18:54.763604 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1123> 2014-04-25 09:18:54.763617 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1122> 2014-04-25 09:18:54.763628 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 -1121> 2014-04-25 09:18:54.763639 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] take_waiters -1120> 2014-04-25 09:18:54.763650 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000223 1 0.000254 -1119> 2014-04-25 09:18:54.763661 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started -1118> 2014-04-25 09:18:54.763672 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Start -1117> 2014-04-25 09:18:54.763682 7f77f18e8700 1 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1116> 2014-04-25 09:18:54.763694 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000022 0 0.000000 -1115> 2014-04-25 09:18:54.763706 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1114> 2014-04-25 09:18:54.763717 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1113> 2014-04-25 09:18:54.763728 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1112> 2014-04-25 09:18:54.763738 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 -1111> 2014-04-25 09:18:54.763750 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) -1110> 2014-04-25 09:18:54.763762 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.184 is down -1109> 2014-04-25 09:18:54.763772 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.14 is down -1108> 2014-04-25 09:18:54.763782 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [184,14]/[184,14]) -1107> 2014-04-25 09:18:54.763794 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 14,184 blocked_by {} -1106> 2014-04-25 09:18:54.763805 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1105> 2014-04-25 09:18:54.763818 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000091 0 0.000000 -1104> 2014-04-25 09:18:54.763833 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -1103> 2014-04-25 09:18:54.763846 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) -1102> 2014-04-25 09:18:54.763860 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) -1101> 2014-04-25 09:18:54.763875 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary -1100> 2014-04-25 09:18:54.763884 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) -1099> 2014-04-25 09:18:54.763898 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) -1098> 2014-04-25 09:18:54.763910 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog -1097> 2014-04-25 09:18:54.763924 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000091 0 0.000000 -1096> 2014-04-25 09:18:54.763938 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -1095> 2014-04-25 09:18:54.763948 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active -1094> 2014-04-25 09:18:54.763961 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000023 0 0.000000 -1093> 2014-04-25 09:18:54.763973 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -1092> 2014-04-25 09:18:54.763984 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1091> 2014-04-25 09:18:54.764033 7f77f18e8700 10 log is not dirty -1090> 2014-04-25 09:18:54.764072 7f77f18e8700 10 osd.210 pg_epoch: 3798 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] handle_advance_map [210]/[210] -1089> 2014-04-25 09:18:54.764091 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: Started advmap -1088> 2014-04-25 09:18:54.764102 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1087> 2014-04-25 09:18:54.764114 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 173.178391 135 0.002388 -1086> 2014-04-25 09:18:54.764127 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 173.178420 0 0.000000 -1085> 2014-04-25 09:18:54.764139 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] exit Started 177.340467 0 0.000000 -1084> 2014-04-25 09:18:54.764151 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3753 pi=3099-3752/15 luod=0'0 lcod 0'0 active] enter Reset -1083> 2014-04-25 09:18:54.764170 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: Reset advmap -1082> 2014-04-25 09:18:54.764181 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 -1081> 2014-04-25 09:18:54.764192 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] dropping down osd.14 info 0.0( DNE empty local-les=0 n=0 ec=0 les/c 0/0 0/0/0) -1080> 2014-04-25 09:18:54.764207 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] update_heartbeat_peers unchanged -1079> 2014-04-25 09:18:54.764217 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1078> 2014-04-25 09:18:54.764227 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [184,14,210] r=2 lpr=3799 pi=3099-3752/15 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1077> 2014-04-25 09:18:54.764250 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [184,14,210]/[184,14,210]): not rw, up_thru 3796 up_from 3717 last_epoch_clean 3715 generate_past_intervals interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) : primary up 3717-3796 includes interval -1076> 2014-04-25 09:18:54.764263 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3753/3753/3717) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) -1075> 2014-04-25 09:18:54.764275 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 luod=0'0 lcod 0'0 mlcod 0'0 active] up [184,14,210] -> [210], acting [184,14,210] -> [210], role 2 -> 0 -1074> 2014-04-25 09:18:54.764287 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -1073> 2014-04-25 09:18:54.764298 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] on_change -1072> 2014-04-25 09:18:54.764314 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] on_role_change -1071> 2014-04-25 09:18:54.764324 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1070> 2014-04-25 09:18:54.764334 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1069> 2014-04-25 09:18:54.764353 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1068> 2014-04-25 09:18:54.764364 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210 -1067> 2014-04-25 09:18:54.764374 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] take_waiters -1066> 2014-04-25 09:18:54.764384 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000233 1 0.000262 -1065> 2014-04-25 09:18:54.764396 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started -1064> 2014-04-25 09:18:54.764406 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Start -1063> 2014-04-25 09:18:54.764416 7f77f18e8700 1 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1062> 2014-04-25 09:18:54.764427 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 -1061> 2014-04-25 09:18:54.764438 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1060> 2014-04-25 09:18:54.764448 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1059> 2014-04-25 09:18:54.764458 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1058> 2014-04-25 09:18:54.764469 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 -1057> 2014-04-25 09:18:54.764480 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [184,14,210]/[184,14,210] maybe_went_rw) -1056> 2014-04-25 09:18:54.764492 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.184 is down -1055> 2014-04-25 09:18:54.764502 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.14 is down -1054> 2014-04-25 09:18:54.764512 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [184,14]/[184,14]) -1053> 2014-04-25 09:18:54.764524 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 14,184 blocked_by {} -1052> 2014-04-25 09:18:54.764535 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1051> 2014-04-25 09:18:54.764547 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000089 0 0.000000 -1050> 2014-04-25 09:18:54.764560 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -1049> 2014-04-25 09:18:54.764573 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) -1048> 2014-04-25 09:18:54.764587 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) -1047> 2014-04-25 09:18:54.764599 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary -1046> 2014-04-25 09:18:54.764609 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) -1045> 2014-04-25 09:18:54.764623 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) -1044> 2014-04-25 09:18:54.764634 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog -1043> 2014-04-25 09:18:54.764647 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000087 0 0.000000 -1042> 2014-04-25 09:18:54.764660 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -1041> 2014-04-25 09:18:54.764670 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active -1040> 2014-04-25 09:18:54.764684 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000024 0 0.000000 -1039> 2014-04-25 09:18:54.764696 7f77f18e8700 5 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -1038> 2014-04-25 09:18:54.764707 7f77f18e8700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -1037> 2014-04-25 09:18:54.764757 7f77f18e8700 10 log is not dirty -1036> 2014-04-25 09:18:54.766893 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] handle_advance_map [210,434]/[210,434] -1035> 2014-04-25 09:18:54.766919 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Started advmap -1034> 2014-04-25 09:18:54.766930 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -1033> 2014-04-25 09:18:54.766944 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 170.066336 129 0.002302 -1032> 2014-04-25 09:18:54.766957 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 170.066366 0 0.000000 -1031> 2014-04-25 09:18:54.766970 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started 177.339327 0 0.000000 -1030> 2014-04-25 09:18:54.766981 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] enter Reset -1029> 2014-04-25 09:18:54.766993 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Reset advmap -1028> 2014-04-25 09:18:54.767008 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3718 -1027> 2014-04-25 09:18:54.767020 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -1026> 2014-04-25 09:18:54.767030 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -1025> 2014-04-25 09:18:54.767052 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [129,210,434]/[129,210,434]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3718 generate_past_intervals interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) : primary up 3720-3796 includes interval -1024> 2014-04-25 09:18:54.767064 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3753/3753/3720) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) -1023> 2014-04-25 09:18:54.767076 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] up [129,210,434] -> [210,434], acting [129,210,434] -> [210,434], role 1 -> 0 -1022> 2014-04-25 09:18:54.767089 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -1021> 2014-04-25 09:18:54.767101 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] on_change -1020> 2014-04-25 09:18:54.767119 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] on_role_change -1019> 2014-04-25 09:18:54.767129 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -1018> 2014-04-25 09:18:54.767138 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -1017> 2014-04-25 09:18:54.767150 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -1016> 2014-04-25 09:18:54.767169 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210,434 -1015> 2014-04-25 09:18:54.767180 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] take_waiters -1014> 2014-04-25 09:18:54.767191 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000209 1 0.000233 -1013> 2014-04-25 09:18:54.767203 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started -1012> 2014-04-25 09:18:54.767214 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Start -1011> 2014-04-25 09:18:54.767224 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -1010> 2014-04-25 09:18:54.767235 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -1009> 2014-04-25 09:18:54.767249 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -1008> 2014-04-25 09:18:54.767259 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -1007> 2014-04-25 09:18:54.767270 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -1006> 2014-04-25 09:18:54.767282 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3718 -1005> 2014-04-25 09:18:54.767294 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) -1004> 2014-04-25 09:18:54.767306 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.129 is down -1003> 2014-04-25 09:18:54.767316 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [129,434]/[129,434]) -1002> 2014-04-25 09:18:54.767327 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,434 down 129 blocked_by {} -1001> 2014-04-25 09:18:54.767339 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -1000> 2014-04-25 09:18:54.767351 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.434 -999> 2014-04-25 09:18:54.767365 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.617f( v 1110'719 (0'0,1110'719] local-les=3755 n=719 ec=633 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -998> 2014-04-25 09:18:54.767406 7f77f2cea700 10 log is not dirty -997> 2014-04-25 09:18:54.767459 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] handle_advance_map [210,434]/[210,434] -996> 2014-04-25 09:18:54.767479 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Started advmap -995> 2014-04-25 09:18:54.767490 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -994> 2014-04-25 09:18:54.767502 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 170.066516 129 0.002459 -993> 2014-04-25 09:18:54.767515 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 170.066545 0 0.000000 -992> 2014-04-25 09:18:54.767527 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] exit Started 177.338797 0 0.000000 -991> 2014-04-25 09:18:54.767540 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3753 pi=3716-3752/5 luod=0'0 lcod 0'0 active] enter Reset -990> 2014-04-25 09:18:54.767551 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: Reset advmap -989> 2014-04-25 09:18:54.767561 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3718 -988> 2014-04-25 09:18:54.767573 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -987> 2014-04-25 09:18:54.767583 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [129,210,434] r=1 lpr=3799 pi=3716-3752/5 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -986> 2014-04-25 09:18:54.767605 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [129,210,434]/[129,210,434]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3718 generate_past_intervals interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) : primary up 3720-3796 includes interval -985> 2014-04-25 09:18:54.767617 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3753/3753/3720) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) -984> 2014-04-25 09:18:54.767629 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 luod=0'0 lcod 0'0 mlcod 0'0 active] up [129,210,434] -> [210,434], acting [129,210,434] -> [210,434], role 1 -> 0 -983> 2014-04-25 09:18:54.767641 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -982> 2014-04-25 09:18:54.767652 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] on_change -981> 2014-04-25 09:18:54.767668 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] on_role_change -980> 2014-04-25 09:18:54.767681 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -979> 2014-04-25 09:18:54.767690 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -978> 2014-04-25 09:18:54.767702 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -977> 2014-04-25 09:18:54.767720 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 210,434 -976> 2014-04-25 09:18:54.767731 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] take_waiters -975> 2014-04-25 09:18:54.767742 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000202 1 0.000224 -974> 2014-04-25 09:18:54.767754 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started -973> 2014-04-25 09:18:54.767763 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Start -972> 2014-04-25 09:18:54.767773 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -971> 2014-04-25 09:18:54.767785 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000022 0 0.000000 -970> 2014-04-25 09:18:54.767798 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -969> 2014-04-25 09:18:54.767808 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -968> 2014-04-25 09:18:54.767818 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -967> 2014-04-25 09:18:54.767828 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3718 -966> 2014-04-25 09:18:54.767840 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [129,210,434]/[129,210,434] maybe_went_rw) -965> 2014-04-25 09:18:54.767852 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.129 is down -964> 2014-04-25 09:18:54.767861 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [129,434]/[129,434]) -963> 2014-04-25 09:18:54.767874 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,434 down 129 blocked_by {} -962> 2014-04-25 09:18:54.767885 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -961> 2014-04-25 09:18:54.767896 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.434 -960> 2014-04-25 09:18:54.767910 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.617e( v 1110'1480 (0'0,1110'1480] local-les=3755 n=1480 ec=638 les/c 3755/3718 3799/3799/3799) [210,434] r=0 lpr=3799 pi=3716-3798/6 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -959> 2014-04-25 09:18:54.767951 7f77f2cea700 10 log is not dirty -958> 2014-04-25 09:18:54.767981 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,62,7]/[210,62,7] -957> 2014-04-25 09:18:54.768000 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -956> 2014-04-25 09:18:54.768012 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -955> 2014-04-25 09:18:54.768023 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -954> 2014-04-25 09:18:54.768033 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -953> 2014-04-25 09:18:54.768043 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -952> 2014-04-25 09:18:54.768054 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -951> 2014-04-25 09:18:54.768064 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -950> 2014-04-25 09:18:54.768075 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -949> 2014-04-25 09:18:54.768086 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2e6b( v 1110'661 (0'0,1110'661] local-les=3754 n=661 ec=633 les/c 3754/3754 3753/3753/3753) [210,62,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -948> 2014-04-25 09:18:54.768099 7f77f2cea700 10 log is not dirty -947> 2014-04-25 09:18:54.768116 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210]/[210] -946> 2014-04-25 09:18:54.768134 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -945> 2014-04-25 09:18:54.768146 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.243 info 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3749 n=1426 ec=638 les/c 3718/3718 3764/3764/3753) -944> 2014-04-25 09:18:54.768167 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.435 info 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3754 n=1426 ec=638 les/c 3754/3754 3764/3764/3753) -943> 2014-04-25 09:18:54.768184 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 210,243,435 unchanged -942> 2014-04-25 09:18:54.768195 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -941> 2014-04-25 09:18:54.768205 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -940> 2014-04-25 09:18:54.768214 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset -939> 2014-04-25 09:18:54.768226 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 125.984272 55 0.002052 -938> 2014-04-25 09:18:54.768238 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 156.737850 0 0.000000 -937> 2014-04-25 09:18:54.768252 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started/Primary 158.279280 0 0.000000 -936> 2014-04-25 09:18:54.768264 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started 158.279344 0 0.000000 -935> 2014-04-25 09:18:54.768276 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] enter Reset -934> 2014-04-25 09:18:54.768292 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap -933> 2014-04-25 09:18:54.768307 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3780 >= end epoch 3764, nothing to do -932> 2014-04-25 09:18:54.768318 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -931> 2014-04-25 09:18:54.768327 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -930> 2014-04-25 09:18:54.768347 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3764-3798 [210,243,435]/[210,243,435]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3780 generate_past_intervals interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) : primary up 3753-3797 includes interval -929> 2014-04-25 09:18:54.768359 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3764/3764/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) -928> 2014-04-25 09:18:54.768370 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] up [210,243,435] -> [210], acting [210,243,435] -> [210], role 0 -> 0 -927> 2014-04-25 09:18:54.768382 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -926> 2014-04-25 09:18:54.768393 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change -925> 2014-04-25 09:18:54.768409 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,243,435] -> [210], replicas changed -924> 2014-04-25 09:18:54.768420 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -923> 2014-04-25 09:18:54.768430 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -922> 2014-04-25 09:18:54.768442 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -921> 2014-04-25 09:18:54.768453 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 210,243,435 -> 210 -920> 2014-04-25 09:18:54.768464 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters -919> 2014-04-25 09:18:54.768474 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000199 1 0.000308 -918> 2014-04-25 09:18:54.768486 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started -917> 2014-04-25 09:18:54.768496 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start -916> 2014-04-25 09:18:54.768505 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -915> 2014-04-25 09:18:54.768516 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -914> 2014-04-25 09:18:54.768528 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -913> 2014-04-25 09:18:54.768538 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -912> 2014-04-25 09:18:54.768548 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -911> 2014-04-25 09:18:54.768558 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3780 -910> 2014-04-25 09:18:54.768570 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) -909> 2014-04-25 09:18:54.768583 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.243 is down -908> 2014-04-25 09:18:54.768592 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down -907> 2014-04-25 09:18:54.768602 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 243,435 blocked_by {} -906> 2014-04-25 09:18:54.768613 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -905> 2014-04-25 09:18:54.768626 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000079 0 0.000000 -904> 2014-04-25 09:18:54.768640 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -903> 2014-04-25 09:18:54.768652 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) -902> 2014-04-25 09:18:54.768665 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) -901> 2014-04-25 09:18:54.768679 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary -900> 2014-04-25 09:18:54.768688 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) -899> 2014-04-25 09:18:54.768702 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) -898> 2014-04-25 09:18:54.768714 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog -897> 2014-04-25 09:18:54.768727 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000088 0 0.000000 -896> 2014-04-25 09:18:54.768740 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -895> 2014-04-25 09:18:54.768750 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active -894> 2014-04-25 09:18:54.768763 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000024 0 0.000000 -893> 2014-04-25 09:18:54.768775 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -892> 2014-04-25 09:18:54.768786 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -891> 2014-04-25 09:18:54.768818 7f77f2cea700 10 log is not dirty -890> 2014-04-25 09:18:54.768844 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210]/[210] -889> 2014-04-25 09:18:54.768862 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -888> 2014-04-25 09:18:54.768873 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.243 info 3.2863( v 1110'677 (0'0,1110'677] local-les=3749 n=677 ec=633 les/c 3718/3718 3764/3764/3753) -887> 2014-04-25 09:18:54.768894 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.435 info 3.2863( v 1110'677 (0'0,1110'677] local-les=3754 n=677 ec=633 les/c 3754/3754 3764/3764/3753) -886> 2014-04-25 09:18:54.768910 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 210,243,435 unchanged -885> 2014-04-25 09:18:54.768922 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -884> 2014-04-25 09:18:54.768932 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -883> 2014-04-25 09:18:54.768942 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset -882> 2014-04-25 09:18:54.768953 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 125.984903 55 0.002070 -881> 2014-04-25 09:18:54.768966 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 156.737979 0 0.000000 -880> 2014-04-25 09:18:54.768979 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started/Primary 158.278308 0 0.000000 -879> 2014-04-25 09:18:54.768990 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] exit Started 158.278372 0 0.000000 -878> 2014-04-25 09:18:54.769002 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3764 lcod 0'0 mlcod 0'0 active] enter Reset -877> 2014-04-25 09:18:54.769012 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap -876> 2014-04-25 09:18:54.769024 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3780 >= end epoch 3764, nothing to do -875> 2014-04-25 09:18:54.769034 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -874> 2014-04-25 09:18:54.769044 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210,243,435] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -873> 2014-04-25 09:18:54.769064 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3764-3798 [210,243,435]/[210,243,435]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3780 generate_past_intervals interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) : primary up 3753-3797 includes interval -872> 2014-04-25 09:18:54.769075 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3764/3764/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) -871> 2014-04-25 09:18:54.769087 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 active] up [210,243,435] -> [210], acting [210,243,435] -> [210], role 0 -> 0 -870> 2014-04-25 09:18:54.769098 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -869> 2014-04-25 09:18:54.769110 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change -868> 2014-04-25 09:18:54.769125 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,243,435] -> [210], replicas changed -867> 2014-04-25 09:18:54.769138 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -866> 2014-04-25 09:18:54.769147 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -865> 2014-04-25 09:18:54.769171 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -864> 2014-04-25 09:18:54.769183 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 210,243,435 -> 210 -863> 2014-04-25 09:18:54.769194 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters -862> 2014-04-25 09:18:54.769204 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000202 1 0.000310 -861> 2014-04-25 09:18:54.769216 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started -860> 2014-04-25 09:18:54.769226 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start -859> 2014-04-25 09:18:54.769235 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -858> 2014-04-25 09:18:54.769246 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -857> 2014-04-25 09:18:54.769258 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -856> 2014-04-25 09:18:54.769269 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -855> 2014-04-25 09:18:54.769280 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -854> 2014-04-25 09:18:54.769289 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3780 -853> 2014-04-25 09:18:54.769302 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3764-3798 [210,243,435]/[210,243,435] maybe_went_rw) -852> 2014-04-25 09:18:54.769313 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.243 is down -851> 2014-04-25 09:18:54.769324 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.435 is down -850> 2014-04-25 09:18:54.769333 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210 down 243,435 blocked_by {} -849> 2014-04-25 09:18:54.769345 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -848> 2014-04-25 09:18:54.769358 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.000078 0 0.000000 -847> 2014-04-25 09:18:54.769370 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -846> 2014-04-25 09:18:54.769382 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting osd.210 3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) -845> 2014-04-25 09:18:54.769396 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting newest update on osd.210 with 3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) -844> 2014-04-25 09:18:54.769409 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] up[0](osd.210) selected as primary -843> 2014-04-25 09:18:54.769419 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] calc_acting primary is osd.210 with 3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) -842> 2014-04-25 09:18:54.769438 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] choose_acting want [210] (== acting) -841> 2014-04-25 09:18:54.769449 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog -840> 2014-04-25 09:18:54.769463 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000093 0 0.000000 -839> 2014-04-25 09:18:54.769474 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -838> 2014-04-25 09:18:54.769484 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] state: still need up_thru update before going active -837> 2014-04-25 09:18:54.769496 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000022 0 0.000000 -836> 2014-04-25 09:18:54.769508 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -835> 2014-04-25 09:18:54.769519 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -834> 2014-04-25 09:18:54.769553 7f77f2cea700 10 log is not dirty -833> 2014-04-25 09:18:54.769579 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,430,74]/[210,430,74] -832> 2014-04-25 09:18:54.769597 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -831> 2014-04-25 09:18:54.769609 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -830> 2014-04-25 09:18:54.769620 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -829> 2014-04-25 09:18:54.769630 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -828> 2014-04-25 09:18:54.769640 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -827> 2014-04-25 09:18:54.769651 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -826> 2014-04-25 09:18:54.769661 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -825> 2014-04-25 09:18:54.769673 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -824> 2014-04-25 09:18:54.769684 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.4c61( v 1156'1443 (0'0,1156'1443] local-les=3754 n=1443 ec=638 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -823> 2014-04-25 09:18:54.769696 7f77f2cea700 10 log is not dirty -822> 2014-04-25 09:18:54.769738 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,430,74]/[210,430,74] -821> 2014-04-25 09:18:54.769758 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -820> 2014-04-25 09:18:54.769770 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -819> 2014-04-25 09:18:54.769780 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -818> 2014-04-25 09:18:54.769790 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -817> 2014-04-25 09:18:54.769800 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -816> 2014-04-25 09:18:54.769810 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -815> 2014-04-25 09:18:54.769820 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -814> 2014-04-25 09:18:54.769831 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -813> 2014-04-25 09:18:54.769842 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.4c62( v 1110'673 (0'0,1110'673] local-les=3754 n=673 ec=633 les/c 3754/3798 3753/3753/3753) [210,430,74] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -812> 2014-04-25 09:18:54.769854 7f77f2cea700 10 log is not dirty -811> 2014-04-25 09:18:54.769870 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] handle_advance_map [210,195]/[210,195] -810> 2014-04-25 09:18:54.769886 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: Started advmap -809> 2014-04-25 09:18:54.769897 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -808> 2014-04-25 09:18:54.769910 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 173.320337 136 0.002477 -807> 2014-04-25 09:18:54.769924 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 173.320371 0 0.000000 -806> 2014-04-25 09:18:54.769937 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started 177.276108 0 0.000000 -805> 2014-04-25 09:18:54.769952 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] enter Reset -804> 2014-04-25 09:18:54.769964 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: Reset advmap -803> 2014-04-25 09:18:54.769975 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 -802> 2014-04-25 09:18:54.769986 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -801> 2014-04-25 09:18:54.769996 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -800> 2014-04-25 09:18:54.770019 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [33,210,195]/[33,210,195]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3754 generate_past_intervals interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) : primary up 3720-3796 includes interval -799> 2014-04-25 09:18:54.770031 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3753/3753/3720) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) -798> 2014-04-25 09:18:54.770043 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] up [33,210,195] -> [210,195], acting [33,210,195] -> [210,195], role 1 -> 0 -797> 2014-04-25 09:18:54.770056 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -796> 2014-04-25 09:18:54.770068 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] on_change -795> 2014-04-25 09:18:54.770084 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] on_role_change -794> 2014-04-25 09:18:54.770099 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -793> 2014-04-25 09:18:54.770108 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -792> 2014-04-25 09:18:54.770120 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -791> 2014-04-25 09:18:54.770130 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 195,210 -790> 2014-04-25 09:18:54.770141 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] take_waiters -789> 2014-04-25 09:18:54.770151 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000199 1 0.000235 -788> 2014-04-25 09:18:54.770177 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started -787> 2014-04-25 09:18:54.770188 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Start -786> 2014-04-25 09:18:54.770197 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -785> 2014-04-25 09:18:54.770209 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 -784> 2014-04-25 09:18:54.770221 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -783> 2014-04-25 09:18:54.770231 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -782> 2014-04-25 09:18:54.770241 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -781> 2014-04-25 09:18:54.770253 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 -780> 2014-04-25 09:18:54.770264 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) -779> 2014-04-25 09:18:54.770276 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.33 is down -778> 2014-04-25 09:18:54.770286 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [33,195]/[33,195] maybe_went_rw) -777> 2014-04-25 09:18:54.770298 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 195,210 down 33 blocked_by {} -776> 2014-04-25 09:18:54.770309 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -775> 2014-04-25 09:18:54.770321 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.195 -774> 2014-04-25 09:18:54.770335 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.525a( v 1201'1424 (0'0,1201'1424] local-les=3754 n=1424 ec=638 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -773> 2014-04-25 09:18:54.770407 7f77f2cea700 10 log is not dirty -772> 2014-04-25 09:18:54.770464 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_advance_map [344,183,210]/[344,183,210] -771> 2014-04-25 09:18:54.770484 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] state: Started advmap -770> 2014-04-25 09:18:54.770496 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -769> 2014-04-25 09:18:54.770507 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_activate_map -768> 2014-04-25 09:18:54.770518 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] take_waiters -767> 2014-04-25 09:18:54.770529 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.7060( v 1110'687 (0'0,1110'687] local-les=3768 n=687 ec=633 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -766> 2014-04-25 09:18:54.770543 7f77f2cea700 10 log is not dirty -765> 2014-04-25 09:18:54.770583 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] handle_advance_map [210,195]/[210,195] -764> 2014-04-25 09:18:54.770602 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: Started advmap -763> 2014-04-25 09:18:54.770613 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: up or acting affected, transitioning to Reset -762> 2014-04-25 09:18:54.770625 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 173.272252 136 0.002332 -761> 2014-04-25 09:18:54.770639 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started/ReplicaActive 173.272281 0 0.000000 -760> 2014-04-25 09:18:54.770651 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] exit Started 177.333309 0 0.000000 -759> 2014-04-25 09:18:54.770663 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3753 pi=2201-3752/35 luod=0'0 lcod 0'0 active] enter Reset -758> 2014-04-25 09:18:54.770674 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: Reset advmap -757> 2014-04-25 09:18:54.770685 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 -756> 2014-04-25 09:18:54.770697 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -755> 2014-04-25 09:18:54.770707 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [33,210,195] r=1 lpr=3799 pi=2201-3752/35 luod=0'0 lcod 0'0 active] state: up or acting affected, calling start_peering_interval again -754> 2014-04-25 09:18:54.770729 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [33,210,195]/[33,210,195]): not rw, up_thru 3796 up_from 3720 last_epoch_clean 3754 generate_past_intervals interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) : primary up 3720-3796 includes interval -753> 2014-04-25 09:18:54.770743 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3753/3753/3720) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) -752> 2014-04-25 09:18:54.770755 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 luod=0'0 lcod 0'0 mlcod 0'0 active] up [33,210,195] -> [210,195], acting [33,210,195] -> [210,195], role 1 -> 0 -751> 2014-04-25 09:18:54.770768 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -750> 2014-04-25 09:18:54.770779 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] on_change -749> 2014-04-25 09:18:54.770795 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] on_role_change -748> 2014-04-25 09:18:54.770806 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -747> 2014-04-25 09:18:54.770815 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -746> 2014-04-25 09:18:54.770833 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -745> 2014-04-25 09:18:54.770845 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers -> 195,210 -744> 2014-04-25 09:18:54.770855 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] take_waiters -743> 2014-04-25 09:18:54.770865 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000202 1 0.000230 -742> 2014-04-25 09:18:54.770876 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started -741> 2014-04-25 09:18:54.770886 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Start -740> 2014-04-25 09:18:54.770896 7f77f2cea700 1 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -739> 2014-04-25 09:18:54.770907 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -738> 2014-04-25 09:18:54.770920 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -737> 2014-04-25 09:18:54.770930 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -736> 2014-04-25 09:18:54.770941 7f77f2cea700 5 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -735> 2014-04-25 09:18:54.770951 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 -734> 2014-04-25 09:18:54.770964 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [33,210,195]/[33,210,195] maybe_went_rw) -733> 2014-04-25 09:18:54.770976 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.33 is down -732> 2014-04-25 09:18:54.770982 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -731> 2014-04-25 09:18:54.770986 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3751-3752 [33,195]/[33,195] maybe_went_rw) -730> 2014-04-25 09:18:54.770998 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 195,210 down 33 blocked_by {} -729> 2014-04-25 09:18:54.771009 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -728> 2014-04-25 09:18:54.771022 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.195 -727> 2014-04-25 09:18:54.771035 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.525b( v 1110'726 (0'0,1110'726] local-les=3754 n=726 ec=633 les/c 3754/3754 3799/3799/3799) [210,195] r=0 lpr=3799 pi=2201-3798/36 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -726> 2014-04-25 09:18:54.771106 7f77f2cea700 10 log is not dirty -725> 2014-04-25 09:18:54.771133 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_advance_map [71,352,210]/[71,352,210] -724> 2014-04-25 09:18:54.771324 7f77f22e9700 10 osd.210 pg_epoch: 3798 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,172,132]/[210,172,132] -723> 2014-04-25 09:18:54.771347 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -722> 2014-04-25 09:18:54.771359 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -721> 2014-04-25 09:18:54.771370 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -720> 2014-04-25 09:18:54.771381 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -719> 2014-04-25 09:18:54.771397 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -718> 2014-04-25 09:18:54.771407 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -717> 2014-04-25 09:18:54.771417 7f77f22e9700 7 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -716> 2014-04-25 09:18:54.771429 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] take_waiters -715> 2014-04-25 09:18:54.771440 7f77f22e9700 10 osd.210 pg_epoch: 3799 pg[4.1010( v 1110'1425 (0'0,1110'1425] local-les=3762 n=1425 ec=638 les/c 3762/3763 3756/3756/3753) [210,172,132] r=0 lpr=3756 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -714> 2014-04-25 09:18:54.771453 7f77f22e9700 10 log is not dirty -713> 2014-04-25 09:18:54.771984 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[4.798b( v 1138'1388 (0'0,1138'1388] local-les=3754 n=1388 ec=638 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] flushed -712> 2014-04-25 09:18:54.772013 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[3.798c( v 1110'663 (0'0,1110'663] local-les=3754 n=663 ec=633 les/c 3754/3715 3799/3799/3799) [210] r=0 lpr=3799 pi=3099-3798/16 lcod 0'0 mlcod 0'0 peering] flushed -711> 2014-04-25 09:18:54.772811 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[4.2862( v 1110'1427 (0'0,1110'1427] local-les=3765 n=1426 ec=638 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] flushed -710> 2014-04-25 09:18:54.772836 7f78019e7700 10 osd.210 pg_epoch: 3799 pg[3.2863( v 1110'677 (0'0,1110'677] local-les=3765 n=677 ec=633 les/c 3765/3780 3799/3799/3753) [210] r=0 lpr=3799 pi=3764-3798/1 lcod 0'0 mlcod 0'0 peering] flushed -709> 2014-04-25 09:18:54.771152 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] state: Started advmap -708> 2014-04-25 09:18:55.052913 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -707> 2014-04-25 09:18:55.052948 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_activate_map -706> 2014-04-25 09:18:55.052962 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] take_waiters -705> 2014-04-25 09:18:55.052974 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.6459( v 1195'1415 (0'0,1195'1415] local-les=3779 n=1415 ec=638 les/c 3779/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -704> 2014-04-25 09:18:55.052989 7f77f2cea700 10 log is not dirty -703> 2014-04-25 09:18:55.053069 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_advance_map [71,352,210]/[71,352,210] -702> 2014-04-25 09:18:55.053091 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] state: Started advmap -701> 2014-04-25 09:18:55.053104 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -700> 2014-04-25 09:18:55.053115 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_activate_map -699> 2014-04-25 09:18:55.053127 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] take_waiters -698> 2014-04-25 09:18:55.053138 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.645a( v 1110'726 (0'0,1110'726] local-les=3780 n=726 ec=633 les/c 3780/3782 3755/3755/3755) [71,352,210] r=2 lpr=3755 pi=1915-3754/39 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -697> 2014-04-25 09:18:55.053152 7f77f2cea700 10 log is not dirty -696> 2014-04-25 09:18:55.053179 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_advance_map [182,210,62]/[182,210,62] -695> 2014-04-25 09:18:55.053198 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] state: Started advmap -694> 2014-04-25 09:18:55.053210 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -693> 2014-04-25 09:18:55.053222 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_activate_map -692> 2014-04-25 09:18:55.053232 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] take_waiters -691> 2014-04-25 09:18:55.053243 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.5850( v 1110'1489 (0'0,1110'1489] local-les=3769 n=1489 ec=638 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -690> 2014-04-25 09:18:55.053256 7f77f2cea700 10 log is not dirty -689> 2014-04-25 09:18:55.053300 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,384]/[210,384] -688> 2014-04-25 09:18:55.053322 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: Peering advmap -687> 2014-04-25 09:18:55.053336 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -686> 2014-04-25 09:18:55.053347 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: Started advmap -685> 2014-04-25 09:18:55.053358 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -684> 2014-04-25 09:18:55.053370 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_activate_map -683> 2014-04-25 09:18:55.053381 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary -682> 2014-04-25 09:18:55.053394 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] take_waiters -681> 2014-04-25 09:18:55.053405 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.464a( v 1110'1366 (0'0,1110'1366] local-les=3755 n=1366 ec=638 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -680> 2014-04-25 09:18:55.053419 7f77f2cea700 10 log is not dirty -679> 2014-04-25 09:18:55.053455 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_advance_map [344,183,210]/[344,183,210] -678> 2014-04-25 09:18:55.053477 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] state: Started advmap -677> 2014-04-25 09:18:55.053489 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -676> 2014-04-25 09:18:55.053501 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_activate_map -675> 2014-04-25 09:18:55.053512 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] take_waiters -674> 2014-04-25 09:18:55.053522 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[4.705f( v 1110'1358 (0'0,1110'1358] local-les=3768 n=1358 ec=638 les/c 3768/3780 3766/3766/3726) [344,183,210] r=2 lpr=3766 pi=3115-3765/18 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -673> 2014-04-25 09:18:55.053535 7f77f2cea700 10 log is not dirty -672> 2014-04-25 09:18:55.053551 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_advance_map [182,210,62]/[182,210,62] -671> 2014-04-25 09:18:55.053568 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] state: Started advmap -670> 2014-04-25 09:18:55.053579 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -669> 2014-04-25 09:18:55.053590 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_activate_map -668> 2014-04-25 09:18:55.053601 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] take_waiters -667> 2014-04-25 09:18:55.053611 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.5851( v 1110'697 (0'0,1110'697] local-les=3769 n=697 ec=633 les/c 3769/3769 3768/3768/3768) [182,210,62] r=1 lpr=3768 pi=3761-3767/1 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -666> 2014-04-25 09:18:55.053625 7f77f2cea700 10 log is not dirty -665> 2014-04-25 09:18:55.053639 7f77f2cea700 10 osd.210 pg_epoch: 3798 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,384]/[210,384] -664> 2014-04-25 09:18:55.053656 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: Peering advmap -663> 2014-04-25 09:18:55.053668 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -662> 2014-04-25 09:18:55.053678 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: Started advmap -661> 2014-04-25 09:18:55.053688 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -660> 2014-04-25 09:18:55.053698 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_activate_map -659> 2014-04-25 09:18:55.053709 7f77f2cea700 7 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary -658> 2014-04-25 09:18:55.053720 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] take_waiters -657> 2014-04-25 09:18:55.053731 7f77f2cea700 10 osd.210 pg_epoch: 3799 pg[3.464b( v 1110'656 (0'0,1110'656] local-les=3755 n=656 ec=633 les/c 3755/3769 3798/3798/3798) [210,384] r=0 lpr=3798 pi=3718-3797/9 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -656> 2014-04-25 09:18:55.053745 7f77f2cea700 10 log is not dirty -655> 2014-04-25 09:18:54.773112 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] -654> 2014-04-25 09:18:55.054072 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] recovery raced and were queued twice, ignoring! -653> 2014-04-25 09:18:55.054095 7f77edce2700 10 osd.210 3800 do_recovery started 0/5 on pg[3.6109( v 1110'710 (0'0,1110'710] local-les=3754 n=710 ec=633 les/c 3754/3718 3753/3753/3753) [210,309,198] r=0 lpr=3753 pi=3716-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] -652> 2014-04-25 09:18:55.054107 7f77edce2700 10 log is not dirty -651> 2014-04-25 09:18:54.773146 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,434]/[210,434] -650> 2014-04-25 09:18:55.054242 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: Peering advmap -649> 2014-04-25 09:18:55.054265 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -648> 2014-04-25 09:18:55.054277 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: Started advmap -647> 2014-04-25 09:18:55.054288 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -646> 2014-04-25 09:18:55.054299 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_activate_map -645> 2014-04-25 09:18:55.054312 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary -644> 2014-04-25 09:18:55.054324 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] take_waiters -643> 2014-04-25 09:18:55.054336 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.21f0( v 1110'723 (0'0,1110'723] local-les=3764 n=723 ec=633 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -642> 2014-04-25 09:18:55.054351 7f77ef0e4700 10 log is not dirty -641> 2014-04-25 09:18:55.054374 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,7]/[210,7] -640> 2014-04-25 09:18:55.054391 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -639> 2014-04-25 09:18:55.054403 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] dropping down osd.14 info 3.1ef1( v 1139'748 (0'0,1139'748] local-les=3731 n=748 ec=633 les/c 3731/3742 3753/3753/3753) -638> 2014-04-25 09:18:55.054422 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] update_heartbeat_peers 7,14,210 unchanged -637> 2014-04-25 09:18:55.054434 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -636> 2014-04-25 09:18:55.054444 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -635> 2014-04-25 09:18:55.054454 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: up or acting affected, transitioning to Reset -634> 2014-04-25 09:18:55.054467 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 173.435357 133 0.008101 -633> 2014-04-25 09:18:55.054483 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active 173.913060 0 0.000000 -632> 2014-04-25 09:18:55.054498 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started/Primary 175.949633 0 0.000000 -631> 2014-04-25 09:18:55.054511 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] exit Started 175.949758 0 0.000000 -630> 2014-04-25 09:18:55.054522 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active] enter Reset -629> 2014-04-25 09:18:55.054534 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: Reset advmap -628> 2014-04-25 09:18:55.054549 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range start epoch 3754 >= end epoch 3753, nothing to do -627> 2014-04-25 09:18:55.054561 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -626> 2014-04-25 09:18:55.054570 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,14,7] r=0 lpr=3799 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -625> 2014-04-25 09:18:55.054594 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3753-3798 [210,14,7]/[210,14,7]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) : primary up 3753-3797 includes interval -624> 2014-04-25 09:18:55.054609 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3753/3753/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] noting past interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) -623> 2014-04-25 09:18:55.054621 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 active] up [210,14,7] -> [210,7], acting [210,14,7] -> [210,7], role 0 -> 0 -622> 2014-04-25 09:18:55.054635 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -621> 2014-04-25 09:18:55.054647 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] on_change -620> 2014-04-25 09:18:55.054663 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] [210,14,7] -> [210,7], replicas changed -619> 2014-04-25 09:18:55.054674 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -618> 2014-04-25 09:18:55.054684 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -617> 2014-04-25 09:18:55.054696 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -616> 2014-04-25 09:18:55.054707 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 7,14,210 -> 7,210 -615> 2014-04-25 09:18:55.054718 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] take_waiters -614> 2014-04-25 09:18:55.054729 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000207 1 0.000307 -613> 2014-04-25 09:18:55.054741 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started -612> 2014-04-25 09:18:55.054751 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Start -611> 2014-04-25 09:18:55.054761 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -610> 2014-04-25 09:18:55.054772 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -609> 2014-04-25 09:18:55.054785 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -608> 2014-04-25 09:18:55.054796 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -607> 2014-04-25 09:18:55.054806 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -606> 2014-04-25 09:18:55.054818 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 -605> 2014-04-25 09:18:55.054830 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3798 [210,14,7]/[210,14,7] maybe_went_rw) -604> 2014-04-25 09:18:55.054844 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.14 is down -603> 2014-04-25 09:18:55.054854 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 7,210 down 14 blocked_by {} -602> 2014-04-25 09:18:55.054865 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -601> 2014-04-25 09:18:55.054878 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.7 -600> 2014-04-25 09:18:55.054894 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.1ef1( v 1139'748 (0'0,1139'748] local-les=3754 n=748 ec=633 les/c 3754/3754 3799/3799/3753) [210,7] r=0 lpr=3799 pi=3753-3798/1 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -599> 2014-04-25 09:18:55.054934 7f77ef0e4700 10 log is not dirty -598> 2014-04-25 09:18:55.054981 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_advance_map [67,88,210]/[67,88,210] -597> 2014-04-25 09:18:55.055001 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] state: Started advmap -596> 2014-04-25 09:18:55.055012 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -595> 2014-04-25 09:18:55.055023 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_activate_map -594> 2014-04-25 09:18:55.055035 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] take_waiters -593> 2014-04-25 09:18:55.055046 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.7308( v 1110'709 (0'0,1110'709] local-les=3777 n=709 ec=633 les/c 3777/3779 3753/3753/3753) [67,88,210] r=2 lpr=3753 pi=3718-3752/6 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -592> 2014-04-25 09:18:55.055058 7f77ef0e4700 10 log is not dirty -591> 2014-04-25 09:18:55.055075 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,57]/[210,57] -590> 2014-04-25 09:18:55.055093 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -589> 2014-04-25 09:18:55.055108 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.201 info 3.4bfd( v 1110'721 lc 0'0 (0'0,1110'721] local-les=3748 n=721 ec=633 les/c 3748/3715 3756/3756/3753) -588> 2014-04-25 09:18:55.055173 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 57,201,210 unchanged -587> 2014-04-25 09:18:55.055187 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -586> 2014-04-25 09:18:55.055198 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -585> 2014-04-25 09:18:55.055207 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset -584> 2014-04-25 09:18:55.055220 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 162.230589 118 0.006154 -583> 2014-04-25 09:18:55.055233 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 162.231052 0 0.000000 -582> 2014-04-25 09:18:55.055249 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] exit Started/Primary 170.954643 0 0.000000 -581> 2014-04-25 09:18:55.055261 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] exit Started 170.954700 0 0.000000 -580> 2014-04-25 09:18:55.055273 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] enter Reset -579> 2014-04-25 09:18:55.055285 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] state: Reset advmap -578> 2014-04-25 09:18:55.055295 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 -577> 2014-04-25 09:18:55.055306 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -576> 2014-04-25 09:18:55.055316 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -575> 2014-04-25 09:18:55.055338 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3756-3798 [210,201,57]/[210,201,57]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3715 generate_past_intervals interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) : primary up 3753-3797 includes interval -574> 2014-04-25 09:18:55.055349 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3756/3756/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] noting past interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) -573> 2014-04-25 09:18:55.055361 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] up [210,201,57] -> [210,57], acting [210,201,57] -> [210,57], role 0 -> 0 -572> 2014-04-25 09:18:55.055374 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -571> 2014-04-25 09:18:55.055394 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] on_change -570> 2014-04-25 09:18:55.055410 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] [210,201,57] -> [210,57], replicas changed -569> 2014-04-25 09:18:55.055421 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -568> 2014-04-25 09:18:55.055430 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -567> 2014-04-25 09:18:55.055442 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -566> 2014-04-25 09:18:55.055454 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 57,201,210 -> 57,210 -565> 2014-04-25 09:18:55.055465 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] take_waiters -564> 2014-04-25 09:18:55.055475 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000202 1 0.000351 -563> 2014-04-25 09:18:55.055487 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started -562> 2014-04-25 09:18:55.055498 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Start -561> 2014-04-25 09:18:55.055508 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -560> 2014-04-25 09:18:55.055519 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -559> 2014-04-25 09:18:55.055531 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -558> 2014-04-25 09:18:55.055542 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -557> 2014-04-25 09:18:55.055552 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -556> 2014-04-25 09:18:55.055562 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 -555> 2014-04-25 09:18:55.055573 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) -554> 2014-04-25 09:18:55.055584 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.201 is down -553> 2014-04-25 09:18:55.055594 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3755 [210,57]/[210,57] maybe_went_rw) -552> 2014-04-25 09:18:55.055606 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 57,210 down 201 blocked_by {} -551> 2014-04-25 09:18:55.055617 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -550> 2014-04-25 09:18:55.055629 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.57 -549> 2014-04-25 09:18:55.055643 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.4bfd( v 1110'721 (0'0,1110'721] local-les=3761 n=721 ec=633 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -548> 2014-04-25 09:18:55.055684 7f77ef0e4700 10 log is not dirty -547> 2014-04-25 09:18:55.055710 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_advance_map [210,434]/[210,434] -546> 2014-04-25 09:18:55.055729 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: Peering advmap -545> 2014-04-25 09:18:55.055742 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -544> 2014-04-25 09:18:55.055753 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: Started advmap -543> 2014-04-25 09:18:55.055764 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] check_recovery_sources no source osds () went down -542> 2014-04-25 09:18:55.055775 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_activate_map -541> 2014-04-25 09:18:55.055786 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] state: handle ActMap primary -540> 2014-04-25 09:18:55.055798 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] take_waiters -539> 2014-04-25 09:18:55.055810 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.21ef( v 1110'1474 (0'0,1110'1474] local-les=3764 n=1474 ec=638 les/c 3764/2305 3798/3798/3753) [210,434] r=0 lpr=3798 pi=1850-3797/28 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -538> 2014-04-25 09:18:55.055823 7f77ef0e4700 10 log is not dirty -537> 2014-04-25 09:18:55.055839 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,44,365]/[210,44,365] -536> 2014-04-25 09:18:55.055857 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -535> 2014-04-25 09:18:55.055868 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -534> 2014-04-25 09:18:55.055878 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -533> 2014-04-25 09:18:55.055889 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -532> 2014-04-25 09:18:55.055899 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -531> 2014-04-25 09:18:55.055909 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -530> 2014-04-25 09:18:55.055921 7f77ef0e4700 7 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -529> 2014-04-25 09:18:55.055933 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] take_waiters -528> 2014-04-25 09:18:55.055944 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.54f8( v 1110'1457 (0'0,1110'1457] local-les=3766 n=1456 ec=638 les/c 3766/3767 3765/3765/3753) [210,44,365] r=0 lpr=3765 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -527> 2014-04-25 09:18:55.055956 7f77ef0e4700 10 log is not dirty -526> 2014-04-25 09:18:55.055976 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,335]/[210,335] -525> 2014-04-25 09:18:55.055994 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -524> 2014-04-25 09:18:55.056005 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.289 info 3.18e4( v 1196'732 (0'0,1196'732] local-les=3754 n=732 ec=633 les/c 3754/3754 3755/3755/3753) -523> 2014-04-25 09:18:55.056023 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 210,289,335 unchanged -522> 2014-04-25 09:18:55.056034 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -521> 2014-04-25 09:18:55.056044 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -520> 2014-04-25 09:18:55.056054 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset -519> 2014-04-25 09:18:55.056066 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 162.845289 115 0.005654 -518> 2014-04-25 09:18:55.056079 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 162.845809 0 0.000000 -517> 2014-04-25 09:18:55.056095 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started/Primary 172.293876 0 0.000000 -516> 2014-04-25 09:18:55.056106 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started 172.293946 0 0.000000 -515> 2014-04-25 09:18:55.056117 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] enter Reset -514> 2014-04-25 09:18:55.056129 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: Reset advmap -513> 2014-04-25 09:18:55.056139 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 -512> 2014-04-25 09:18:55.056149 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -511> 2014-04-25 09:18:55.056166 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -510> 2014-04-25 09:18:55.056187 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3755-3798 [210,289,335]/[210,289,335]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) : primary up 3753-3797 includes interval -509> 2014-04-25 09:18:55.056200 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3755/3755/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] noting past interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) -508> 2014-04-25 09:18:55.056211 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] up [210,289,335] -> [210,335], acting [210,289,335] -> [210,335], role 0 -> 0 -507> 2014-04-25 09:18:55.056264 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -506> 2014-04-25 09:18:55.056276 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change -505> 2014-04-25 09:18:55.056292 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] [210,289,335] -> [210,335], replicas changed -504> 2014-04-25 09:18:55.056305 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -503> 2014-04-25 09:18:55.056314 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -502> 2014-04-25 09:18:55.056326 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -501> 2014-04-25 09:18:55.056337 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 210,289,335 -> 210,335 -500> 2014-04-25 09:18:55.056348 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters -499> 2014-04-25 09:18:55.056358 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000240 1 0.000333 -498> 2014-04-25 09:18:55.056369 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started -497> 2014-04-25 09:18:55.056379 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start -496> 2014-04-25 09:18:55.056389 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -495> 2014-04-25 09:18:55.056400 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000021 0 0.000000 -494> 2014-04-25 09:18:55.056412 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -493> 2014-04-25 09:18:55.056422 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -492> 2014-04-25 09:18:55.056432 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -491> 2014-04-25 09:18:55.056442 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 -490> 2014-04-25 09:18:55.056456 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) -489> 2014-04-25 09:18:55.056468 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.289 is down -488> 2014-04-25 09:18:55.056478 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3754 [210,289]/[210,289] maybe_went_rw) -487> 2014-04-25 09:18:55.056490 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,335 down 289 blocked_by {} -486> 2014-04-25 09:18:55.056500 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -485> 2014-04-25 09:18:55.056512 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.335 -484> 2014-04-25 09:18:55.056525 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[3.18e4( v 1196'732 (0'0,1196'732] local-les=3761 n=732 ec=633 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -483> 2014-04-25 09:18:55.056559 7f77ef0e4700 10 log is not dirty -482> 2014-04-25 09:18:55.056617 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,335]/[210,335] -481> 2014-04-25 09:18:55.056638 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -480> 2014-04-25 09:18:55.056651 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.289 info 4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3754 n=1514 ec=638 les/c 3754/3754 3755/3755/3753) -479> 2014-04-25 09:18:55.056668 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 210,289,335 unchanged -478> 2014-04-25 09:18:55.056680 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -477> 2014-04-25 09:18:55.056691 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -476> 2014-04-25 09:18:55.056700 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset -475> 2014-04-25 09:18:55.056712 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 162.865985 115 0.005636 -474> 2014-04-25 09:18:55.056725 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 162.866614 0 0.000000 -473> 2014-04-25 09:18:55.056740 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started/Primary 172.292423 0 0.000000 -472> 2014-04-25 09:18:55.056752 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] exit Started 172.292490 0 0.000000 -471> 2014-04-25 09:18:55.056764 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3755 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] enter Reset -470> 2014-04-25 09:18:55.056775 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: Reset advmap -469> 2014-04-25 09:18:55.056785 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3754 -468> 2014-04-25 09:18:55.056796 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -467> 2014-04-25 09:18:55.056806 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,289,335] r=0 lpr=3799 pi=3753-3754/1 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -466> 2014-04-25 09:18:55.056826 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3755-3798 [210,289,335]/[210,289,335]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3754 generate_past_intervals interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) : primary up 3753-3797 includes interval -465> 2014-04-25 09:18:55.056838 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3755/3755/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] noting past interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) -464> 2014-04-25 09:18:55.056850 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 active] up [210,289,335] -> [210,335], acting [210,289,335] -> [210,335], role 0 -> 0 -463> 2014-04-25 09:18:55.056949 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -462> 2014-04-25 09:18:55.056962 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] on_change -461> 2014-04-25 09:18:55.056979 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] [210,289,335] -> [210,335], replicas changed -460> 2014-04-25 09:18:55.056990 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -459> 2014-04-25 09:18:55.056999 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -458> 2014-04-25 09:18:55.057011 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -457> 2014-04-25 09:18:55.057022 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 210,289,335 -> 210,335 -456> 2014-04-25 09:18:55.057034 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] take_waiters -455> 2014-04-25 09:18:55.057045 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000280 1 0.000374 -454> 2014-04-25 09:18:55.057056 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started -453> 2014-04-25 09:18:55.057066 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Start -452> 2014-04-25 09:18:55.057076 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -451> 2014-04-25 09:18:55.057086 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000020 0 0.000000 -450> 2014-04-25 09:18:55.057098 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -449> 2014-04-25 09:18:55.057108 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -448> 2014-04-25 09:18:55.057119 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -447> 2014-04-25 09:18:55.057128 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3754 -446> 2014-04-25 09:18:55.057140 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3755-3798 [210,289,335]/[210,289,335] maybe_went_rw) -445> 2014-04-25 09:18:55.057151 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.289 is down -444> 2014-04-25 09:18:55.057174 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3754 [210,289]/[210,289] maybe_went_rw) -443> 2014-04-25 09:18:55.057186 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 210,335 down 289 blocked_by {} -442> 2014-04-25 09:18:55.057197 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -441> 2014-04-25 09:18:55.057210 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.335 -440> 2014-04-25 09:18:55.057223 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.18e3( v 1110'1514 (0'0,1110'1514] local-les=3761 n=1514 ec=638 les/c 3761/3754 3799/3799/3753) [210,335] r=0 lpr=3799 pi=3753-3798/2 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -439> 2014-04-25 09:18:55.057259 7f77ef0e4700 10 log is not dirty -438> 2014-04-25 09:18:55.057290 7f77ef0e4700 10 osd.210 pg_epoch: 3798 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,57]/[210,57] -437> 2014-04-25 09:18:55.057309 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -436> 2014-04-25 09:18:55.057321 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] dropping down osd.201 info 4.4bfc( v 1110'1416 lc 0'0 (0'0,1110'1416] local-les=3748 n=1416 ec=638 les/c 3748/3715 3756/3756/3753) -435> 2014-04-25 09:18:55.057424 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] update_heartbeat_peers 57,201,210 unchanged -434> 2014-04-25 09:18:55.057436 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -433> 2014-04-25 09:18:55.057448 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -432> 2014-04-25 09:18:55.057457 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] state: up or acting affected, transitioning to Reset -431> 2014-04-25 09:18:55.057470 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 162.247034 117 0.161419 -430> 2014-04-25 09:18:55.057483 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active 162.247613 0 0.000000 -429> 2014-04-25 09:18:55.057499 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] exit Started/Primary 170.954834 0 0.000000 -428> 2014-04-25 09:18:55.057510 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] exit Started 170.954891 0 0.000000 -427> 2014-04-25 09:18:55.057522 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3756 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] enter Reset -426> 2014-04-25 09:18:55.057533 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] state: Reset advmap -425> 2014-04-25 09:18:55.057544 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] _calc_past_interval_range: already have past intervals back to 3715 -424> 2014-04-25 09:18:55.057556 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] check_recovery_sources no source osds () went down -423> 2014-04-25 09:18:55.057566 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,201,57] r=0 lpr=3799 pi=3714-3755/6 lcod 0'0 mlcod 0'0 active] state: up or acting affected, calling start_peering_interval again -422> 2014-04-25 09:18:55.057588 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] start_peering_interval: check_new_interval output: generate_past_intervals interval(3756-3798 [210,201,57]/[210,201,57]): not rw, up_thru 3797 up_from 3753 last_epoch_clean 3715 generate_past_intervals interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) : primary up 3753-3797 includes interval -421> 2014-04-25 09:18:55.057600 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3756/3756/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] noting past interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) -420> 2014-04-25 09:18:55.057610 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 active] up [210,201,57] -> [210,57], acting [210,201,57] -> [210,57], role 0 -> 0 -419> 2014-04-25 09:18:55.057622 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] clear_primary_state -418> 2014-04-25 09:18:55.057635 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] on_change -417> 2014-04-25 09:18:55.057651 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] [210,201,57] -> [210,57], replicas changed -416> 2014-04-25 09:18:55.057661 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] cancel_recovery -415> 2014-04-25 09:18:55.057670 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] clear_recovery_state -414> 2014-04-25 09:18:55.057682 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] handle_activate_map -413> 2014-04-25 09:18:55.057694 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] update_heartbeat_peers 57,201,210 -> 57,210 -412> 2014-04-25 09:18:55.057706 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] take_waiters -411> 2014-04-25 09:18:55.057718 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] exit Reset 0.000197 1 0.000374 -410> 2014-04-25 09:18:55.057732 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started -409> 2014-04-25 09:18:55.057743 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Start -408> 2014-04-25 09:18:55.057754 7f77ef0e4700 1 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] state: transitioning to Primary -407> 2014-04-25 09:18:55.057766 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] exit Start 0.000023 0 0.000000 -406> 2014-04-25 09:18:55.057779 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary -405> 2014-04-25 09:18:55.057789 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Peering -404> 2014-04-25 09:18:55.057801 7f77ef0e4700 5 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -403> 2014-04-25 09:18:55.057811 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] _calc_past_interval_range: already have past intervals back to 3715 -402> 2014-04-25 09:18:55.057823 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3756-3798 [210,201,57]/[210,201,57] maybe_went_rw) -401> 2014-04-25 09:18:55.057837 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.201 is down -400> 2014-04-25 09:18:55.057848 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior interval(3753-3755 [210,57]/[210,57] maybe_went_rw) -399> 2014-04-25 09:18:55.057861 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 57,210 down 201 blocked_by {} -398> 2014-04-25 09:18:55.057872 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] up_thru 3797 < same_since 3799, must notify monitor -397> 2014-04-25 09:18:55.057885 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] state: querying info from osd.57 -396> 2014-04-25 09:18:55.057898 7f77ef0e4700 10 osd.210 pg_epoch: 3799 pg[4.4bfc( v 1110'1416 (0'0,1110'1416] local-les=3761 n=1416 ec=638 les/c 3761/3715 3799/3799/3753) [210,57] r=0 lpr=3799 pi=3714-3798/7 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -395> 2014-04-25 09:18:55.057942 7f77ef0e4700 10 log is not dirty -394> 2014-04-25 09:18:54.773389 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_advance_map [426,210,432]/[426,210,432] -393> 2014-04-25 09:18:54.776072 7f77a69a2700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -392> 2014-04-25 09:18:55.057983 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] state: Started advmap -391> 2014-04-25 09:18:55.058003 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] check_recovery_sources no source osds () went down -390> 2014-04-25 09:18:55.058017 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_activate_map -389> 2014-04-25 09:18:55.058028 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] take_waiters -388> 2014-04-25 09:18:55.058040 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.1f00( v 1175'1383 (0'0,1175'1383] local-les=3754 n=1383 ec=638 les/c 3754/3754 3753/3753/3726) [426,210,432] r=1 lpr=3753 pi=3119-3752/11 luod=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -387> 2014-04-25 09:18:55.058053 7f77ee6e3700 10 log is not dirty -386> 2014-04-25 09:18:54.778173 7f77c1e4f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -385> 2014-04-25 09:18:55.058073 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_advance_map [210,452,445]/[210,452,445] -384> 2014-04-25 09:18:55.058091 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active advmap -383> 2014-04-25 09:18:55.058102 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -382> 2014-04-25 09:18:54.778224 7f77c8dc4700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -381> 2014-04-25 09:18:55.058112 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Started advmap -380> 2014-04-25 09:18:55.058122 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] check_recovery_sources no source osds () went down -379> 2014-04-25 09:18:55.058133 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_activate_map -378> 2014-04-25 09:18:55.058144 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: Active: handling ActMap -377> 2014-04-25 09:18:55.058155 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] state: handle ActMap primary -376> 2014-04-25 09:18:54.783573 7f77dac2d700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.179:0/97831 pipe(0x119a0780 sd=137 :6919 s=2 pgs=3358 cs=1 l=1 c=0x12426000).reader couldn't read tag, Success -375> 2014-04-25 09:18:55.058173 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] take_waiters -374> 2014-04-25 09:18:55.058184 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.4c16( v 1176'1446 (0'0,1176'1446] local-les=3754 n=1446 ec=638 les/c 3754/3754 3753/3753/3753) [210,452,445] r=0 lpr=3753 lcod 0'0 mlcod 0'0 active+clean] handle_peering_event: epoch_sent: 3799 epoch_requested: 3799 NullEvt -373> 2014-04-25 09:18:55.058196 7f77ee6e3700 10 log is not dirty -372> 2014-04-25 09:18:55.058190 7f77dac2d700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.179:0/97831 pipe(0x119a0780 sd=137 :6919 s=2 pgs=3358 cs=1 l=1 c=0x12426000).fault 0: Success -371> 2014-04-25 09:18:55.058213 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,221,153]/[210,221,153] -370> 2014-04-25 09:18:55.058232 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -369> 2014-04-25 09:18:55.058244 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -368> 2014-04-25 09:18:55.058255 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -367> 2014-04-25 09:18:54.784810 7f77a659e700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -366> 2014-04-25 09:18:55.058267 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -365> 2014-04-25 09:18:55.058280 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -364> 2014-04-25 09:18:55.058291 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -363> 2014-04-25 09:18:55.058315 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] -362> 2014-04-25 09:18:55.058325 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -361> 2014-04-25 09:18:55.058340 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -360> 2014-04-25 09:18:55.058351 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3754 epoch_requested: 3754 LocalRecoveryReserved -359> 2014-04-25 09:18:55.058365 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 173.487659 137 3.004374 -358> 2014-04-25 09:18:55.058378 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/WaitRemoteRecoveryReserved -357> 2014-04-25 09:18:54.784865 7f77d063c700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -356> 2014-04-25 09:18:55.058443 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitRemoteRecoveryReserved 0.000065 0 0.000000 -355> 2014-04-25 09:18:55.058459 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/Recovering -354> 2014-04-25 09:18:54.784922 7f77d4278700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -353> 2014-04-25 09:18:55.058472 7f77ee6e3700 10 osd.210 3799 queue_for_recovery already queued pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] -352> 2014-04-25 09:18:55.058484 7f77ee6e3700 10 log is not dirty -351> 2014-04-25 09:18:55.058501 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,193,171]/[210,193,171] -350> 2014-04-25 09:18:54.784967 7f77a75ae700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -349> 2014-04-25 09:18:55.058520 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -348> 2014-04-25 09:18:55.058532 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -347> 2014-04-25 09:18:55.058543 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -346> 2014-04-25 09:18:55.058554 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -345> 2014-04-25 09:18:55.058565 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -344> 2014-04-25 09:18:54.786627 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.172 10.193.207.185:6802/2012373 42 ==== osd_ping(ping_reply e3798 stamp 2014-04-25 09:18:47.960267) v2 ==== 47+0+0 (31478524 0 0) 0x1c45e8c0 con 0x123b54a0 -343> 2014-04-25 09:18:55.058575 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -342> 2014-04-25 09:18:55.058586 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] -341> 2014-04-25 09:18:55.058595 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -340> 2014-04-25 09:18:55.058608 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -339> 2014-04-25 09:18:55.058608 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.420 10.193.207.186:6979/3012241 38 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:47.960267) v2 ==== 47+0+0 (2034709105 0 0) 0x12325180 con 0x104e6b00 -338> 2014-04-25 09:18:55.058619 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3754 epoch_requested: 3754 LocalRecoveryReserved -337> 2014-04-25 09:18:55.058630 7f77f72f1700 10 osd.210 3800 handle_osd_ping canceling queued failure report for osd.420 -336> 2014-04-25 09:18:55.058632 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 173.322826 138 0.006657 -335> 2014-04-25 09:18:55.058645 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.5d97( v 1110'1442 (0'0,1110'1442] local-les=3754 n=1442 ec=638 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/WaitRemoteRecoveryReserved -334> 2014-04-25 09:18:55.058645 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.200 10.193.207.176:7019/4060413 46 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x15e36540 con 0x10dee000 -333> 2014-04-25 09:18:55.058659 7f77ee6e3700 10 log is not dirty -332> 2014-04-25 09:18:55.058665 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.309 10.193.207.189:6993/2084267 46 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x12eb0000 con 0x10dfa9a0 -331> 2014-04-25 09:18:55.058679 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.309 has 3799 -330> 2014-04-25 09:18:55.058686 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.309 10.193.207.189:6994/2084267 46 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x17e88c40 con 0x12727080 -329> 2014-04-25 09:18:55.058699 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,193,171]/[210,193,171] -328> 2014-04-25 09:18:55.058710 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.105 10.193.207.179:6839/6097831 46 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x1426ae00 con 0x12651a20 -327> 2014-04-25 09:18:55.058721 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -326> 2014-04-25 09:18:55.058728 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.221 10.193.207.177:6984/1036197 46 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x11762e00 con 0x132ac840 -325> 2014-04-25 09:18:55.058734 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -324> 2014-04-25 09:18:55.058744 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.121 10.193.207.177:6810/3034909 45 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x14229c00 con 0x127ba580 -323> 2014-04-25 09:18:55.058746 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -322> 2014-04-25 09:18:55.058757 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -321> 2014-04-25 09:18:55.058759 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.172 10.193.207.185:6802/2012373 43 ==== osd_ping(ping_reply e3800 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (3058409455 0 0) 0x1c45e700 con 0x123b54a0 -320> 2014-04-25 09:18:55.058770 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.172 has 3800 -319> 2014-04-25 09:18:55.058769 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -318> 2014-04-25 09:18:55.058776 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.172 10.193.207.185:6803/2012373 42 ==== osd_ping(ping_reply e3798 stamp 2014-04-25 09:18:47.960267) v2 ==== 47+0+0 (31478524 0 0) 0x187fcc40 con 0x133b18c0 -317> 2014-04-25 09:18:55.058780 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -316> 2014-04-25 09:18:55.058792 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] -315> 2014-04-25 09:18:55.058790 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.371 10.193.207.178:6914/3050164 37 ==== osd_ping(ping_reply e3798 stamp 2014-04-25 09:18:47.960267) v2 ==== 47+0+0 (31478524 0 0) 0x12e16540 con 0x1328f8c0 -314> 2014-04-25 09:18:55.058800 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -313> 2014-04-25 09:18:55.058806 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.217 10.193.207.184:7070/5065063 36 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:47.960267) v2 ==== 47+0+0 (2034709105 0 0) 0x1b6b8000 con 0x11df4000 -312> 2014-04-25 09:18:55.058816 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -311> 2014-04-25 09:18:55.058821 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.448 10.193.207.184:7067/5069058 30 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x13c66a80 con 0x1242c6e0 -310> 2014-04-25 09:18:55.058828 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3754 epoch_requested: 3754 LocalRecoveryReserved -309> 2014-04-25 09:18:55.058837 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.268 10.193.207.188:6865/4084837 37 ==== osd_ping(ping_reply e3800 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (3058409455 0 0) 0x1cf5f340 con 0x157769a0 -308> 2014-04-25 09:18:55.058851 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.268 has 3800 -307> 2014-04-25 09:18:55.058842 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 173.322969 138 0.006648 -306> 2014-04-25 09:18:55.058856 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.163 10.193.207.184:7073/6064447 34 ==== osd_ping(ping_reply e3800 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (3058409455 0 0) 0x142f2e00 con 0x1333a000 -305> 2014-04-25 09:18:55.058856 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[3.5d98( v 1163'719 (0'0,1163'719] local-les=3754 n=719 ec=633 les/c 3754/3716 3753/3753/3753) [210,193,171] r=0 lpr=3753 pi=3714-3752/7 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/WaitRemoteRecoveryReserved -304> 2014-04-25 09:18:55.058870 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.163 has 3800 -303> 2014-04-25 09:18:55.058871 7f77ee6e3700 10 log is not dirty -302> 2014-04-25 09:18:55.058876 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.214 10.193.207.185:6955/2012741 48 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x1806d500 con 0x133ab4a0 -301> 2014-04-25 09:18:55.058887 7f77ee6e3700 10 osd.210 pg_epoch: 3798 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_advance_map [210,80,105]/[210,80,105] -300> 2014-04-25 09:18:55.058898 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.63 10.193.207.177:6976/4034340 45 ==== osd_ping(ping_reply e3798 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (1803236683 0 0) 0x14f58c40 con 0x110906e0 -299> 2014-04-25 09:18:55.058904 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active advmap -298> 2014-04-25 09:18:55.058918 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.288 10.193.207.181:7002/4026708 44 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:47.960267) v2 ==== 47+0+0 (2034709105 0 0) 0x11aadc00 con 0x11a3eb00 -297> 2014-04-25 09:18:55.058918 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -296> 2014-04-25 09:18:55.058930 7f77f72f1700 10 osd.210 3800 note_peer_epoch osd.288 has 3799 -295> 2014-04-25 09:18:55.058929 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Started advmap -294> 2014-04-25 09:18:55.058936 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.316 10.193.207.177:6911/3037932 44 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x14228a80 con 0x1227cb00 -293> 2014-04-25 09:18:55.058940 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] check_recovery_sources no source osds () went down -292> 2014-04-25 09:18:55.058949 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.432 10.193.207.186:6934/4015949 43 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x15e37180 con 0x10e06c60 -291> 2014-04-25 09:18:55.058952 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_activate_map -290> 2014-04-25 09:18:55.058963 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: Active: handling ActMap -289> 2014-04-25 09:18:55.058967 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.135 10.193.207.177:6978/4035079 44 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x10faae00 con 0x12725760 -288> 2014-04-25 09:18:55.058975 7f77ee6e3700 10 osd.210 3799 queue_for_recovery queued pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] -287> 2014-04-25 09:18:55.058984 7f77ee6e3700 7 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] state: handle ActMap primary -286> 2014-04-25 09:18:55.058986 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.181 10.193.207.189:7015/2085746 46 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x1d0a6fc0 con 0x132c0000 -285> 2014-04-25 09:18:55.058997 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] take_waiters -284> 2014-04-25 09:18:55.059002 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.420 10.193.207.186:6979/3012241 39 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x11b068c0 con 0x104e6b00 -283> 2014-04-25 09:18:55.059008 7f77ee6e3700 10 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] handle_peering_event: epoch_sent: 3754 epoch_requested: 3754 LocalRecoveryReserved -282> 2014-04-25 09:18:55.059016 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.172 10.193.207.185:6802/2012373 44 ==== osd_ping(you_died e3800 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (4204542390 0 0) 0x1c45e540 con 0x123b54a0 -281> 2014-04-25 09:18:55.059021 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] exit Started/Primary/Active/WaitLocalRecoveryReserved 172.805408 138 0.161930 -280> 2014-04-25 09:18:55.059027 7f77f72f1700 10 osd.210 3800 handle_osd_ping osd.172 10.193.207.185:6802/2012373 says i am down in 3800 -279> 2014-04-25 09:18:55.059035 7f77ee6e3700 5 osd.210 pg_epoch: 3799 pg[4.5817( v 1217'1414 (0'0,1217'1414] local-les=3754 n=1414 ec=638 les/c 3754/3151 3753/3753/3753) [210,80,105] r=0 lpr=3753 pi=3144-3752/9 lcod 0'0 mlcod 0'0 active+recovery_wait] enter Started/Primary/Active/WaitRemoteRecoveryReserved -278> 2014-04-25 09:18:55.059039 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.172 10.193.207.185:6803/2012373 43 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x187fd880 con 0x133b18c0 -277> 2014-04-25 09:18:55.059050 7f77ee6e3700 1 -- 10.193.207.183:6889/6044082 --> 10.193.207.176:6963/6057494 -- MRecoveryReserve REQUEST pgid: 4.5817, query_epoch: 3799 v1 -- ?+0 0x1850bdc0 con 0x15d39340 -276> 2014-04-25 09:18:55.059058 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.217 10.193.207.184:7070/5065063 37 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x159a5340 con 0x11df4000 -275> 2014-04-25 09:18:55.059079 7f77ee6e3700 10 log is not dirty -274> 2014-04-25 09:18:55.059073 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.268 10.193.207.188:6865/4084837 38 ==== osd_ping(you_died e3800 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (4204542390 0 0) 0x1cf5e1c0 con 0x157769a0 -273> 2014-04-25 09:18:55.059084 7f77f72f1700 10 osd.210 3800 handle_osd_ping osd.268 10.193.207.188:6865/4084837 says i am down in 3800 -272> 2014-04-25 09:18:55.059093 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.163 10.193.207.184:7073/6064447 35 ==== osd_ping(you_died e3800 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (4204542390 0 0) 0x142f21c0 con 0x1333a000 -271> 2014-04-25 09:18:55.059103 7f77f72f1700 10 osd.210 3800 handle_osd_ping osd.163 10.193.207.184:7073/6064447 says i am down in 3800 -270> 2014-04-25 09:18:55.059119 7f77f72f1700 1 -- 10.193.207.183:0/44082 <== osd.288 10.193.207.181:7002/4026708 45 ==== osd_ping(ping_reply e3799 stamp 2014-04-25 09:18:50.669073) v2 ==== 47+0+0 (333206982 0 0) 0x11aad500 con 0x11a3eb00 -269> 2014-04-25 09:18:54.789769 7f77f54ee700 1 -- 10.193.207.183:6890/6044082 <== osd.438 10.193.207.180:0/93604 42 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:51.247930) v2 ==== 47+0+0 (4266706538 0 0) 0x12da1180 con 0x1105f600 -268> 2014-04-25 09:18:55.059146 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -267> 2014-04-25 09:18:55.059152 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -266> 2014-04-25 09:18:55.059161 7f77f54ee700 1 -- 10.193.207.183:6890/6044082 <== osd.102 10.193.207.180:0/69208 45 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:49.196073) v2 ==== 47+0+0 (3850918587 0 0) 0x127c01c0 con 0x11a3f8c0 -265> 2014-04-25 09:18:55.059174 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -264> 2014-04-25 09:18:55.059177 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -263> 2014-04-25 09:18:55.059182 7f77f54ee700 1 -- 10.193.207.183:6890/6044082 <== osd.259 10.193.207.180:0/74390 49 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:52.234415) v2 ==== 47+0+0 (3610206378 0 0) 0x114b0540 con 0x13310c60 -262> 2014-04-25 09:18:55.059193 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -261> 2014-04-25 09:18:55.059195 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -260> 2014-04-25 09:18:54.790094 7f77a427b700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -259> 2014-04-25 09:18:54.791225 7f778cf7e700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.178:0/52313 pipe(0x17a90000 sd=853 :6890 s=2 pgs=3192 cs=1 l=1 c=0x12b2f1e0).reader couldn't read tag, Success -258> 2014-04-25 09:18:55.059284 7f778cf7e700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.178:0/52313 pipe(0x17a90000 sd=853 :6890 s=2 pgs=3192 cs=1 l=1 c=0x12b2f1e0).fault 0: Success -257> 2014-04-25 09:18:54.791879 7f77d8304700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -256> 2014-04-25 09:18:54.791986 7f77c5c8d700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -255> 2014-04-25 09:18:54.792268 7f77f9af5700 1 accepter.accepter.rebind avoid 6889,6890,6919 -254> 2014-04-25 09:18:54.793941 7f77cb6ed700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -253> 2014-04-25 09:18:54.796611 7f77c69a0700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.177:0/34041 pipe(0x3db8780 sd=409 :6919 s=2 pgs=5827 cs=1 l=1 c=0x10dee420).reader couldn't read tag, Success -252> 2014-04-25 09:18:55.059630 7f77c69a0700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.177:0/34041 pipe(0x3db8780 sd=409 :6919 s=2 pgs=5827 cs=1 l=1 c=0x10dee420).fault 0: Success -251> 2014-04-25 09:18:54.796865 7f77c649b700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.177:0/35079 pipe(0x18092000 sd=411 :6919 s=2 pgs=7296 cs=1 l=1 c=0x1107b760).reader couldn't read tag, Success -250> 2014-04-25 09:18:55.059711 7f77c649b700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.177:0/35079 pipe(0x18092000 sd=411 :6919 s=2 pgs=7296 cs=1 l=1 c=0x1107b760).fault 0: Success -249> 2014-04-25 09:18:55.059811 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=74 :47454 s=2 pgs=7812 cs=1 l=0 c=0x15d39340).reader couldn't read tag, Success -248> 2014-04-25 09:18:55.059934 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=74 :47454 s=2 pgs=7812 cs=1 l=0 c=0x15d39340).fault 0: Success -247> 2014-04-25 09:18:55.059950 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=74 :47454 s=2 pgs=7812 cs=1 l=0 c=0x15d39340).fault, initiating reconnect -246> 2014-04-25 09:18:54.800254 7f77ca2d9700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -245> 2014-04-25 09:18:54.800314 7f77e91cd700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -244> 2014-04-25 09:18:54.800774 7f77b5a0f700 2 -- 10.193.207.183:0/44082 >> 10.193.207.176:6987/4058234 pipe(0x17fe6000 sd=501 :51494 s=4 pgs=1057 cs=1 l=1 c=0x1294eb00).reader couldn't read tag, Success -243> 2014-04-25 09:18:55.060149 7f77b5a0f700 2 -- 10.193.207.183:0/44082 >> 10.193.207.176:6987/4058234 pipe(0x17fe6000 sd=501 :51494 s=4 pgs=1057 cs=1 l=1 c=0x1294eb00).fault 0: Success -242> 2014-04-25 09:18:54.808317 7f77d8203700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.185:0/14361 pipe(0x11f82780 sd=127 :6919 s=2 pgs=1683 cs=1 l=1 c=0x12426b00).reader couldn't read tag, Success -241> 2014-04-25 09:18:55.060304 7f77d8203700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.185:0/14361 pipe(0x11f82780 sd=127 :6919 s=2 pgs=1683 cs=1 l=1 c=0x12426b00).fault 0: Success -240> 2014-04-25 09:18:54.809606 7f77d0137700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6921/4060151 pipe(0x14a94000 sd=162 :37021 s=4 pgs=9811 cs=1 l=0 c=0x1265c9a0).reader couldn't read tag, Success -239> 2014-04-25 09:18:55.060484 7f77d0137700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6921/4060151 pipe(0x14a94000 sd=162 :37021 s=4 pgs=9811 cs=1 l=0 c=0x1265c9a0).fault 0: Success -238> 2014-04-25 09:18:54.811771 7f77d3c72700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -237> 2014-04-25 09:18:54.812322 7f77c3566700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -236> 2014-04-25 09:18:54.812555 7f77b14ca700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6809/2090830 pipe(0x1195b680 sd=580 :6889 s=4 pgs=1637 cs=1 l=0 c=0x1265c580).reader couldn't read tag, Success -235> 2014-04-25 09:18:55.060623 7f77b14ca700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6809/2090830 pipe(0x1195b680 sd=580 :6889 s=4 pgs=1637 cs=1 l=0 c=0x1265c580).fault 0: Success -234> 2014-04-25 09:18:54.812626 7f77d8c0d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6835/2031623 pipe(0x11902280 sd=58 :6889 s=4 pgs=6959 cs=1 l=0 c=0x123b4000).reader couldn't read tag, Success -233> 2014-04-25 09:18:55.060718 7f77d8c0d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6835/2031623 pipe(0x11902280 sd=58 :6889 s=4 pgs=6959 cs=1 l=0 c=0x123b4000).fault 0: Success -232> 2014-04-25 09:18:54.813495 7f77d8e0f700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7004/5065542 pipe(0x119a0280 sd=231 :51694 s=4 pgs=5007 cs=1 l=0 c=0x104f3a20).reader couldn't read tag, Success -231> 2014-04-25 09:18:55.060749 7f77d8e0f700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7004/5065542 pipe(0x119a0280 sd=231 :51694 s=4 pgs=5007 cs=1 l=0 c=0x104f3a20).fault 0: Success -230> 2014-04-25 09:18:54.814677 7f77a9b51700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.176:0/56313 pipe(0x1d6ecf00 sd=646 :6919 s=2 pgs=10043 cs=1 l=1 c=0x11612840).reader couldn't read tag, Success -229> 2014-04-25 09:18:55.060785 7f77a9b51700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.176:0/56313 pipe(0x1d6ecf00 sd=646 :6919 s=2 pgs=10043 cs=1 l=1 c=0x11612840).fault 0: Success -228> 2014-04-25 09:18:54.822285 7f77e4277700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6835/2005481 pipe(0x1ba19400 sd=68 :40142 s=4 pgs=4306 cs=1 l=0 c=0x11a4a2c0).reader couldn't read tag, Success -227> 2014-04-25 09:18:55.060861 7f77e4277700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6835/2005481 pipe(0x1ba19400 sd=68 :40142 s=4 pgs=4306 cs=1 l=0 c=0x11a4a2c0).fault 0: Success -226> 2014-04-25 09:18:54.824182 7f77e386d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6989/7057278 pipe(0x12578a00 sd=70 :33688 s=4 pgs=10025 cs=1 l=0 c=0x1c65e840).reader couldn't read tag, Success -225> 2014-04-25 09:18:55.060955 7f77e386d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6989/7057278 pipe(0x12578a00 sd=70 :33688 s=4 pgs=10025 cs=1 l=0 c=0x1c65e840).fault 0: Success -224> 2014-04-25 09:18:54.826342 7f77d6aa0700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:6950/3026889 pipe(0x14997900 sd=277 :39744 s=4 pgs=1943 cs=1 l=0 c=0x132eba20).reader couldn't read tag, Success -223> 2014-04-25 09:18:55.060981 7f77d6aa0700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:6950/3026889 pipe(0x14997900 sd=277 :39744 s=4 pgs=1943 cs=1 l=0 c=0x132eba20).fault 0: Success -222> 2014-04-25 09:18:54.828001 7f77b580d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6805/3034113 pipe(0x15d4a280 sd=545 :6889 s=4 pgs=2936 cs=1 l=0 c=0x1329d760).reader couldn't read tag, Success -221> 2014-04-25 09:18:55.061005 7f77b580d700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6805/3034113 pipe(0x15d4a280 sd=545 :6889 s=4 pgs=2936 cs=1 l=0 c=0x1329d760).fault 0: Success -220> 2014-04-25 09:18:54.833690 7f77acf85700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.186:0/96227 pipe(0x1cda3b80 sd=597 :6890 s=2 pgs=2376 cs=1 l=1 c=0x1199dce0).reader couldn't read tag, Success -219> 2014-04-25 09:18:55.061032 7f77acf85700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.186:0/96227 pipe(0x1cda3b80 sd=597 :6890 s=2 pgs=2376 cs=1 l=1 c=0x1199dce0).fault 0: Success -218> 2014-04-25 09:18:54.834725 7f77c99d0700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/67309 pipe(0x1cb4bb80 sd=759 :6890 s=4 pgs=3082 cs=1 l=1 c=0x11a33ce0).reader couldn't read tag, Success -217> 2014-04-25 09:18:55.061092 7f77c99d0700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/67309 pipe(0x1cb4bb80 sd=759 :6890 s=4 pgs=3082 cs=1 l=1 c=0x11a33ce0).fault 0: Success -216> 2014-04-25 09:18:54.835601 7f77f68f0700 1 -- 10.193.207.183:6919/6044082 <== osd.438 10.193.207.180:0/93604 42 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:51.247930) v2 ==== 47+0+0 (4266706538 0 0) 0x1233cfc0 con 0x10dfb600 -215> 2014-04-25 09:18:55.061252 7f77f68f0700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -214> 2014-04-25 09:18:55.061257 7f77f68f0700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -213> 2014-04-25 09:18:55.061270 7f77f68f0700 1 -- 10.193.207.183:6919/6044082 <== osd.438 10.193.207.180:0/93604 43 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:54.832630) v2 ==== 47+0+0 (2740427422 0 0) 0x120108c0 con 0x10dfb600 -212> 2014-04-25 09:18:55.061285 7f77f68f0700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -211> 2014-04-25 09:18:55.061287 7f77f68f0700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -210> 2014-04-25 09:18:54.836424 7f77b86be700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.184:0/67309 pipe(0x119a0a00 sd=760 :6919 s=2 pgs=3080 cs=1 l=1 c=0x10dfb080).reader couldn't read tag, Success -209> 2014-04-25 09:18:55.061313 7f77b86be700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.184:0/67309 pipe(0x119a0a00 sd=760 :6919 s=2 pgs=3080 cs=1 l=1 c=0x10dfb080).fault 0: Success -208> 2014-04-25 09:18:54.836720 7f77cb7ee700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.185:0/15835 pipe(0x1d07a780 sd=367 :6890 s=2 pgs=1536 cs=1 l=1 c=0x132c6b00).reader couldn't read tag, Success -207> 2014-04-25 09:18:55.061397 7f77cb7ee700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.185:0/15835 pipe(0x1d07a780 sd=367 :6890 s=2 pgs=1536 cs=1 l=1 c=0x132c6b00).fault 0: Success -206> 2014-04-25 09:18:55.061500 7f77edce2700 10 osd.210 3800 do_recovery can start 5 (0/15 rops) -205> 2014-04-25 09:18:55.061507 7f77edce2700 10 osd.210 3800 do_recovery starting 5 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] -204> 2014-04-25 09:18:55.061522 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] recover_replicas(5) -203> 2014-04-25 09:18:55.061536 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] peer osd.221 missing 0 objects. -202> 2014-04-25 09:18:55.061547 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] peer osd.153 missing 65 objects. -201> 2014-04-25 09:18:54.839850 7f77cdd13700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/67050 pipe(0x1095ea00 sd=345 :6890 s=2 pgs=2816 cs=1 l=1 c=0x15d38dc0).reader couldn't read tag, Success -200> 2014-04-25 09:18:55.061559 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] recover_replicas: recover_object_replicas(7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3) -199> 2014-04-25 09:18:55.061574 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] prep_object_replica_pushes: on 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 -198> 2014-04-25 09:18:55.061578 7f77cdd13700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/67050 pipe(0x1095ea00 sd=345 :6890 s=2 pgs=2816 cs=1 l=1 c=0x15d38dc0).fault 0: Success -197> 2014-04-25 09:18:54.842776 7f77a8b41700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.185:0/11137 pipe(0x1d626c80 sd=652 :6890 s=2 pgs=1694 cs=1 l=1 c=0x13395ce0).reader couldn't read tag, Success -196> 2014-04-25 09:18:55.061648 7f77a8b41700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.185:0/11137 pipe(0x1d626c80 sd=652 :6890 s=2 pgs=1694 cs=1 l=1 c=0x13395ce0).fault 0: Success -195> 2014-04-25 09:18:54.844156 7f77b28de700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.186:0/8004 pipe(0x19ca8c80 sd=574 :6919 s=2 pgs=2437 cs=1 l=1 c=0x133b4000).reader couldn't read tag, Success -194> 2014-04-25 09:18:55.061784 7f77b28de700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.186:0/8004 pipe(0x19ca8c80 sd=574 :6919 s=2 pgs=2437 cs=1 l=1 c=0x133b4000).fault 0: Success -193> 2014-04-25 09:18:54.844333 7f7793a74700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/68646 pipe(0x17a90280 sd=813 :6890 s=2 pgs=2495 cs=1 l=1 c=0x12b686e0).reader couldn't read tag, Success -192> 2014-04-25 09:18:55.061945 7f7793a74700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/68646 pipe(0x17a90280 sd=813 :6890 s=2 pgs=2495 cs=1 l=1 c=0x12b686e0).fault 0: Success -191> 2014-04-25 09:18:55.061953 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] populate_obc_watchers 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 -190> 2014-04-25 09:18:55.061986 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] get_object_context 0x1c7d2f00 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 0 -> 1 read 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3(979'622 client.25998.0:1201815 wrlock_by=unknown.0.0:0) -189> 2014-04-25 09:18:55.062003 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_lock for 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 -188> 2014-04-25 09:18:55.062017 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 lcod 0'0 mlcod 0'0 active+recovering] start_recovery_op 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 -187> 2014-04-25 09:18:55.062029 7f77edce2700 10 osd.210 3800 start_recovery_op pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 (5/15 rops) -186> 2014-04-25 09:18:54.846401 7f77bc8f4700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.186:0/100290 pipe(0x13375680 sd=486 :6919 s=2 pgs=2494 cs=1 l=1 c=0x126e9a20).reader couldn't read tag, Success -185> 2014-04-25 09:18:55.062052 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] prep_push_to_replica7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 v979'622 size 2355200 to osd.153 -184> 2014-04-25 09:18:55.062093 7f77bc8f4700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.186:0/100290 pipe(0x13375680 sd=486 :6919 s=2 pgs=2494 cs=1 l=1 c=0x126e9a20).fault 0: Success -183> 2014-04-25 09:18:55.062247 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] get_snapset_context eb6ccdaaa67adb1b0ae4417d079fa03c 0 -> 1 -182> 2014-04-25 09:18:55.062299 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 clone_overlap {} -181> 2014-04-25 09:18:55.062314 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 data_subset [0~2355200] clone_subsets {} -180> 2014-04-25 09:18:55.062327 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] put_snapset_context eb6ccdaaa67adb1b0ae4417d079fa03c 1 -> 0 -179> 2014-04-25 09:18:55.062349 7f77edce2700 7 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] send_push_op 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 v 979'622 size 2355200 recovery_info: ObjectRecoveryInfo(7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3@979'622, copy_subset: [0~2355200], clone_subset: {}) -178> 2014-04-25 09:18:54.846522 7f77ded6e700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -177> 2014-04-25 09:18:54.846883 7f77c3f70700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/66751 pipe(0x3db8f00 sd=428 :6890 s=2 pgs=3126 cs=1 l=1 c=0x12a79e40).reader couldn't read tag, Success -176> 2014-04-25 09:18:55.063055 7f77c3f70700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/66751 pipe(0x3db8f00 sd=428 :6890 s=2 pgs=3126 cs=1 l=1 c=0x12a79e40).fault 0: Success -175> 2014-04-25 09:18:54.848564 7f77d083e700 2 -- 10.193.207.183:0/44082 >> 10.193.207.181:6973/4025702 pipe(0x1cd5d400 sd=126 :42818 s=4 pgs=1041 cs=1 l=1 c=0x12937080).reader couldn't read tag, Success -174> 2014-04-25 09:18:55.063132 7f77d083e700 2 -- 10.193.207.183:0/44082 >> 10.193.207.181:6973/4025702 pipe(0x1cd5d400 sd=126 :42818 s=4 pgs=1041 cs=1 l=1 c=0x12937080).fault 0: Success -173> 2014-04-25 09:18:54.849409 7f77afdb3700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.186:0/96227 pipe(0x11902a00 sd=447 :6919 s=2 pgs=2359 cs=1 l=1 c=0x11610c60).reader couldn't read tag, Success -172> 2014-04-25 09:18:55.063275 7f77afdb3700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.186:0/96227 pipe(0x11902a00 sd=447 :6919 s=2 pgs=2359 cs=1 l=1 c=0x11610c60).fault 0: Success -171> 2014-04-25 09:18:55.065865 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -170> 2014-04-25 09:18:55.070401 7f77c1546700 -1 *** Caught signal (Aborted) ** in thread 7f77c1546700 ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73) 1: /home/y/bin64/ceph-osd() [0x80db19] 2: /lib64/libpthread.so.0() [0x3a6e40f500] 3: (gsignal()+0x35) [0x3a6e0328a5] 4: (abort()+0x175) [0x3a6e034085] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x3a700bea5d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. -169> 2014-04-25 09:18:55.078118 7f77a0d46700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -168> 2014-04-25 09:18:55.078241 7f77d356b700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -167> 2014-04-25 09:18:55.079527 7f77e95d1700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -166> 2014-04-25 09:18:55.079963 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48083 s=2 pgs=7816 cs=3 l=0 c=0x15d39340).reader couldn't read tag, Success -165> 2014-04-25 09:18:55.080015 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48083 s=2 pgs=7816 cs=3 l=0 c=0x15d39340).fault 0: Success -164> 2014-04-25 09:18:55.080033 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48083 s=2 pgs=7816 cs=3 l=0 c=0x15d39340).fault, initiating reconnect -163> 2014-04-25 09:18:55.081437 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -162> 2014-04-25 09:18:55.082401 7f77a873d700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -161> 2014-04-25 09:18:54.851735 7f77c5081700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -160> 2014-04-25 09:18:54.854134 7f77db435700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.180:6944/3075847 pipe(0x11948780 sd=227 :36824 s=4 pgs=3109 cs=1 l=0 c=0x12b2f4a0).reader couldn't read tag, Success -159> 2014-04-25 09:18:55.083697 7f77db435700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.180:6944/3075847 pipe(0x11948780 sd=227 :36824 s=4 pgs=3109 cs=1 l=0 c=0x12b2f4a0).fault 0: Success -158> 2014-04-25 09:18:54.856616 7f77dc142700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.187:0/76543 pipe(0x14a95b80 sd=53 :6919 s=2 pgs=1571 cs=1 l=1 c=0x12427760).reader couldn't read tag, Success -157> 2014-04-25 09:18:55.083740 7f77dc142700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.187:0/76543 pipe(0x14a95b80 sd=53 :6919 s=2 pgs=1571 cs=1 l=1 c=0x12427760).fault 0: Success -156> 2014-04-25 09:18:54.857100 7f77b76ae700 1 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x19e43b80 sd=136 :6890 s=0 pgs=0 cs=0 l=0 c=0x1c7ad760).accept sd=136 10.193.207.177:52147/0 -155> 2014-04-25 09:18:55.083837 7f77b76ae700 2 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x19e43b80 sd=136 :6890 s=4 pgs=0 cs=0 l=0 c=0x1c7ad760).fault 0: Success -154> 2014-04-25 09:18:54.857172 7f77d1147700 1 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x19e43680 sd=145 :6890 s=0 pgs=0 cs=0 l=0 c=0x1335ba20).accept sd=145 10.193.207.176:55886/0 -153> 2014-04-25 09:18:54.857232 7f77d649a700 1 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x19e42c80 sd=153 :6890 s=0 pgs=0 cs=0 l=0 c=0x1335bb80).accept sd=153 10.193.207.184:38647/0 -152> 2014-04-25 09:18:55.084036 7f77d649a700 2 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x19e42c80 sd=153 :6890 s=4 pgs=0 cs=0 l=0 c=0x1335bb80).fault 0: Success -151> 2014-04-25 09:18:54.857288 7f77e2459700 1 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x130a7400 sd=155 :6890 s=0 pgs=0 cs=0 l=0 c=0x1199bce0).accept sd=155 10.193.207.189:39286/0 -150> 2014-04-25 09:18:54.857340 7f77e1a9b700 1 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x130a7b80 sd=161 :6890 s=0 pgs=0 cs=0 l=0 c=0x1335e580).accept sd=161 10.193.207.189:39349/0 -149> 2014-04-25 09:18:54.857397 7f77af6ac700 1 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x130a7900 sd=166 :6890 s=0 pgs=0 cs=0 l=0 c=0x12a31080).accept sd=166 10.193.207.188:32880/0 -148> 2014-04-25 09:18:55.084169 7f77af6ac700 2 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x130a7900 sd=166 :6890 s=4 pgs=0 cs=0 l=0 c=0x12a31080).fault 0: Success -147> 2014-04-25 09:18:54.857452 7f779062e700 1 -- 10.193.207.183:6890/6044082 >> :/0 pipe(0x17a91900 sd=174 :6890 s=0 pgs=0 cs=0 l=0 c=0x133942c0).accept sd=174 10.193.207.189:40717/0 -146> 2014-04-25 09:18:54.867350 7f77df071700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7037/6067941 pipe(0x10839180 sd=202 :6889 s=4 pgs=7916 cs=1 l=0 c=0x1107b600).reader couldn't read tag, Success -145> 2014-04-25 09:18:55.084296 7f77df071700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7037/6067941 pipe(0x10839180 sd=202 :6889 s=4 pgs=7916 cs=1 l=0 c=0x1107b600).fault 0: Success -144> 2014-04-25 09:18:54.867544 7f779417b700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.181:0/33204 pipe(0x143e8a00 sd=824 :6919 s=2 pgs=2932 cs=1 l=1 c=0x13288840).reader couldn't read tag, Success -143> 2014-04-25 09:18:55.084334 7f779417b700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.181:0/33204 pipe(0x143e8a00 sd=824 :6919 s=2 pgs=2932 cs=1 l=1 c=0x13288840).fault 0: Success -142> 2014-04-25 09:18:55.084516 7f77f54ee700 1 -- 10.193.207.183:6890/6044082 <== osd.205 10.193.207.189:0/57898 1 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:52.010621) v2 ==== 47+0+0 (896302648 0 0) 0x12e17340 con 0x133942c0 -141> 2014-04-25 09:18:55.084539 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -140> 2014-04-25 09:18:55.084543 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -139> 2014-04-25 09:18:55.084562 7f77f54ee700 1 -- 10.193.207.183:6890/6044082 <== osd.204 10.193.207.176:0/63784 1 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:51.427447) v2 ==== 47+0+0 (840090509 0 0) 0x12e17880 con 0x1335ba20 -138> 2014-04-25 09:18:55.084574 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -137> 2014-04-25 09:18:55.084576 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -136> 2014-04-25 09:18:54.882278 7f77e68a4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.185:7016/3011295 pipe(0x18014500 sd=65 :55022 s=4 pgs=2181 cs=1 l=0 c=0x127838c0).reader couldn't read tag, Success -135> 2014-04-25 09:18:55.084615 7f77e68a4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.185:7016/3011295 pipe(0x18014500 sd=65 :55022 s=4 pgs=2181 cs=1 l=0 c=0x127838c0).fault 0: Success -134> 2014-04-25 09:18:54.882289 7f77cef25700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.189:0/76441 pipe(0x1ba19b80 sd=334 :6890 s=2 pgs=7798 cs=1 l=1 c=0x132bbb80).reader couldn't read tag, Success -133> 2014-04-25 09:18:55.084657 7f77cef25700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.189:0/76441 pipe(0x1ba19b80 sd=334 :6890 s=2 pgs=7798 cs=1 l=1 c=0x132bbb80).fault 0: Success -132> 2014-04-25 09:18:54.882314 7f77a67a0700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.176:0/58234 pipe(0x1d41cf00 sd=676 :6890 s=2 pgs=8837 cs=1 l=1 c=0x10de4dc0).reader couldn't read tag, Success -131> 2014-04-25 09:18:55.084742 7f77a67a0700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.176:0/58234 pipe(0x1d41cf00 sd=676 :6890 s=2 pgs=8837 cs=1 l=1 c=0x10de4dc0).fault 0: Success -130> 2014-04-25 09:18:55.084829 7f77f54ee700 1 -- 10.193.207.183:6890/6044082 <== osd.203 10.193.207.189:0/57507 1 ==== osd_ping(ping e3799 stamp 2014-04-25 09:18:49.523756) v2 ==== 47+0+0 (2164970946 0 0) 0x11882540 con 0x1335e580 -129> 2014-04-25 09:18:55.084844 7f77f54ee700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f77f04e6700' had timed out after 4 -128> 2014-04-25 09:18:55.084846 7f77f54ee700 10 osd.210 3800 internal heartbeat not healthy, dropping ping request -127> 2014-04-25 09:18:54.883907 7f77cbd6b700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6810/1068939 pipe(0x14a94a00 sd=363 :6889 s=4 pgs=1370 cs=1 l=0 c=0x123c4f20).reader couldn't read tag, Success -126> 2014-04-25 09:18:55.084861 7f77cbd6b700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6810/1068939 pipe(0x14a94a00 sd=363 :6889 s=4 pgs=1370 cs=1 l=0 c=0x123c4f20).fault 0: Success -125> 2014-04-25 09:18:54.884747 7f77abe74700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.176:0/59908 pipe(0x17cb4780 sd=624 :6890 s=2 pgs=12818 cs=1 l=1 c=0x1339a580).reader couldn't read tag, Success -124> 2014-04-25 09:18:55.084902 7f77abe74700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.176:0/59908 pipe(0x17cb4780 sd=624 :6890 s=2 pgs=12818 cs=1 l=1 c=0x1339a580).fault 0: Success -123> 2014-04-25 09:18:54.897185 7f77b0fc5700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6805/2068421 pipe(0x1195a500 sd=586 :6889 s=4 pgs=1637 cs=1 l=0 c=0x120e2420).reader couldn't read tag, Success -122> 2014-04-25 09:18:55.084978 7f77b0fc5700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6805/2068421 pipe(0x1195a500 sd=586 :6889 s=4 pgs=1637 cs=1 l=0 c=0x120e2420).fault 0: Success -121> 2014-04-25 09:18:54.899969 7f77c87be700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:6878/5026291 pipe(0x1822a780 sd=388 :36384 s=4 pgs=3549 cs=1 l=0 c=0x1c65fa20).reader couldn't read tag, Success -120> 2014-04-25 09:18:55.085064 7f77c87be700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:6878/5026291 pipe(0x1822a780 sd=388 :36384 s=4 pgs=3549 cs=1 l=0 c=0x1c65fa20).fault 0: Success -119> 2014-04-25 09:18:54.900911 7f77975af700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7048/5067309 pipe(0x19aaa500 sd=150 :6889 s=4 pgs=13083 cs=1 l=0 c=0x13a84580).reader couldn't read tag, Success -118> 2014-04-25 09:18:55.085088 7f77975af700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7048/5067309 pipe(0x19aaa500 sd=150 :6889 s=4 pgs=13083 cs=1 l=0 c=0x13a84580).fault 0: Success -117> 2014-04-25 09:18:54.902608 7f7792757700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.184:0/63523 pipe(0x126cc000 sd=149 :6919 s=2 pgs=4057 cs=1 l=1 c=0x127278c0).reader couldn't read tag, Success -116> 2014-04-25 09:18:55.085121 7f7792757700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.184:0/63523 pipe(0x126cc000 sd=149 :6919 s=2 pgs=4057 cs=1 l=1 c=0x127278c0).fault 0: Success -115> 2014-04-25 09:18:54.904666 7f77e1596700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6863/4084837 pipe(0x11ecd180 sd=599 :6889 s=4 pgs=2143 cs=1 l=0 c=0x13273340).reader couldn't read tag, Success -114> 2014-04-25 09:18:55.085214 7f77e1596700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6863/4084837 pipe(0x11ecd180 sd=599 :6889 s=4 pgs=2143 cs=1 l=0 c=0x13273340).fault 0: Success -113> 2014-04-25 09:18:54.907477 7f77aa359700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.179:0/97667 pipe(0x1d627400 sd=645 :6890 s=2 pgs=3893 cs=1 l=1 c=0x13393340).reader couldn't read tag, Success -112> 2014-04-25 09:18:55.085316 7f77aa359700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.179:0/97667 pipe(0x1d627400 sd=645 :6890 s=2 pgs=3893 cs=1 l=1 c=0x13393340).fault 0: Success -111> 2014-04-25 09:18:54.909487 7f77c82b9700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.179:0/99876 pipe(0x14278280 sd=393 :6890 s=2 pgs=2281 cs=1 l=1 c=0x10dee580).reader couldn't read tag, Success -110> 2014-04-25 09:18:55.085399 7f77c82b9700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.179:0/99876 pipe(0x14278280 sd=393 :6890 s=2 pgs=2281 cs=1 l=1 c=0x10dee580).fault 0: Success -109> 2014-04-25 09:18:54.909836 7f77cc270700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/64160 pipe(0x1d07b680 sd=361 :6890 s=2 pgs=2771 cs=1 l=1 c=0x120b0f20).reader couldn't read tag, Success -108> 2014-04-25 09:18:55.085585 7f77cc270700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.184:0/64160 pipe(0x1d07b680 sd=361 :6890 s=2 pgs=2771 cs=1 l=1 c=0x120b0f20).fault 0: Success -107> 2014-04-25 09:18:54.911111 7f77e5186700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6964/5058379 pipe(0x11949680 sd=264 :59620 s=4 pgs=5959 cs=2 l=0 c=0x12a23b80).reader couldn't read tag, Success -106> 2014-04-25 09:18:55.085705 7f77e5186700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6964/5058379 pipe(0x11949680 sd=264 :59620 s=4 pgs=5959 cs=2 l=0 c=0x12a23b80).fault 0: Success -105> 2014-04-25 09:18:54.918074 7f77d8b0c700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:6982/4067050 pipe(0x12dcf400 sd=258 :52237 s=4 pgs=5463 cs=1 l=0 c=0x11613080).reader couldn't read tag, Success -104> 2014-04-25 09:18:55.085845 7f77d8b0c700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:6982/4067050 pipe(0x12dcf400 sd=258 :52237 s=4 pgs=5463 cs=1 l=0 c=0x11613080).fault 0: Success -103> 2014-04-25 09:18:54.919400 7f77b13c9700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.185:0/15071 pipe(0x1cd5c000 sd=581 :6890 s=2 pgs=2285 cs=1 l=1 c=0x13310f20).reader couldn't read tag, Success -102> 2014-04-25 09:18:55.085975 7f77b13c9700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.185:0/15071 pipe(0x1cd5c000 sd=581 :6890 s=2 pgs=2285 cs=1 l=1 c=0x13310f20).fault 0: Success -101> 2014-04-25 09:18:54.919422 7f77c7ab1700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.188:0/70327 pipe(0x18092c80 sd=397 :6890 s=2 pgs=1437 cs=1 l=1 c=0x11a3fe40).reader couldn't read tag, Success -100> 2014-04-25 09:18:55.086187 7f77c7ab1700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.188:0/70327 pipe(0x18092c80 sd=397 :6890 s=2 pgs=1437 cs=1 l=1 c=0x11a3fe40).fault 0: Success -99> 2014-04-25 09:18:54.922363 7f77cace3700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.185:6829/2010972 pipe(0x19aaa000 sd=5 :6889 s=4 pgs=2011 cs=1 l=0 c=0x117a1600).reader couldn't read tag, Success -98> 2014-04-25 09:18:55.086236 7f77cace3700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.185:6829/2010972 pipe(0x19aaa000 sd=5 :6889 s=4 pgs=2011 cs=1 l=0 c=0x117a1600).fault 0: Success -97> 2014-04-25 09:18:54.922588 7f77a558e700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6809/2075929 pipe(0x12a56500 sd=37 :6889 s=4 pgs=5259 cs=1 l=0 c=0x1277cdc0).reader couldn't read tag, Success -96> 2014-04-25 09:18:55.086334 7f77a558e700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:6809/2075929 pipe(0x12a56500 sd=37 :6889 s=4 pgs=5259 cs=1 l=0 c=0x1277cdc0).fault 0: Success -95> 2014-04-25 09:18:54.923803 7f77d0b41700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.178:0/48932 pipe(0x11eccc80 sd=248 :6919 s=2 pgs=2452 cs=1 l=1 c=0x124278c0).reader couldn't read tag, Success -94> 2014-04-25 09:18:55.086365 7f77d0b41700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.178:0/48932 pipe(0x11eccc80 sd=248 :6919 s=2 pgs=2452 cs=1 l=1 c=0x124278c0).fault 0: Success -93> 2014-04-25 09:18:54.925527 7f7798bc5700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6813/5096227 pipe(0x19aaaa00 sd=464 :6889 s=4 pgs=3133 cs=1 l=0 c=0x110a6f20).reader couldn't read tag, Success -92> 2014-04-25 09:18:55.086436 7f7798bc5700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6813/5096227 pipe(0x19aaaa00 sd=464 :6889 s=4 pgs=3133 cs=1 l=0 c=0x110a6f20).fault 0: Success -91> 2014-04-25 09:18:54.926550 7f77d78ae700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6909/2042527 pipe(0x11ef7680 sd=246 :49995 s=4 pgs=4647 cs=1 l=0 c=0x186ae6e0).reader couldn't read tag, Success -90> 2014-04-25 09:18:55.086490 7f77d78ae700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6909/2042527 pipe(0x11ef7680 sd=246 :49995 s=4 pgs=4647 cs=1 l=0 c=0x186ae6e0).fault 0: Success -89> 2014-04-25 09:18:54.926980 7f77b45fb700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.188:0/79282 pipe(0x19aab900 sd=682 :6919 s=2 pgs=1201 cs=1 l=1 c=0x127efce0).reader couldn't read tag, Success -88> 2014-04-25 09:18:55.086529 7f77b45fb700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.188:0/79282 pipe(0x19aab900 sd=682 :6919 s=2 pgs=1201 cs=1 l=1 c=0x127efce0).fault 0: Success -87> 2014-04-25 09:18:54.928030 7f778ab33700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.187:0/75929 pipe(0x17a90a00 sd=842 :6890 s=2 pgs=1405 cs=1 l=1 c=0x12290000).reader couldn't read tag, Success -86> 2014-04-25 09:18:55.086642 7f778ab33700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.187:0/75929 pipe(0x17a90a00 sd=842 :6890 s=2 pgs=1405 cs=1 l=1 c=0x12290000).fault 0: Success -85> 2014-04-25 09:18:54.930247 7f77da627700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6821/4034041 pipe(0x11eccf00 sd=229 :42371 s=4 pgs=6786 cs=1 l=0 c=0x1241e000).reader couldn't read tag, Success -84> 2014-04-25 09:18:55.086755 7f77da627700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6821/4034041 pipe(0x11eccf00 sd=229 :42371 s=4 pgs=6786 cs=1 l=0 c=0x1241e000).fault 0: Success -83> 2014-04-25 09:18:54.930523 7f77d5d93700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6845/5042140 pipe(0x11e18f00 sd=286 :53118 s=4 pgs=4531 cs=1 l=0 c=0x10e15e40).reader couldn't read tag, Success -82> 2014-04-25 09:18:55.086780 7f77d5d93700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.177:6845/5042140 pipe(0x11e18f00 sd=286 :53118 s=4 pgs=4531 cs=1 l=0 c=0x10e15e40).fault 0: Success -81> 2014-04-25 09:18:54.931004 7f77d4e84700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6905/6098764 pipe(0x5fb6280 sd=296 :52293 s=4 pgs=2680 cs=1 l=0 c=0x10e27a20).reader couldn't read tag, Success -80> 2014-04-25 09:18:55.086804 7f77d4e84700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6905/6098764 pipe(0x5fb6280 sd=296 :52293 s=4 pgs=2680 cs=1 l=0 c=0x10e27a20).fault 0: Success -79> 2014-04-25 09:18:54.931095 7f77ce61c700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.179:0/97831 pipe(0x1095f680 sd=341 :6890 s=2 pgs=3355 cs=1 l=1 c=0x132bb080).reader couldn't read tag, Success -78> 2014-04-25 09:18:55.086837 7f77ce61c700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.179:0/97831 pipe(0x1095f680 sd=341 :6890 s=2 pgs=3355 cs=1 l=1 c=0x132bb080).fault 0: Success -77> 2014-04-25 09:18:54.931905 7f77bc1ed700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.187:0/78947 pipe(0x13375b80 sd=488 :6919 s=2 pgs=1281 cs=1 l=1 c=0x126e9760).reader couldn't read tag, Success -76> 2014-04-25 09:18:55.086920 7f77bc1ed700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.187:0/78947 pipe(0x13375b80 sd=488 :6919 s=2 pgs=1281 cs=1 l=1 c=0x126e9760).fault 0: Success -75> 2014-04-25 09:18:54.932867 7f77aa45a700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.176:0/58234 pipe(0x1d6ed180 sd=644 :6919 s=2 pgs=8835 cs=1 l=1 c=0x11612f20).reader couldn't read tag, Success -74> 2014-04-25 09:18:55.087093 7f77aa45a700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.176:0/58234 pipe(0x1d6ed180 sd=644 :6919 s=2 pgs=8835 cs=1 l=1 c=0x11612f20).fault 0: Success -73> 2014-04-25 09:18:55.087148 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48146 s=2 pgs=7817 cs=5 l=0 c=0x15d39340).reader couldn't read tag, Success -72> 2014-04-25 09:18:55.087195 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48146 s=2 pgs=7817 cs=5 l=0 c=0x15d39340).fault 0: Success -71> 2014-04-25 09:18:54.933636 7f77c4475700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:6814/3040676 pipe(0x1822b180 sd=390 :49729 s=4 pgs=3463 cs=1 l=0 c=0x120d11e0).reader couldn't read tag, Success -70> 2014-04-25 09:18:55.087211 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48146 s=2 pgs=7817 cs=5 l=0 c=0x15d39340).fault, initiating reconnect -69> 2014-04-25 09:18:55.087219 7f77c4475700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:6814/3040676 pipe(0x1822b180 sd=390 :49729 s=4 pgs=3463 cs=1 l=0 c=0x120d11e0).fault 0: Success -68> 2014-04-25 09:18:54.936126 7f77dc94a700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.178:0/48932 pipe(0x12a4b680 sd=237 :6890 s=2 pgs=2451 cs=1 l=1 c=0x133afe40).reader couldn't read tag, Success -67> 2014-04-25 09:18:55.087309 7f77dc94a700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.178:0/48932 pipe(0x12a4b680 sd=237 :6890 s=2 pgs=2451 cs=1 l=1 c=0x133afe40).fault 0: Success -66> 2014-04-25 09:18:54.936524 7f77c3667700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.184:0/68162 pipe(0x17ad8c80 sd=431 :6919 s=2 pgs=4055 cs=1 l=1 c=0x12a31b80).reader couldn't read tag, Success -65> 2014-04-25 09:18:55.087376 7f77c3667700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.184:0/68162 pipe(0x17ad8c80 sd=431 :6919 s=2 pgs=4055 cs=1 l=1 c=0x12a31b80).fault 0: Success -64> 2014-04-25 09:18:54.936754 7f77991cb700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6897/3098032 pipe(0x19aab400 sd=782 :6889 s=4 pgs=6465 cs=1 l=0 c=0x110a6840).reader couldn't read tag, Success -63> 2014-04-25 09:18:55.087522 7f77991cb700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:6897/3098032 pipe(0x19aab400 sd=782 :6889 s=4 pgs=6465 cs=1 l=0 c=0x110a6840).fault 0: Success -62> 2014-04-25 09:18:54.937628 7f77cf42a700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.178:6881/4044612 pipe(0x11ef7180 sd=259 :48784 s=4 pgs=3429 cs=1 l=0 c=0x1100ac60).reader couldn't read tag, Success -61> 2014-04-25 09:18:55.087593 7f77cf42a700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.178:6881/4044612 pipe(0x11ef7180 sd=259 :48784 s=4 pgs=3429 cs=1 l=0 c=0x1100ac60).fault 0: Success -60> 2014-04-25 09:18:54.939477 7f779a0da700 2 -- 10.193.207.183:0/44082 >> 10.193.207.185:6932/3015071 pipe(0x1cb33680 sd=738 :36020 s=4 pgs=817 cs=1 l=1 c=0x1204af20).reader couldn't read tag, Success -59> 2014-04-25 09:18:55.087620 7f779a0da700 2 -- 10.193.207.183:0/44082 >> 10.193.207.185:6932/3015071 pipe(0x1cb33680 sd=738 :36020 s=4 pgs=817 cs=1 l=1 c=0x1204af20).fault 0: Success -58> 2014-04-25 09:18:54.941713 7f77d194f700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:6922/4068415 pipe(0x18015400 sd=169 :38715 s=4 pgs=5784 cs=1 l=0 c=0x1333ba20).reader couldn't read tag, Success -57> 2014-04-25 09:18:55.087668 7f77d194f700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:6922/4068415 pipe(0x18015400 sd=169 :38715 s=4 pgs=5784 cs=1 l=0 c=0x1333ba20).fault 0: Success -56> 2014-04-25 09:18:54.942283 7f77e4378700 2 -- 10.193.207.183:0/44082 >> 10.193.207.179:6986/5098821 pipe(0x11943900 sd=121 :33114 s=4 pgs=1125 cs=1 l=1 c=0x114e5b80).reader couldn't read tag, Success -55> 2014-04-25 09:18:55.087695 7f77e4378700 2 -- 10.193.207.183:0/44082 >> 10.193.207.179:6986/5098821 pipe(0x11943900 sd=121 :33114 s=4 pgs=1125 cs=1 l=1 c=0x114e5b80).fault 0: Success -54> 2014-04-25 09:18:54.943317 7f77e477c700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6917/4060413 pipe(0x14a95900 sd=291 :40457 s=4 pgs=8332 cs=1 l=0 c=0x10e262c0).reader couldn't read tag, Success -53> 2014-04-25 09:18:55.087748 7f77e477c700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6917/4060413 pipe(0x14a95900 sd=291 :40457 s=4 pgs=8332 cs=1 l=0 c=0x10e262c0).fault 0: Success -52> 2014-04-25 09:18:54.943379 7f77d8607700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:7002/6004866 pipe(0x126bf680 sd=239 :42188 s=4 pgs=2749 cs=1 l=0 c=0x129dadc0).reader couldn't read tag, Success -51> 2014-04-25 09:18:55.087773 7f77d8607700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:7002/6004866 pipe(0x126bf680 sd=239 :42188 s=4 pgs=2749 cs=1 l=0 c=0x129dadc0).fault 0: Success -50> 2014-04-25 09:18:54.944762 7f77dcd4e700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:6971/4025702 pipe(0x11ef7b80 sd=204 :56596 s=4 pgs=2846 cs=1 l=0 c=0x132ebb80).reader couldn't read tag, Success -49> 2014-04-25 09:18:55.087795 7f77dcd4e700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.181:6971/4025702 pipe(0x11ef7b80 sd=204 :56596 s=4 pgs=2846 cs=1 l=0 c=0x132ebb80).fault 0: Success -48> 2014-04-25 09:18:54.945328 7f77cf127700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.179:6959/3097173 pipe(0x1822b900 sd=21 :37626 s=4 pgs=2480 cs=8 l=0 c=0x12ad74a0).connect error 10.193.207.179:6959/3097173, 103: Software caused connection abort -47> 2014-04-25 09:18:55.087818 7f77cf127700 3 -- 10.193.207.183:6889/6044082 >> 10.193.207.179:6959/3097173 pipe(0x1822b900 sd=21 :37626 s=4 pgs=2480 cs=8 l=0 c=0x12ad74a0).connect fault, but state = closed != connecting, stopping -46> 2014-04-25 09:18:54.947031 7f77e89c5700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x1ba18280 sd=83 :60907 s=4 pgs=7259 cs=1 l=0 c=0x11a35ce0).reader couldn't read tag, Success -45> 2014-04-25 09:18:55.087842 7f77e89c5700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x1ba18280 sd=83 :60907 s=4 pgs=7259 cs=1 l=0 c=0x11a35ce0).fault 0: Success -44> 2014-04-25 09:18:54.954362 7f77c4c7d700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.189:0/85746 pipe(0x17ad8280 sd=421 :6890 s=2 pgs=7149 cs=1 l=1 c=0x12a30580).reader couldn't read tag, Success -43> 2014-04-25 09:18:55.087878 7f77c4c7d700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.189:0/85746 pipe(0x17ad8280 sd=421 :6890 s=2 pgs=7149 cs=1 l=1 c=0x12a30580).fault 0: Success -42> 2014-04-25 09:18:54.954662 7f77cc4fb700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.184:0/64160 pipe(0x14278c80 sd=360 :6919 s=2 pgs=2772 cs=1 l=1 c=0x120b1760).reader couldn't read tag, Success -41> 2014-04-25 09:18:55.087955 7f77cc4fb700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.184:0/64160 pipe(0x14278c80 sd=360 :6919 s=2 pgs=2772 cs=1 l=1 c=0x120b1760).fault 0: Success -40> 2014-04-25 09:18:54.954684 7f77ced23700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.187:0/76543 pipe(0x10fad900 sd=337 :6890 s=2 pgs=1570 cs=1 l=1 c=0x132ba840).reader couldn't read tag, Success -39> 2014-04-25 09:18:55.088082 7f77ced23700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.187:0/76543 pipe(0x10fad900 sd=337 :6890 s=2 pgs=1570 cs=1 l=1 c=0x132ba840).fault 0: Success -38> 2014-04-25 09:18:54.958014 7f77a71aa700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.177:0/36338 pipe(0x1d41db80 sd=666 :6890 s=2 pgs=4649 cs=1 l=1 c=0x12849080).reader couldn't read tag, Success -37> 2014-04-25 09:18:55.088154 7f77a71aa700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.177:0/36338 pipe(0x1d41db80 sd=666 :6890 s=2 pgs=4649 cs=1 l=1 c=0x12849080).fault 0: Success -36> 2014-04-25 09:18:54.964366 7f77dd152700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:6975/5068162 pipe(0x14a95400 sd=172 :60288 s=4 pgs=9879 cs=1 l=0 c=0x1227adc0).reader couldn't read tag, Success -35> 2014-04-25 09:18:55.088261 7f77dd152700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:6975/5068162 pipe(0x14a95400 sd=172 :60288 s=4 pgs=9879 cs=1 l=0 c=0x1227adc0).fault 0: Success -34> 2014-04-25 09:18:54.969541 7f77de061700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.180:6898/5069208 pipe(0x10fad180 sd=187 :34551 s=4 pgs=3125 cs=1 l=0 c=0x111d71e0).reader couldn't read tag, Success -33> 2014-04-25 09:18:55.088336 7f77de061700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.180:6898/5069208 pipe(0x10fad180 sd=187 :34551 s=4 pgs=3125 cs=1 l=0 c=0x111d71e0).fault 0: Success -32> 2014-04-25 09:18:54.970883 7f77d0a40700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:7008/4097156 pipe(0x14996f00 sd=302 :45360 s=4 pgs=3550 cs=1 l=0 c=0x12c174a0).reader couldn't read tag, Success -31> 2014-04-25 09:18:55.088362 7f77d0a40700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.186:7008/4097156 pipe(0x14996f00 sd=302 :45360 s=4 pgs=3550 cs=1 l=0 c=0x12c174a0).fault 0: Success -30> 2014-04-25 09:18:54.970939 7f77e9eda700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.185:6887/3019345 pipe(0x130a6000 sd=82 :34564 s=4 pgs=5591 cs=1 l=0 c=0x13279600).reader couldn't read tag, Success -29> 2014-04-25 09:18:55.088384 7f77e9eda700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.185:6887/3019345 pipe(0x130a6000 sd=82 :34564 s=4 pgs=5591 cs=1 l=0 c=0x13279600).fault 0: Success -28> 2014-04-25 09:18:54.972160 7f77ad389700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.188:0/59228 pipe(0x17cb4a00 sd=621 :6890 s=2 pgs=2219 cs=1 l=1 c=0x1339b4a0).reader couldn't read tag, Success -27> 2014-04-25 09:18:55.088417 7f77ad389700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.188:0/59228 pipe(0x17cb4a00 sd=621 :6890 s=2 pgs=2219 cs=1 l=1 c=0x1339b4a0).fault 0: Success -26> 2014-04-25 09:18:54.974229 7f778b23a700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6875/2063770 pipe(0x11f82c80 sd=859 :6889 s=4 pgs=1695 cs=1 l=0 c=0x1277cf20).reader couldn't read tag, Success -25> 2014-04-25 09:18:55.088492 7f778b23a700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6875/2063770 pipe(0x11f82c80 sd=859 :6889 s=4 pgs=1695 cs=1 l=0 c=0x1277cf20).fault 0: Success -24> 2014-04-25 09:18:54.974596 7f77e20a1700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:7022/1076329 pipe(0x18014c80 sd=138 :48117 s=4 pgs=5586 cs=1 l=0 c=0x1241f600).reader couldn't read tag, Success -23> 2014-04-25 09:18:55.088567 7f77e20a1700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.187:7022/1076329 pipe(0x18014c80 sd=138 :48117 s=4 pgs=5586 cs=1 l=0 c=0x1241f600).fault 0: Success -22> 2014-04-25 09:18:54.978259 7f77af9af700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6822/2098206 pipe(0x1195aa00 sd=595 :6889 s=4 pgs=1589 cs=1 l=0 c=0x12ab89a0).reader couldn't read tag, Success -21> 2014-04-25 09:18:55.088590 7f77af9af700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.188:6822/2098206 pipe(0x1195aa00 sd=595 :6889 s=4 pgs=1589 cs=1 l=0 c=0x12ab89a0).fault 0: Success -20> 2014-04-25 09:18:54.985434 7f77dfd7e700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7066/5069058 pipe(0x10839400 sd=607 :6889 s=4 pgs=5079 cs=1 l=0 c=0x1227ac60).reader couldn't read tag, Success -19> 2014-04-25 09:18:55.088612 7f77dfd7e700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7066/5069058 pipe(0x10839400 sd=607 :6889 s=4 pgs=5079 cs=1 l=0 c=0x1227ac60).fault 0: Success -18> 2014-04-25 09:18:54.985873 7f77a74ad700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.176:0/57278 pipe(0x1d626000 sd=665 :6890 s=2 pgs=12137 cs=1 l=1 c=0x12849340).reader couldn't read tag, Success -17> 2014-04-25 09:18:55.088644 7f77a74ad700 2 -- 10.193.207.183:6890/6044082 >> 10.193.207.176:0/57278 pipe(0x1d626000 sd=665 :6890 s=2 pgs=12137 cs=1 l=1 c=0x12849340).fault 0: Success -16> 2014-04-25 09:18:54.993934 7f77b3cf2700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7056/7065384 pipe(0x19ca9900 sd=559 :6889 s=4 pgs=9580 cs=1 l=0 c=0x1335a000).reader couldn't read tag, Success -15> 2014-04-25 09:18:55.088703 7f77b3cf2700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.184:7056/7065384 pipe(0x19ca9900 sd=559 :6889 s=4 pgs=9580 cs=1 l=0 c=0x1335a000).fault 0: Success -14> 2014-04-25 09:18:54.994485 7f77c1f50700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.176:0/56013 pipe(0x17556280 sd=445 :6919 s=2 pgs=11524 cs=1 l=1 c=0x105042c0).reader couldn't read tag, Success -13> 2014-04-25 09:18:55.088827 7f77c1f50700 2 -- 10.193.207.183:6919/6044082 >> 10.193.207.176:0/56013 pipe(0x17556280 sd=445 :6919 s=2 pgs=11524 cs=1 l=1 c=0x105042c0).fault 0: Success -12> 2014-04-25 09:18:54.994742 7f77cfe34700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6944/4058513 pipe(0x18015b80 sd=57 :41513 s=4 pgs=11250 cs=1 l=0 c=0x1265db80).reader couldn't read tag, Success -11> 2014-04-25 09:18:55.088887 7f77cfe34700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6944/4058513 pipe(0x18015b80 sd=57 :41513 s=4 pgs=11250 cs=1 l=0 c=0x1265db80).fault 0: Success -10> 2014-04-25 09:18:55.003319 7f77ad288700 2 -- 10.193.207.183:0/44082 >> 10.193.207.180:6945/3075847 pipe(0x170baf00 sd=619 :44869 s=4 pgs=1042 cs=1 l=1 c=0x41dbce0).reader couldn't read tag, Success -9> 2014-04-25 09:18:55.088959 7f77ad288700 2 -- 10.193.207.183:0/44082 >> 10.193.207.180:6945/3075847 pipe(0x170baf00 sd=619 :44869 s=4 pgs=1042 cs=1 l=1 c=0x41dbce0).fault 0: Success -8> 2014-04-25 09:18:55.003424 7f77ac379700 2 -- 10.193.207.183:0/44082 >> 10.193.207.180:6946/3075847 pipe(0x170baa00 sd=622 :56685 s=4 pgs=1032 cs=1 l=1 c=0x1100b600).reader couldn't read tag, Success -7> 2014-04-25 09:18:55.089009 7f77ac379700 2 -- 10.193.207.183:0/44082 >> 10.193.207.180:6946/3075847 pipe(0x170baa00 sd=622 :56685 s=4 pgs=1032 cs=1 l=1 c=0x1100b600).fault 0: Success -6> 2014-04-25 09:18:55.089175 7f77d890a700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6964/5058379 pipe(0x11949680 sd=136 :59620 s=4 pgs=5959 cs=2 l=0 c=0x12a23b80).connect error 10.193.207.176:6964/5058379, 111: Connection refused -5> 2014-04-25 09:18:55.089195 7f77d890a700 3 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6964/5058379 pipe(0x11949680 sd=136 :59620 s=4 pgs=5959 cs=2 l=0 c=0x12a23b80).connect fault, but state = closed != connecting, stopping -4> 2014-04-25 09:18:55.090072 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -3> 2014-04-25 09:18:55.094157 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48189 s=2 pgs=7818 cs=7 l=0 c=0x15d39340).reader couldn't read tag, Success -2> 2014-04-25 09:18:55.094420 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48189 s=2 pgs=7818 cs=7 l=0 c=0x15d39340).fault 0: Success -1> 2014-04-25 09:18:55.094441 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48189 s=2 pgs=7818 cs=7 l=0 c=0x15d39340).fault, initiating reconnect 0> 2014-04-25 09:18:55.095041 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 10/10 osd 0/ 5 optracker 0/ 5 objclass 3/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 0/ 0 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 3/ 3 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /home/y/logs/ceph/ceph-osd.210.log --- end dump of recent events --- 2014-04-25 09:18:55.095846 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_unlock on 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 2014-04-25 09:18:55.095925 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] put_object_context 0x1c7d2f00 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 1 -> 0 2014-04-25 09:18:55.095978 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] recover_replicas: recover_object_replicas(1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3) 2014-04-25 09:18:55.095996 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] prep_object_replica_pushes: on 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 2014-04-25 09:18:55.097701 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] populate_obc_watchers 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 2014-04-25 09:18:55.097721 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] get_object_context 0x1c7d2f00 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 0 -> 1 read 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3(979'623 client.40137.0:1207807 wrlock_by=unknown.0.0:0) 2014-04-25 09:18:55.097739 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_lock for 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 2014-04-25 09:18:55.097754 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] start_recovery_op 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 2014-04-25 09:18:55.097766 7f77edce2700 10 osd.210 3800 start_recovery_op pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 (6/15 rops) 2014-04-25 09:18:55.097841 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] prep_push_to_replica1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 v979'623 size 2355200 to osd.153 2014-04-25 09:18:55.098036 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] get_snapset_context 3cc747d2c1c4bbb087f14d3eef7970f0 0 -> 1 2014-04-25 09:18:55.098095 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 clone_overlap {} 2014-04-25 09:18:55.098110 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 data_subset [0~2355200] clone_subsets {} 2014-04-25 09:18:55.098123 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] put_snapset_context 3cc747d2c1c4bbb087f14d3eef7970f0 1 -> 0 2014-04-25 09:18:55.098143 7f77edce2700 7 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] send_push_op 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 v 979'623 size 2355200 recovery_info: ObjectRecoveryInfo(1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3@979'623, copy_subset: [0~2355200], clone_subset: {}) 2014-04-25 09:18:55.100006 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48226 s=2 pgs=7819 cs=9 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:55.100518 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.110815 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48263 s=2 pgs=7820 cs=11 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:55.111285 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.116639 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48303 s=2 pgs=7821 cs=13 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:55.117264 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.121228 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48330 s=2 pgs=7822 cs=15 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:55.121764 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.124010 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_unlock on 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 2014-04-25 09:18:55.124116 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] put_object_context 0x1c7d2f00 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 1 -> 0 2014-04-25 09:18:55.124136 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] recover_replicas: recover_object_replicas(46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3) 2014-04-25 09:18:55.124152 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] prep_object_replica_pushes: on 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 2014-04-25 09:18:55.124439 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] populate_obc_watchers 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 2014-04-25 09:18:55.124458 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] get_object_context 0x1c7d2f00 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 0 -> 1 read 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3(988'624 client.56006.0:1200742 wrlock_by=unknown.0.0:0) 2014-04-25 09:18:55.124474 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_lock for 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 2014-04-25 09:18:55.124487 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] start_recovery_op 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 2014-04-25 09:18:55.124500 7f77edce2700 10 osd.210 3800 start_recovery_op pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 (7/15 rops) 2014-04-25 09:18:55.124532 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] prep_push_to_replica46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 v988'624 size 2355200 to osd.153 2014-04-25 09:18:55.124701 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] get_snapset_context 8b2bcbd741bc4f272c9c4331297f15ff 0 -> 1 2014-04-25 09:18:55.124721 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 clone_overlap {} 2014-04-25 09:18:55.124736 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 data_subset [0~2355200] clone_subsets {} 2014-04-25 09:18:55.124749 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] put_snapset_context 8b2bcbd741bc4f272c9c4331297f15ff 1 -> 0 2014-04-25 09:18:55.124770 7f77edce2700 7 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] send_push_op 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 v 988'624 size 2355200 recovery_info: ObjectRecoveryInfo(46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3@988'624, copy_subset: [0~2355200], clone_subset: {}) 2014-04-25 09:18:55.128994 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48349 s=2 pgs=7823 cs=17 l=0 c=0x15d39340).fault, initiating reconnect 2014-04-25 09:18:55.129628 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd 2014-04-25 09:18:55.135839 7f77d356b700 -1 msg/Pipe.cc: In function 'int Pipe::connect()' thread 7f77d356b700 time 2014-04-25 09:18:55.135010 msg/Pipe.cc: 1045: FAILED assert(m) ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73) 1: (Pipe::connect()+0x3cb5) [0x992435] 2: (Pipe::writer()+0x5f3) [0x992d83] 3: (Pipe::Writer::entry()+0xd) [0x99ddfd] 4: /lib64/libpthread.so.0() [0x3a6e407851] 5: (clone()+0x6d) [0x3a6e0e890d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -51> 2014-04-25 09:18:55.095846 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_unlock on 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 -50> 2014-04-25 09:18:55.095925 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] put_object_context 0x1c7d2f00 7cab6964/eb6ccdaaa67adb1b0ae4417d079fa03c/head//3 1 -> 0 -49> 2014-04-25 09:18:55.095978 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] recover_replicas: recover_object_replicas(1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3) -48> 2014-04-25 09:18:55.095996 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] prep_object_replica_pushes: on 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 -47> 2014-04-25 09:18:55.097701 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] populate_obc_watchers 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 -46> 2014-04-25 09:18:55.097721 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] get_object_context 0x1c7d2f00 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 0 -> 1 read 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3(979'623 client.40137.0:1207807 wrlock_by=unknown.0.0:0) -45> 2014-04-25 09:18:55.097739 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_lock for 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 -44> 2014-04-25 09:18:55.097754 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=1 lcod 0'0 mlcod 0'0 active+recovering] start_recovery_op 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 -43> 2014-04-25 09:18:55.097766 7f77edce2700 10 osd.210 3800 start_recovery_op pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 (6/15 rops) -42> 2014-04-25 09:18:55.097841 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] prep_push_to_replica1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 v979'623 size 2355200 to osd.153 -41> 2014-04-25 09:18:55.098036 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] get_snapset_context 3cc747d2c1c4bbb087f14d3eef7970f0 0 -> 1 -40> 2014-04-25 09:18:55.098095 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 clone_overlap {} -39> 2014-04-25 09:18:55.098110 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 data_subset [0~2355200] clone_subsets {} -38> 2014-04-25 09:18:55.098123 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] put_snapset_context 3cc747d2c1c4bbb087f14d3eef7970f0 1 -> 0 -37> 2014-04-25 09:18:55.098143 7f77edce2700 7 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] send_push_op 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 v 979'623 size 2355200 recovery_info: ObjectRecoveryInfo(1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3@979'623, copy_subset: [0~2355200], clone_subset: {}) -36> 2014-04-25 09:18:55.099942 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48226 s=2 pgs=7819 cs=9 l=0 c=0x15d39340).reader couldn't read tag, Success -35> 2014-04-25 09:18:55.099989 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48226 s=2 pgs=7819 cs=9 l=0 c=0x15d39340).fault 0: Success -34> 2014-04-25 09:18:55.100006 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48226 s=2 pgs=7819 cs=9 l=0 c=0x15d39340).fault, initiating reconnect -33> 2014-04-25 09:18:55.100518 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -32> 2014-04-25 09:18:55.110741 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48263 s=2 pgs=7820 cs=11 l=0 c=0x15d39340).reader couldn't read tag, Success -31> 2014-04-25 09:18:55.110800 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48263 s=2 pgs=7820 cs=11 l=0 c=0x15d39340).fault 0: Success -30> 2014-04-25 09:18:55.110815 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48263 s=2 pgs=7820 cs=11 l=0 c=0x15d39340).fault, initiating reconnect -29> 2014-04-25 09:18:55.111285 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -28> 2014-04-25 09:18:55.116567 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48303 s=2 pgs=7821 cs=13 l=0 c=0x15d39340).reader couldn't read tag, Success -27> 2014-04-25 09:18:55.116623 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48303 s=2 pgs=7821 cs=13 l=0 c=0x15d39340).fault 0: Success -26> 2014-04-25 09:18:55.116639 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48303 s=2 pgs=7821 cs=13 l=0 c=0x15d39340).fault, initiating reconnect -25> 2014-04-25 09:18:55.117264 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -24> 2014-04-25 09:18:55.121160 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48330 s=2 pgs=7822 cs=15 l=0 c=0x15d39340).reader couldn't read tag, Success -23> 2014-04-25 09:18:55.121212 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48330 s=2 pgs=7822 cs=15 l=0 c=0x15d39340).fault 0: Success -22> 2014-04-25 09:18:55.121228 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48330 s=2 pgs=7822 cs=15 l=0 c=0x15d39340).fault, initiating reconnect -21> 2014-04-25 09:18:55.121764 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -20> 2014-04-25 09:18:55.124010 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_unlock on 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 -19> 2014-04-25 09:18:55.124116 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] put_object_context 0x1c7d2f00 1b566964/3cc747d2c1c4bbb087f14d3eef7970f0/head//3 1 -> 0 -18> 2014-04-25 09:18:55.124136 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] recover_replicas: recover_object_replicas(46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3) -17> 2014-04-25 09:18:55.124152 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] prep_object_replica_pushes: on 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 -16> 2014-04-25 09:18:55.124439 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] populate_obc_watchers 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 -15> 2014-04-25 09:18:55.124458 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] get_object_context 0x1c7d2f00 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 0 -> 1 read 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3(988'624 client.56006.0:1200742 wrlock_by=unknown.0.0:0) -14> 2014-04-25 09:18:55.124474 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] ondisk_read_lock for 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 -13> 2014-04-25 09:18:55.124487 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=2 lcod 0'0 mlcod 0'0 active+recovering] start_recovery_op 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 -12> 2014-04-25 09:18:55.124500 7f77edce2700 10 osd.210 3800 start_recovery_op pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 (7/15 rops) -11> 2014-04-25 09:18:55.124532 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] prep_push_to_replica46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 v988'624 size 2355200 to osd.153 -10> 2014-04-25 09:18:55.124701 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] get_snapset_context 8b2bcbd741bc4f272c9c4331297f15ff 0 -> 1 -9> 2014-04-25 09:18:55.124721 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 clone_overlap {} -8> 2014-04-25 09:18:55.124736 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] calc_head_subsets 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 data_subset [0~2355200] clone_subsets {} -7> 2014-04-25 09:18:55.124749 7f77edce2700 10 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] put_snapset_context 8b2bcbd741bc4f272c9c4331297f15ff 1 -> 0 -6> 2014-04-25 09:18:55.124770 7f77edce2700 7 osd.210 pg_epoch: 3799 pg[3.6964( v 1110'686 (0'0,1110'686] local-les=3754 n=686 ec=633 les/c 3754/3715 3753/3753/3753) [210,221,153] r=0 lpr=3753 pi=3714-3752/5 rops=3 lcod 0'0 mlcod 0'0 active+recovering] send_push_op 46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3 v 988'624 size 2355200 recovery_info: ObjectRecoveryInfo(46c46964/8b2bcbd741bc4f272c9c4331297f15ff/head//3@988'624, copy_subset: [0~2355200], clone_subset: {}) -5> 2014-04-25 09:18:55.128847 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48349 s=2 pgs=7823 cs=17 l=0 c=0x15d39340).reader couldn't read tag, Success -4> 2014-04-25 09:18:55.128972 7f77cade4700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48349 s=2 pgs=7823 cs=17 l=0 c=0x15d39340).fault 0: Success -3> 2014-04-25 09:18:55.128994 7f77cade4700 0 -- 10.193.207.183:6889/6044082 >> 10.193.207.176:6963/6057494 pipe(0x12dce500 sd=56 :48349 s=2 pgs=7823 cs=17 l=0 c=0x15d39340).fault, initiating reconnect -2> 2014-04-25 09:18:55.129628 7f77d093f700 10 osd.210 3800 OSD::ms_get_authorizer type=osd -1> 2014-04-25 09:18:55.134989 7f77d356b700 2 -- 10.193.207.183:6889/6044082 >> 10.193.207.178:6817/5052082 pipe(0x13374500 sd=128 :50975 s=1 pgs=0 cs=0 l=0 c=0x123b4b00). got newly_acked_seq 3 vs out_seq 0 0> 2014-04-25 09:18:55.135839 7f77d356b700 -1 msg/Pipe.cc: In function 'int Pipe::connect()' thread 7f77d356b700 time 2014-04-25 09:18:55.135010 msg/Pipe.cc: 1045: FAILED assert(m) ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73) 1: (Pipe::connect()+0x3cb5) [0x992435] 2: (Pipe::writer()+0x5f3) [0x992d83] 3: (Pipe::Writer::entry()+0xd) [0x99ddfd] 4: /lib64/libpthread.so.0() [0x3a6e407851] 5: (clone()+0x6d) [0x3a6e0e890d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.