Bug #9082
closedCeph Firefly 0.80.5 : PG has invalid (post-split) stats; must scrub before tier agent can activate
Description
Hello
Ceph version : 0.80.5
Centos 6.5
Features in use : erasure coding and cache tiering
Few hours back my cluster was working absolutely fine , all OSD's were up and running.
As soon as i started benchmarking my erasure-coded pool which is has a tired cache pool , my osds are started getting DOWN and OUT . Immediately 6 OSDS went down and all of them are throwing the same error. All of these osds which went down are from cache pool crush ruleset.
2014-08-12 17:43:21.586831 7f4c5e33a700 0 log [WRN] : pg 24.104 has invalid (post-split) stats; must scrub before tier agent can activate
2014-08-12 17:43:21.593174 7f4c5d939700 -1 ** Caught signal (Floating point exception) *
When i am manually trying to start these ODS , they are not coming up and throwing the same error message (mentioned above)
I have been reading Firefly release notes ( http://ceph.com/releases/v0-80-firefly-released/ ) , looks like it was fixed earlier
osd: warn if agent cannot enable due to invalid (post-split) stats (Sage Weil)
But in version 0.80.5 , i am still facing this. Is this something you already know ??
Below are the OSD detailed LOGS:
2014-08-12 17:43:21.584841 7f4c5d939700 0 log [WRN] : pg 24.30f has invalid (post-split) stats; must scrub before tier agent can activate -175> 2014-08-12 17:43:21.584995 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.6 192.168.1.107:6809/53141 13 ==== pg_info(1 pgs e194300:22.795) v4 ==== 714+0+0 (1092772418 0 0) 0x62ba300 con 0x40cc360 -174> 2014-08-12 17:43:21.585030 7f4c65145700 5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.584855, event: header_read, request: pg_info(1 pgs e194300:22.795) v4 -173> 2014-08-12 17:43:21.585044 7f4c65145700 5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.584859, event: throttled, request: pg_info(1 pgs e194300:22.795) v4 -172> 2014-08-12 17:43:21.585055 7f4c65145700 5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.584962, event: all_read, request: pg_info(1 pgs e194300:22.795) v4 -171> 2014-08-12 17:43:21.585064 7f4c65145700 5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.585024, event: dispatched, request: pg_info(1 pgs e194300:22.795) v4 -170> 2014-08-12 17:43:21.585074 7f4c65145700 5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.585074, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:22.795) v4 -169> 2014-08-12 17:43:21.585087 7f4c65145700 5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.585087, event: started, request: pg_info(1 pgs e194300:22.795) v4 -168> 2014-08-12 17:43:21.585116 7f4c65145700 5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.585116, event: done, request: pg_info(1 pgs e194300:22.795) v4 -167> 2014-08-12 17:43:21.585173 7f4c5e33a700 0 log [WRN] : pg 24.2a9 has invalid (post-split) stats; must scrub before tier agent can activate -166> 2014-08-12 17:43:21.585232 7f4c5d939700 0 log [WRN] : pg 24.254 has invalid (post-split) stats; must scrub before tier agent can activate -165> 2014-08-12 17:43:21.585226 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[24.2a9( v 194216'12832 (192834'10938,194216'12832] local-les=194300 n=4 ec=192688 les/c 194300/194235 194298/194299/194299) [306,24,61] r=0 lpr=194299 pi=194232-194298/5 crt=192888'12436 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.366973 12 0.000638 -164> 2014-08-12 17:43:21.585259 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[24.2a9( v 194216'12832 (192834'10938,194216'12832] local-les=194300 n=4 ec=192688 les/c 194300/194235 194298/194299/194299) [306,24,61] r=0 lpr=194299 pi=194232-194298/5 crt=192888'12436 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -163> 2014-08-12 17:43:21.585270 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.30f( v 194217'14634 (192718'11887,194217'14634] local-les=194300 n=5 ec=192688 les/c 194300/194235 194298/194299/194299) [306,308,62] r=0 lpr=194299 pi=194232-194298/5 crt=192888'14229 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.368068 12 0.000658 -162> 2014-08-12 17:43:21.585288 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[24.2a9( v 194216'12832 (192834'10938,194216'12832] local-les=194300 n=4 ec=192688 les/c 194300/194235 194298/194299/194299) [306,24,61] r=0 lpr=194299 pi=194232-194298/5 crt=192888'12436 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000028 0 0.000000 -161> 2014-08-12 17:43:21.585309 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.30f( v 194217'14634 (192718'11887,194217'14634] local-les=194300 n=5 ec=192688 les/c 194300/194235 194298/194299/194299) [306,308,62] r=0 lpr=194299 pi=194232-194298/5 crt=192888'14229 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -160> 2014-08-12 17:43:21.585311 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[24.2a9( v 194216'12832 (192834'10938,194216'12832] local-les=194300 n=4 ec=192688 les/c 194300/194235 194298/194299/194299) [306,24,61] r=0 lpr=194299 pi=194232-194298/5 crt=192888'12436 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -159> 2014-08-12 17:43:21.585339 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.30f( v 194217'14634 (192718'11887,194217'14634] local-les=194300 n=5 ec=192688 les/c 194300/194235 194298/194299/194299) [306,308,62] r=0 lpr=194299 pi=194232-194298/5 crt=192888'14229 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000031 0 0.000000 -158> 2014-08-12 17:43:21.585363 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.30f( v 194217'14634 (192718'11887,194217'14634] local-les=194300 n=5 ec=192688 les/c 194300/194235 194298/194299/194299) [306,308,62] r=0 lpr=194299 pi=194232-194298/5 crt=192888'14229 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -157> 2014-08-12 17:43:21.585368 7f4c5e33a700 1 -- 192.168.1.101:6800/51417 --> osd.24 192.168.1.107:6800/52669 -- pg_info(1 pgs e194300:24.2a9) v4 -- ?+0 0x6038c40 -156> 2014-08-12 17:43:21.585370 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.6 192.168.1.107:6809/53141 14 ==== MRecoveryReserve REQUEST pgid: 22.795, query_epoch: 194300 v2 ==== 26+0+0 (3761680802 0 0) 0x62b8540 con 0x40cc360 -155> 2014-08-12 17:43:21.585406 7f4c65145700 5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585265, event: header_read, request: MRecoveryReserve REQUEST pgid: 22.795, query_epoch: 194300 v2 -154> 2014-08-12 17:43:21.585419 7f4c65145700 5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585268, event: throttled, request: MRecoveryReserve REQUEST pgid: 22.795, query_epoch: 194300 v2 -153> 2014-08-12 17:43:21.585430 7f4c65145700 5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585345, event: all_read, request: MRecoveryReserve REQUEST pgid: 22.795, query_epoch: 194300 v2 -152> 2014-08-12 17:43:21.585421 7f4c5d939700 1 -- 192.168.1.101:6800/51417 --> osd.62 192.168.1.112:6824/5505 -- pg_info(1 pgs e194300:24.30f) v4 -- ?+0 0x62baf40 -151> 2014-08-12 17:43:21.585439 7f4c65145700 5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585402, event: dispatched, request: MRecoveryReserve REQUEST pgid: 22.795, query_epoch: 194300 v2 -150> 2014-08-12 17:43:21.585446 7f4c65145700 5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585446, event: waiting_for_osdmap, request: MRecoveryReserve REQUEST pgid: 22.795, query_epoch: 194300 v2 -149> 2014-08-12 17:43:21.585467 7f4c65145700 5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585467, event: done, request: MRecoveryReserve REQUEST pgid: 22.795, query_epoch: 194300 v2 -148> 2014-08-12 17:43:21.585465 7f4c5d939700 1 -- 192.168.1.101:6800/51417 --> osd.308 192.168.1.113:6802/14403 -- pg_info(1 pgs e194300:24.30f) v4 -- ?+0 0x62babc0 -147> 2014-08-12 17:43:21.585650 7f4c5e33a700 1 -- 192.168.1.101:6800/51417 --> osd.61 192.168.1.111:6840/16566 -- pg_info(1 pgs e194300:24.2a9) v4 -- ?+0 0x603b2c0 -146> 2014-08-12 17:43:21.585680 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.254( v 194215'17471 (192803'14472,194215'17471] local-les=194300 n=-9 ec=192688 les/c 194300/194235 194298/194299/194299) [306,289,6] r=0 lpr=194299 pi=193810-194298/7 crt=192888'17092 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.368052 12 0.000598 -145> 2014-08-12 17:43:21.585737 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.254( v 194215'17471 (192803'14472,194215'17471] local-les=194300 n=-9 ec=192688 les/c 194300/194235 194298/194299/194299) [306,289,6] r=0 lpr=194299 pi=193810-194298/7 crt=192888'17092 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -144> 2014-08-12 17:43:21.585766 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.254( v 194215'17471 (192803'14472,194215'17471] local-les=194300 n=-9 ec=192688 les/c 194300/194235 194298/194299/194299) [306,289,6] r=0 lpr=194299 pi=193810-194298/7 crt=192888'17092 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000029 0 0.000000 -143> 2014-08-12 17:43:21.585786 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.254( v 194215'17471 (192803'14472,194215'17471] local-les=194300 n=-9 ec=192688 les/c 194300/194235 194298/194299/194299) [306,289,6] r=0 lpr=194299 pi=193810-194298/7 crt=192888'17092 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -142> 2014-08-12 17:43:21.585798 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.49 192.168.1.111:6824/11112 -- pg_info(1 pgs e194300:24.104) v4 -- ?+0 0x62bb9c0 -141> 2014-08-12 17:43:21.585845 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.59 192.168.1.112:6822/5265 -- osd pg remove(epoch 194300; pg24.30f; ) v2 -- ?+0 0x6d840c0 -140> 2014-08-12 17:43:21.585892 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.301 192.168.1.113:6814/16727 -- osd pg remove(epoch 194300; pg24.30f; ) v2 -- ?+0 0x6d83f00 -139> 2014-08-12 17:43:21.585922 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.292 192.168.1.113:6812/16056 -- pg_info(1 pgs e194300:24.104) v4 -- ?+0 0x603f380 -138> 2014-08-12 17:43:21.586096 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.379 192.168.1.108:6830/4861 -- osd pg remove(epoch 194300; pg24.30f; ) v2 -- ?+0 0x6d83d40 -137> 2014-08-12 17:43:21.586156 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[22.795( v 194215'502 lc 0'0 (0'0,194215'502] local-les=194299 n=26 ec=183096 les/c 194299/194273 194298/194298/192967) [6,306,374] r=1 lpr=194299 pi=194270-194297/3 luod=0'0 crt=194215'502 active m=26] exit Started/ReplicaActive/RepNotRecovering 0.801323 7 0.000751 -136> 2014-08-12 17:43:21.586175 7f4c5d939700 1 -- 192.168.1.101:6800/51417 --> osd.6 192.168.1.107:6809/53141 -- pg_info(1 pgs e194300:24.254) v4 -- ?+0 0x62bb800 -135> 2014-08-12 17:43:21.586208 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[22.795( v 194215'502 lc 0'0 (0'0,194215'502] local-les=194299 n=26 ec=183096 les/c 194299/194273 194298/194298/192967) [6,306,374] r=1 lpr=194299 pi=194270-194297/3 luod=0'0 crt=194215'502 active m=26] enter Started/ReplicaActive/RepWaitRecoveryReserved -134> 2014-08-12 17:43:21.586229 7f4c5d939700 1 -- 192.168.1.101:6800/51417 --> osd.289 192.168.1.113:6808/15284 -- pg_info(1 pgs e194300:24.254) v4 -- ?+0 0x62ba140 -133> 2014-08-12 17:43:21.586394 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.6 192.168.1.107:6809/53141 -- osd pg remove(epoch 194300; pg24.2a9; ) v2 -- ?+0 0x6d81880 -132> 2014-08-12 17:43:21.586444 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.29 192.168.1.115:6806/59028 -- osd pg remove(epoch 194300; pg24.2a9; ) v2 -- ?+0 0x6d81c00 -131> 2014-08-12 17:43:21.586474 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.47 192.168.1.111:6828/12102 -- osd pg remove(epoch 194300; pg24.2a9; ) v2 -- ?+0 0x62bdcc0 -130> 2014-08-12 17:43:21.586625 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.18 192.168.1.107:6805/52903 -- osd pg remove(epoch 194300; pg24.254; ) v2 -- ?+0 0x62bbf00 -129> 2014-08-12 17:43:21.586676 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.292 192.168.1.113:6812/16056 -- osd pg remove(epoch 194300; pg24.254; ) v2 -- ?+0 0x62bc980 -128> 2014-08-12 17:43:21.586701 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.344 192.168.1.106:6812/6757 -- osd pg remove(epoch 194300; pg24.254; ) v2 -- ?+0 0x62b9880 -127> 2014-08-12 17:43:21.586831 7f4c5e33a700 0 log [WRN] : pg 24.104 has invalid (post-split) stats; must scrub before tier agent can activate -126> 2014-08-12 17:43:21.586875 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[24.104( v 194215'15298 (192718'12373,194215'15298] local-les=194300 n=12 ec=192688 les/c 194300/194235 194298/194299/194299) [306,49,292] r=0 lpr=194299 pi=194232-194298/5 crt=192834'14593 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.368029 12 0.000767 -125> 2014-08-12 17:43:21.586909 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[24.104( v 194215'15298 (192718'12373,194215'15298] local-les=194300 n=12 ec=192688 les/c 194300/194235 194298/194299/194299) [306,49,292] r=0 lpr=194299 pi=194232-194298/5 crt=192834'14593 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -124> 2014-08-12 17:43:21.586935 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[24.104( v 194215'15298 (192718'12373,194215'15298] local-les=194300 n=12 ec=192688 les/c 194300/194235 194298/194299/194299) [306,49,292] r=0 lpr=194299 pi=194232-194298/5 crt=192834'14593 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000026 0 0.000000 -123> 2014-08-12 17:43:21.586958 7f4c5e33a700 5 osd.306 pg_epoch: 194300 pg[24.104( v 194215'15298 (192718'12373,194215'15298] local-les=194300 n=12 ec=192688 les/c 194300/194235 194298/194299/194299) [306,49,292] r=0 lpr=194299 pi=194232-194298/5 crt=192834'14593 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -122> 2014-08-12 17:43:21.587005 7f4c5e33a700 1 -- 192.168.1.101:6800/51417 --> osd.49 192.168.1.111:6824/11112 -- pg_info(1 pgs e194300:24.104) v4 -- ?+0 0x6d863c0 -121> 2014-08-12 17:43:21.587043 7f4c5e33a700 1 -- 192.168.1.101:6800/51417 --> osd.292 192.168.1.113:6812/16056 -- pg_info(1 pgs e194300:24.104) v4 -- ?+0 0x6d86200 -120> 2014-08-12 17:43:21.587364 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.344 192.168.1.106:6812/6757 -- pg_info(1 pgs e194300:24.58) v4 -- ?+0 0x603a680 -119> 2014-08-12 17:43:21.587407 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.47 192.168.1.111:6828/12102 -- osd pg remove(epoch 194300; pg24.104; ) v2 -- ?+0 0x62bbb80 -118> 2014-08-12 17:43:21.587444 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.352 192.168.1.108:6826/4273 -- pg_info(1 pgs e194300:24.58) v4 -- ?+0 0x603aa00 -117> 2014-08-12 17:43:21.587478 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.295 192.168.1.113:6806/14945 -- osd pg remove(epoch 194300; pg24.104; ) v2 -- ?+0 0x62baa00 -116> 2014-08-12 17:43:21.587510 7f4c69567700 1 -- 192.168.1.101:6800/51417 --> osd.397 192.168.1.105:6808/41085 -- osd pg remove(epoch 194300; pg24.104; ) v2 -- ?+0 0x62ba4c0 -115> 2014-08-12 17:43:21.587533 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.58( v 194217'14171 (192718'11167,194217'14171] local-les=194300 n=-1 ec=192688 les/c 194300/193814 194298/194299/194299) [306,352,344]/[306] r=0 lpr=194299 pi=193813-194298/13 bft=344,352 crt=0'0 lcod 0'0 mlcod 0'0 active+degraded+remapped] on activate: bft=344,352 from 0//0//-1 -114> 2014-08-12 17:43:21.587566 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.58( v 194217'14171 (192718'11167,194217'14171] local-les=194300 n=-1 ec=192688 les/c 194300/193814 194298/194299/194299) [306,352,344]/[306] r=0 lpr=194299 pi=193813-194298/13 bft=344,352 crt=0'0 lcod 0'0 mlcod 0'0 active+degraded+remapped] target shard 344 from 0//0//-1 -113> 2014-08-12 17:43:21.587584 7f4c5d939700 5 osd.306 pg_epoch: 194300 pg[24.58( v 194217'14171 (192718'11167,194217'14171] local-les=194300 n=-1 ec=192688 les/c 194300/193814 194298/194299/194299) [306,352,344]/[306] r=0 lpr=194299 pi=193813-194298/13 bft=344,352 crt=0'0 lcod 0'0 mlcod 0'0 active+degraded+remapped] target shard 352 from 0//0//-1 -112> 2014-08-12 17:43:21.587939 7f4c5d939700 0 log [WRN] : pg 24.58 has invalid (post-split) stats; must scrub before tier agent can activate -111> 2014-08-12 17:43:21.588814 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.302 192.168.1.113:6810/15731 -- pg_info(1 pgs e194300:24.3f1) v4 -- ?+0 0x603b640 -110> 2014-08-12 17:43:21.588955 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.290 192.168.1.109:6808/31607 -- pg_info(1 pgs e194300:24.119) v4 -- ?+0 0x6039a40 -109> 2014-08-12 17:43:21.589098 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.18 192.168.1.107:6805/52903 -- pg_info(1 pgs e194300:24.1c4) v4 -- ?+0 0x603db00 -108> 2014-08-12 17:43:21.589383 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.313 192.168.1.109:6820/35445 -- pg_info(1 pgs e194300:24.13b) v4 -- ?+0 0x6038540 -107> 2014-08-12 17:43:21.589673 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.342 192.168.1.110:6802/41438 -- pg_info(1 pgs e194300:24.36d) v4 -- ?+0 0x603d240 -106> 2014-08-12 17:43:21.589818 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.18 192.168.1.107:6805/52903 13 ==== pg_info(1 pgs e194300:24.1c4) v4 ==== 748+0+0 (2739292217 0 0) 0x603c280 con 0x40cdac0 -105> 2014-08-12 17:43:21.589857 7f4c65145700 5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589678, event: header_read, request: pg_info(1 pgs e194300:24.1c4) v4 -104> 2014-08-12 17:43:21.589872 7f4c65145700 5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589680, event: throttled, request: pg_info(1 pgs e194300:24.1c4) v4 -103> 2014-08-12 17:43:21.589881 7f4c65145700 5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589760, event: all_read, request: pg_info(1 pgs e194300:24.1c4) v4 -102> 2014-08-12 17:43:21.589888 7f4c65145700 5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589852, event: dispatched, request: pg_info(1 pgs e194300:24.1c4) v4 -101> 2014-08-12 17:43:21.589898 7f4c65145700 5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589898, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.1c4) v4 -100> 2014-08-12 17:43:21.589912 7f4c65145700 5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589911, event: started, request: pg_info(1 pgs e194300:24.1c4) v4 -99> 2014-08-12 17:43:21.589972 7f4c65145700 5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589972, event: done, request: pg_info(1 pgs e194300:24.1c4) v4 -98> 2014-08-12 17:43:21.589998 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.302 192.168.1.113:6810/15731 12 ==== pg_info(1 pgs e194300:24.3f1) v4 ==== 744+0+0 (1504649001 0 0) 0x603b640 con 0x48baec0 -97> 2014-08-12 17:43:21.590017 7f4c65145700 5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.589838, event: header_read, request: pg_info(1 pgs e194300:24.3f1) v4 -96> 2014-08-12 17:43:21.590028 7f4c65145700 5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.589842, event: throttled, request: pg_info(1 pgs e194300:24.3f1) v4 -95> 2014-08-12 17:43:21.590037 7f4c65145700 5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.589945, event: all_read, request: pg_info(1 pgs e194300:24.3f1) v4 -94> 2014-08-12 17:43:21.590046 7f4c65145700 5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.590015, event: dispatched, request: pg_info(1 pgs e194300:24.3f1) v4 -93> 2014-08-12 17:43:21.590056 7f4c65145700 5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.590055, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.3f1) v4 -92> 2014-08-12 17:43:21.590067 7f4c65145700 5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.590066, event: started, request: pg_info(1 pgs e194300:24.3f1) v4 -91> 2014-08-12 17:43:21.590095 7f4c65145700 5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.590094, event: done, request: pg_info(1 pgs e194300:24.3f1) v4 -90> 2014-08-12 17:43:21.590113 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.290 192.168.1.109:6808/31607 16 ==== pg_info(1 pgs e194300:24.119) v4 ==== 748+0+0 (847792910 0 0) 0x6039a40 con 0x40c9760 -89> 2014-08-12 17:43:21.590110 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.395 192.168.1.105:6810/41287 -- pg_info(1 pgs e194300:24.354) v4 -- ?+0 0x6039880 -88> 2014-08-12 17:43:21.590131 7f4c65145700 5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.589950, event: header_read, request: pg_info(1 pgs e194300:24.119) v4 -87> 2014-08-12 17:43:21.590141 7f4c65145700 5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.589953, event: throttled, request: pg_info(1 pgs e194300:24.119) v4 -86> 2014-08-12 17:43:21.590148 7f4c65145700 5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590035, event: all_read, request: pg_info(1 pgs e194300:24.119) v4 -85> 2014-08-12 17:43:21.590154 7f4c65145700 5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590129, event: dispatched, request: pg_info(1 pgs e194300:24.119) v4 -84> 2014-08-12 17:43:21.590160 7f4c65145700 5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590159, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.119) v4 -83> 2014-08-12 17:43:21.590167 7f4c65145700 5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590167, event: started, request: pg_info(1 pgs e194300:24.119) v4 -82> 2014-08-12 17:43:21.590186 7f4c65145700 5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590186, event: done, request: pg_info(1 pgs e194300:24.119) v4 -81> 2014-08-12 17:43:21.590199 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.18 192.168.1.107:6805/52903 14 ==== pg_info(1 pgs e194300:24.1c4) v4 ==== 748+0+0 (660960379 0 0) 0x603db00 con 0x40cdac0 -80> 2014-08-12 17:43:21.590213 7f4c65145700 5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.589972, event: header_read, request: pg_info(1 pgs e194300:24.1c4) v4 -79> 2014-08-12 17:43:21.590219 7f4c65145700 5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.589974, event: throttled, request: pg_info(1 pgs e194300:24.1c4) v4 -78> 2014-08-12 17:43:21.590225 7f4c65145700 5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590060, event: all_read, request: pg_info(1 pgs e194300:24.1c4) v4 -77> 2014-08-12 17:43:21.590230 7f4c65145700 5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590211, event: dispatched, request: pg_info(1 pgs e194300:24.1c4) v4 -76> 2014-08-12 17:43:21.590240 7f4c65145700 5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590240, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.1c4) v4 -75> 2014-08-12 17:43:21.590253 7f4c65145700 5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590252, event: started, request: pg_info(1 pgs e194300:24.1c4) v4 -74> 2014-08-12 17:43:21.590266 7f4c69f68700 1 -- 192.168.1.101:6800/51417 --> osd.308 192.168.1.113:6802/14403 -- pg_info(1 pgs e194300:24.2dc) v4 -- ?+0 0x603c440 -73> 2014-08-12 17:43:21.590274 7f4c65145700 5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590274, event: done, request: pg_info(1 pgs e194300:24.1c4) v4 -72> 2014-08-12 17:43:21.590293 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.313 192.168.1.109:6820/35445 11 ==== pg_info(1 pgs e194300:24.13b) v4 ==== 748+0+0 (1395045469 0 0) 0x6038540 con 0x40ca680 -71> 2014-08-12 17:43:21.590308 7f4c65145700 5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590201, event: header_read, request: pg_info(1 pgs e194300:24.13b) v4 -70> 2014-08-12 17:43:21.590315 7f4c65145700 5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590203, event: throttled, request: pg_info(1 pgs e194300:24.13b) v4 -69> 2014-08-12 17:43:21.590320 7f4c65145700 5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590275, event: all_read, request: pg_info(1 pgs e194300:24.13b) v4 -68> 2014-08-12 17:43:21.590326 7f4c65145700 5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590306, event: dispatched, request: pg_info(1 pgs e194300:24.13b) v4 -67> 2014-08-12 17:43:21.590332 7f4c65145700 5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590331, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.13b) v4 -66> 2014-08-12 17:43:21.590343 7f4c65145700 5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590343, event: started, request: pg_info(1 pgs e194300:24.13b) v4 -65> 2014-08-12 17:43:21.590366 7f4c65145700 5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590366, event: done, request: pg_info(1 pgs e194300:24.13b) v4 -64> 2014-08-12 17:43:21.590795 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.342 192.168.1.110:6802/41438 7 ==== pg_info(1 pgs e194300:24.36d) v4 ==== 748+0+0 (330703328 0 0) 0x2aec0c0 con 0x41d3860 -63> 2014-08-12 17:43:21.590820 7f4c65145700 5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590687, event: header_read, request: pg_info(1 pgs e194300:24.36d) v4 -62> 2014-08-12 17:43:21.590832 7f4c65145700 5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590689, event: throttled, request: pg_info(1 pgs e194300:24.36d) v4 -61> 2014-08-12 17:43:21.590841 7f4c65145700 5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590773, event: all_read, request: pg_info(1 pgs e194300:24.36d) v4 -60> 2014-08-12 17:43:21.590850 7f4c65145700 5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590817, event: dispatched, request: pg_info(1 pgs e194300:24.36d) v4 -59> 2014-08-12 17:43:21.590859 7f4c65145700 5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590859, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.36d) v4 -58> 2014-08-12 17:43:21.590871 7f4c65145700 5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590871, event: started, request: pg_info(1 pgs e194300:24.36d) v4 -57> 2014-08-12 17:43:21.590900 7f4c65145700 5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590900, event: done, request: pg_info(1 pgs e194300:24.36d) v4 -56> 2014-08-12 17:43:21.591093 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.313 192.168.1.109:6820/35445 12 ==== pg_info(1 pgs e194300:24.13b) v4 ==== 748+0+0 (2101955393 0 0) 0x6d82840 con 0x40ca680 -55> 2014-08-12 17:43:21.591136 7f4c65145700 5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.590938, event: header_read, request: pg_info(1 pgs e194300:24.13b) v4 -54> 2014-08-12 17:43:21.591152 7f4c65145700 5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.590940, event: throttled, request: pg_info(1 pgs e194300:24.13b) v4 -53> 2014-08-12 17:43:21.591162 7f4c65145700 5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591008, event: all_read, request: pg_info(1 pgs e194300:24.13b) v4 -52> 2014-08-12 17:43:21.591171 7f4c65145700 5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591130, event: dispatched, request: pg_info(1 pgs e194300:24.13b) v4 -51> 2014-08-12 17:43:21.591187 7f4c65145700 5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591187, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.13b) v4 -50> 2014-08-12 17:43:21.591200 7f4c65145700 5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591200, event: started, request: pg_info(1 pgs e194300:24.13b) v4 -49> 2014-08-12 17:43:21.591232 7f4c65145700 5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591231, event: done, request: pg_info(1 pgs e194300:24.13b) v4 -48> 2014-08-12 17:43:21.591256 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.395 192.168.1.105:6810/41287 7 ==== pg_info(1 pgs e194300:24.354) v4 ==== 748+0+0 (4289877745 0 0) 0x6039880 con 0x40cf0c0 -47> 2014-08-12 17:43:21.591273 7f4c65145700 5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.590969, event: header_read, request: pg_info(1 pgs e194300:24.354) v4 -46> 2014-08-12 17:43:21.591280 7f4c65145700 5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.590972, event: throttled, request: pg_info(1 pgs e194300:24.354) v4 -45> 2014-08-12 17:43:21.591285 7f4c65145700 5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591052, event: all_read, request: pg_info(1 pgs e194300:24.354) v4 -44> 2014-08-12 17:43:21.591291 7f4c65145700 5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591270, event: dispatched, request: pg_info(1 pgs e194300:24.354) v4 -43> 2014-08-12 17:43:21.591297 7f4c65145700 5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591296, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.354) v4 -42> 2014-08-12 17:43:21.591304 7f4c65145700 5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591304, event: started, request: pg_info(1 pgs e194300:24.354) v4 -41> 2014-08-12 17:43:21.591324 7f4c65145700 5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591324, event: done, request: pg_info(1 pgs e194300:24.354) v4 -40> 2014-08-12 17:43:21.591344 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.308 192.168.1.113:6802/14403 11 ==== pg_info(1 pgs e194300:24.2dc) v4 ==== 748+0+0 (1765976911 0 0) 0x603d240 con 0x41d0f20 -39> 2014-08-12 17:43:21.591365 7f4c65145700 5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591016, event: header_read, request: pg_info(1 pgs e194300:24.2dc) v4 -38> 2014-08-12 17:43:21.591373 7f4c65145700 5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591019, event: throttled, request: pg_info(1 pgs e194300:24.2dc) v4 -37> 2014-08-12 17:43:21.591378 7f4c65145700 5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591104, event: all_read, request: pg_info(1 pgs e194300:24.2dc) v4 -36> 2014-08-12 17:43:21.591384 7f4c65145700 5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591363, event: dispatched, request: pg_info(1 pgs e194300:24.2dc) v4 -35> 2014-08-12 17:43:21.591390 7f4c65145700 5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591390, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.2dc) v4 -34> 2014-08-12 17:43:21.591397 7f4c65145700 5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591397, event: started, request: pg_info(1 pgs e194300:24.2dc) v4 -33> 2014-08-12 17:43:21.591416 7f4c65145700 5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591416, event: done, request: pg_info(1 pgs e194300:24.2dc) v4 -32> 2014-08-12 17:43:21.591430 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.290 192.168.1.109:6808/31607 17 ==== pg_info(1 pgs e194300:24.119) v4 ==== 748+0+0 (3829287638 0 0) 0x629dcc0 con 0x40c9760 -31> 2014-08-12 17:43:21.591445 7f4c65145700 5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591217, event: header_read, request: pg_info(1 pgs e194300:24.119) v4 -30> 2014-08-12 17:43:21.591451 7f4c65145700 5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591220, event: throttled, request: pg_info(1 pgs e194300:24.119) v4 -29> 2014-08-12 17:43:21.591457 7f4c65145700 5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591314, event: all_read, request: pg_info(1 pgs e194300:24.119) v4 -28> 2014-08-12 17:43:21.591462 7f4c65145700 5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591443, event: dispatched, request: pg_info(1 pgs e194300:24.119) v4 -27> 2014-08-12 17:43:21.591468 7f4c65145700 5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591468, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.119) v4 -26> 2014-08-12 17:43:21.591475 7f4c65145700 5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591475, event: started, request: pg_info(1 pgs e194300:24.119) v4 -25> 2014-08-12 17:43:21.591494 7f4c65145700 5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591494, event: done, request: pg_info(1 pgs e194300:24.119) v4 -24> 2014-08-12 17:43:21.591509 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.395 192.168.1.105:6810/41287 8 ==== pg_info(1 pgs e194300:24.354) v4 ==== 748+0+0 (4226165281 0 0) 0x6039500 con 0x40cf0c0 -23> 2014-08-12 17:43:21.591528 7f4c65145700 5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591413, event: header_read, request: pg_info(1 pgs e194300:24.354) v4 -22> 2014-08-12 17:43:21.591538 7f4c65145700 5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591416, event: throttled, request: pg_info(1 pgs e194300:24.354) v4 -21> 2014-08-12 17:43:21.591548 7f4c65145700 5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591482, event: all_read, request: pg_info(1 pgs e194300:24.354) v4 -20> 2014-08-12 17:43:21.591557 7f4c65145700 5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591526, event: dispatched, request: pg_info(1 pgs e194300:24.354) v4 -19> 2014-08-12 17:43:21.591566 7f4c65145700 5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591566, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.354) v4 -18> 2014-08-12 17:43:21.591577 7f4c65145700 5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591577, event: started, request: pg_info(1 pgs e194300:24.354) v4 -17> 2014-08-12 17:43:21.591604 7f4c65145700 5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591604, event: done, request: pg_info(1 pgs e194300:24.354) v4 -16> 2014-08-12 17:43:21.592053 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.308 192.168.1.113:6802/14403 12 ==== pg_info(1 pgs e194300:24.2dc) v4 ==== 748+0+0 (2691013303 0 0) 0x6039340 con 0x41d0f20 -15> 2014-08-12 17:43:21.592096 7f4c65145700 5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.591898, event: header_read, request: pg_info(1 pgs e194300:24.2dc) v4 -14> 2014-08-12 17:43:21.592112 7f4c65145700 5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.591901, event: throttled, request: pg_info(1 pgs e194300:24.2dc) v4 -13> 2014-08-12 17:43:21.592122 7f4c65145700 5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.591987, event: all_read, request: pg_info(1 pgs e194300:24.2dc) v4 -12> 2014-08-12 17:43:21.592131 7f4c65145700 5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.592090, event: dispatched, request: pg_info(1 pgs e194300:24.2dc) v4 -11> 2014-08-12 17:43:21.592142 7f4c65145700 5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.592142, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.2dc) v4 -10> 2014-08-12 17:43:21.592155 7f4c65145700 5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.592155, event: started, request: pg_info(1 pgs e194300:24.2dc) v4 -9> 2014-08-12 17:43:21.592191 7f4c65145700 5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.592191, event: done, request: pg_info(1 pgs e194300:24.2dc) v4 -8> 2014-08-12 17:43:21.592226 7f4c65145700 1 -- 192.168.1.101:6800/51417 <== osd.342 192.168.1.110:6802/41438 8 ==== pg_info(1 pgs e194300:24.36d) v4 ==== 748+0+0 (2521894262 0 0) 0x629bb80 con 0x41d3860 -7> 2014-08-12 17:43:21.592247 7f4c65145700 5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.591904, event: header_read, request: pg_info(1 pgs e194300:24.36d) v4 -6> 2014-08-12 17:43:21.592258 7f4c65145700 5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.591906, event: throttled, request: pg_info(1 pgs e194300:24.36d) v4 -5> 2014-08-12 17:43:21.592267 7f4c65145700 5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.591989, event: all_read, request: pg_info(1 pgs e194300:24.36d) v4 -4> 2014-08-12 17:43:21.592275 7f4c65145700 5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.592245, event: dispatched, request: pg_info(1 pgs e194300:24.36d) v4 -3> 2014-08-12 17:43:21.592285 7f4c65145700 5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.592285, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.36d) v4 -2> 2014-08-12 17:43:21.592296 7f4c65145700 5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.592296, event: started, request: pg_info(1 pgs e194300:24.36d) v4 -1> 2014-08-12 17:43:21.592322 7f4c65145700 5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.592322, event: done, request: pg_info(1 pgs e194300:24.36d) v4 0> 2014-08-12 17:43:21.593174 7f4c5d939700 -1 *** Caught signal (Floating point exception) ** in thread 7f4c5d939700 ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6) 1: /usr/bin/ceph-osd() [0x9ae061] 2: (()+0xf500) [0x7f4c77a51500] 3: (ReplicatedPG::agent_choose_mode(bool)+0xa6e) [0x84f57e] 4: (ReplicatedPG::agent_setup()+0x1d2) [0x866d72] 5: (ReplicatedPG::on_activate()+0x457) [0x8677a7] 6: (PG::RecoveryState::Active::react(PG::AllReplicasActivated const&)+0x73) [0x7dd173] 7: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::QueryState>, boost::statechart::custom_reaction<PG::ActMap>, boost::statechart::custom_reaction<PG::AdvMap>, boost::statechart::custom_reaction<PG::MInfoRec>, boost::statechart::custom_reaction<PG::MNotifyRec>, boost::statechart::custom_reaction<PG::MLogRec>, boost::statechart::custom_reaction<PG::Backfilled>, boost::statechart::custom_reaction<PG::AllReplicasActivated>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x36c) [0x844fec] 8: (boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x26) [0x845066] 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x4b) [0x83b4db] 10: (PG::handle_peering_event(std::tr1::shared_ptr<PG::CephPeeringEvt>, PG::RecoveryCtx*)+0x32f) [0x7f756f] 11: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x2f2) [0x6475b2] 12: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x16) [0x69e686] 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x551) [0xab8561] 14: (ThreadPool::WorkThread::entry()+0x10) [0xabb5a0] 15: (()+0x7851) [0x7f4c77a49851] 16: (clone()+0x6d) [0x7f4c769da94d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.306.log --- end dump of recent events ---
Files