Project

General

Profile

Actions
Copy link

Bug #9082

closed

Ceph Firefly 0.80.5 : PG has invalid (post-split) stats; must scrub before tier agent can activate

Added by karan singh over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Sage Weil
Category:
OSD
Target version:
0.85
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hello

Ceph version : 0.80.5
Centos 6.5
Features in use : erasure coding and cache tiering

Few hours back my cluster was working absolutely fine , all OSD's were up and running.

As soon as i started benchmarking my erasure-coded pool which is has a tired cache pool , my osds are started getting DOWN and OUT . Immediately 6 OSDS went down and all of them are throwing the same error. All of these osds which went down are from cache pool crush ruleset.

2014-08-12 17:43:21.586831 7f4c5e33a700 0 log [WRN] : pg 24.104 has invalid (post-split) stats; must scrub before tier agent can activate
2014-08-12 17:43:21.593174 7f4c5d939700 -1 ** Caught signal (Floating point exception) *

When i am manually trying to start these ODS , they are not coming up and throwing the same error message (mentioned above)

I have been reading Firefly release notes ( http://ceph.com/releases/v0-80-firefly-released/ ) , looks like it was fixed earlier
osd: warn if agent cannot enable due to invalid (post-split) stats (Sage Weil)

But in version 0.80.5 , i am still facing this. Is this something you already know ??

Below are the OSD detailed LOGS:

2014-08-12 17:43:21.584841 7f4c5d939700  0 log [WRN] : pg 24.30f has invalid (post-split) stats; must scrub before tier agent can activate
  -175> 2014-08-12 17:43:21.584995 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.6 192.168.1.107:6809/53141 13 ==== pg_info(1 pgs e194300:22.795) v4 ==== 714+0+0 (1092772418 0 0) 0x62ba300 con 0x40cc360
  -174> 2014-08-12 17:43:21.585030 7f4c65145700  5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.584855, event: header_read, request: pg_info(1 pgs e194300:22.795) v4
  -173> 2014-08-12 17:43:21.585044 7f4c65145700  5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.584859, event: throttled, request: pg_info(1 pgs e194300:22.795) v4
  -172> 2014-08-12 17:43:21.585055 7f4c65145700  5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.584962, event: all_read, request: pg_info(1 pgs e194300:22.795) v4
  -171> 2014-08-12 17:43:21.585064 7f4c65145700  5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.585024, event: dispatched, request: pg_info(1 pgs e194300:22.795) v4
  -170> 2014-08-12 17:43:21.585074 7f4c65145700  5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.585074, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:22.795) v4
  -169> 2014-08-12 17:43:21.585087 7f4c65145700  5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.585087, event: started, request: pg_info(1 pgs e194300:22.795) v4
  -168> 2014-08-12 17:43:21.585116 7f4c65145700  5 -- op tracker -- , seq: 647, time: 2014-08-12 17:43:21.585116, event: done, request: pg_info(1 pgs e194300:22.795) v4
  -167> 2014-08-12 17:43:21.585173 7f4c5e33a700  0 log [WRN] : pg 24.2a9 has invalid (post-split) stats; must scrub before tier agent can activate
  -166> 2014-08-12 17:43:21.585232 7f4c5d939700  0 log [WRN] : pg 24.254 has invalid (post-split) stats; must scrub before tier agent can activate
  -165> 2014-08-12 17:43:21.585226 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[24.2a9( v 194216'12832 (192834'10938,194216'12832] local-les=194300 n=4 ec=192688 les/c 194300/194235 194298/194299/194299) [306,24,61] r=0 lpr=194299 pi=194232-194298/5 crt=192888'12436 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.366973 12 0.000638
  -164> 2014-08-12 17:43:21.585259 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[24.2a9( v 194216'12832 (192834'10938,194216'12832] local-les=194300 n=4 ec=192688 les/c 194300/194235 194298/194299/194299) [306,24,61] r=0 lpr=194299 pi=194232-194298/5 crt=192888'12436 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered
  -163> 2014-08-12 17:43:21.585270 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.30f( v 194217'14634 (192718'11887,194217'14634] local-les=194300 n=5 ec=192688 les/c 194300/194235 194298/194299/194299) [306,308,62] r=0 lpr=194299 pi=194232-194298/5 crt=192888'14229 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.368068 12 0.000658
  -162> 2014-08-12 17:43:21.585288 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[24.2a9( v 194216'12832 (192834'10938,194216'12832] local-les=194300 n=4 ec=192688 les/c 194300/194235 194298/194299/194299) [306,24,61] r=0 lpr=194299 pi=194232-194298/5 crt=192888'12436 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000028 0 0.000000
  -161> 2014-08-12 17:43:21.585309 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.30f( v 194217'14634 (192718'11887,194217'14634] local-les=194300 n=5 ec=192688 les/c 194300/194235 194298/194299/194299) [306,308,62] r=0 lpr=194299 pi=194232-194298/5 crt=192888'14229 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered
  -160> 2014-08-12 17:43:21.585311 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[24.2a9( v 194216'12832 (192834'10938,194216'12832] local-les=194300 n=4 ec=192688 les/c 194300/194235 194298/194299/194299) [306,24,61] r=0 lpr=194299 pi=194232-194298/5 crt=192888'12436 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean
  -159> 2014-08-12 17:43:21.585339 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.30f( v 194217'14634 (192718'11887,194217'14634] local-les=194300 n=5 ec=192688 les/c 194300/194235 194298/194299/194299) [306,308,62] r=0 lpr=194299 pi=194232-194298/5 crt=192888'14229 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000031 0 0.000000
  -158> 2014-08-12 17:43:21.585363 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.30f( v 194217'14634 (192718'11887,194217'14634] local-les=194300 n=5 ec=192688 les/c 194300/194235 194298/194299/194299) [306,308,62] r=0 lpr=194299 pi=194232-194298/5 crt=192888'14229 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean
  -157> 2014-08-12 17:43:21.585368 7f4c5e33a700  1 -- 192.168.1.101:6800/51417 --> osd.24 192.168.1.107:6800/52669 -- pg_info(1 pgs e194300:24.2a9) v4 -- ?+0 0x6038c40
  -156> 2014-08-12 17:43:21.585370 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.6 192.168.1.107:6809/53141 14 ==== MRecoveryReserve REQUEST  pgid: 22.795, query_epoch: 194300 v2 ==== 26+0+0 (3761680802 0 0) 0x62b8540 con 0x40cc360
  -155> 2014-08-12 17:43:21.585406 7f4c65145700  5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585265, event: header_read, request: MRecoveryReserve REQUEST  pgid: 22.795, query_epoch: 194300 v2
  -154> 2014-08-12 17:43:21.585419 7f4c65145700  5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585268, event: throttled, request: MRecoveryReserve REQUEST  pgid: 22.795, query_epoch: 194300 v2
  -153> 2014-08-12 17:43:21.585430 7f4c65145700  5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585345, event: all_read, request: MRecoveryReserve REQUEST  pgid: 22.795, query_epoch: 194300 v2
  -152> 2014-08-12 17:43:21.585421 7f4c5d939700  1 -- 192.168.1.101:6800/51417 --> osd.62 192.168.1.112:6824/5505 -- pg_info(1 pgs e194300:24.30f) v4 -- ?+0 0x62baf40
  -151> 2014-08-12 17:43:21.585439 7f4c65145700  5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585402, event: dispatched, request: MRecoveryReserve REQUEST  pgid: 22.795, query_epoch: 194300 v2
  -150> 2014-08-12 17:43:21.585446 7f4c65145700  5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585446, event: waiting_for_osdmap, request: MRecoveryReserve REQUEST  pgid: 22.795, query_epoch: 194300 v2
  -149> 2014-08-12 17:43:21.585467 7f4c65145700  5 -- op tracker -- , seq: 648, time: 2014-08-12 17:43:21.585467, event: done, request: MRecoveryReserve REQUEST  pgid: 22.795, query_epoch: 194300 v2
  -148> 2014-08-12 17:43:21.585465 7f4c5d939700  1 -- 192.168.1.101:6800/51417 --> osd.308 192.168.1.113:6802/14403 -- pg_info(1 pgs e194300:24.30f) v4 -- ?+0 0x62babc0
  -147> 2014-08-12 17:43:21.585650 7f4c5e33a700  1 -- 192.168.1.101:6800/51417 --> osd.61 192.168.1.111:6840/16566 -- pg_info(1 pgs e194300:24.2a9) v4 -- ?+0 0x603b2c0
  -146> 2014-08-12 17:43:21.585680 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.254( v 194215'17471 (192803'14472,194215'17471] local-les=194300 n=-9 ec=192688 les/c 194300/194235 194298/194299/194299) [306,289,6] r=0 lpr=194299 pi=193810-194298/7 crt=192888'17092 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.368052 12 0.000598
  -145> 2014-08-12 17:43:21.585737 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.254( v 194215'17471 (192803'14472,194215'17471] local-les=194300 n=-9 ec=192688 les/c 194300/194235 194298/194299/194299) [306,289,6] r=0 lpr=194299 pi=193810-194298/7 crt=192888'17092 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered
  -144> 2014-08-12 17:43:21.585766 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.254( v 194215'17471 (192803'14472,194215'17471] local-les=194300 n=-9 ec=192688 les/c 194300/194235 194298/194299/194299) [306,289,6] r=0 lpr=194299 pi=193810-194298/7 crt=192888'17092 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000029 0 0.000000
  -143> 2014-08-12 17:43:21.585786 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.254( v 194215'17471 (192803'14472,194215'17471] local-les=194300 n=-9 ec=192688 les/c 194300/194235 194298/194299/194299) [306,289,6] r=0 lpr=194299 pi=193810-194298/7 crt=192888'17092 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean
  -142> 2014-08-12 17:43:21.585798 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.49 192.168.1.111:6824/11112 -- pg_info(1 pgs e194300:24.104) v4 -- ?+0 0x62bb9c0
  -141> 2014-08-12 17:43:21.585845 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.59 192.168.1.112:6822/5265 -- osd pg remove(epoch 194300; pg24.30f; ) v2 -- ?+0 0x6d840c0
  -140> 2014-08-12 17:43:21.585892 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.301 192.168.1.113:6814/16727 -- osd pg remove(epoch 194300; pg24.30f; ) v2 -- ?+0 0x6d83f00
  -139> 2014-08-12 17:43:21.585922 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.292 192.168.1.113:6812/16056 -- pg_info(1 pgs e194300:24.104) v4 -- ?+0 0x603f380
  -138> 2014-08-12 17:43:21.586096 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.379 192.168.1.108:6830/4861 -- osd pg remove(epoch 194300; pg24.30f; ) v2 -- ?+0 0x6d83d40
  -137> 2014-08-12 17:43:21.586156 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[22.795( v 194215'502 lc 0'0 (0'0,194215'502] local-les=194299 n=26 ec=183096 les/c 194299/194273 194298/194298/192967) [6,306,374] r=1 lpr=194299 pi=194270-194297/3 luod=0'0 crt=194215'502 active m=26] exit Started/ReplicaActive/RepNotRecovering 0.801323 7 0.000751
  -136> 2014-08-12 17:43:21.586175 7f4c5d939700  1 -- 192.168.1.101:6800/51417 --> osd.6 192.168.1.107:6809/53141 -- pg_info(1 pgs e194300:24.254) v4 -- ?+0 0x62bb800
  -135> 2014-08-12 17:43:21.586208 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[22.795( v 194215'502 lc 0'0 (0'0,194215'502] local-les=194299 n=26 ec=183096 les/c 194299/194273 194298/194298/192967) [6,306,374] r=1 lpr=194299 pi=194270-194297/3 luod=0'0 crt=194215'502 active m=26] enter Started/ReplicaActive/RepWaitRecoveryReserved
  -134> 2014-08-12 17:43:21.586229 7f4c5d939700  1 -- 192.168.1.101:6800/51417 --> osd.289 192.168.1.113:6808/15284 -- pg_info(1 pgs e194300:24.254) v4 -- ?+0 0x62ba140
  -133> 2014-08-12 17:43:21.586394 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.6 192.168.1.107:6809/53141 -- osd pg remove(epoch 194300; pg24.2a9; ) v2 -- ?+0 0x6d81880
  -132> 2014-08-12 17:43:21.586444 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.29 192.168.1.115:6806/59028 -- osd pg remove(epoch 194300; pg24.2a9; ) v2 -- ?+0 0x6d81c00
  -131> 2014-08-12 17:43:21.586474 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.47 192.168.1.111:6828/12102 -- osd pg remove(epoch 194300; pg24.2a9; ) v2 -- ?+0 0x62bdcc0
  -130> 2014-08-12 17:43:21.586625 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.18 192.168.1.107:6805/52903 -- osd pg remove(epoch 194300; pg24.254; ) v2 -- ?+0 0x62bbf00
  -129> 2014-08-12 17:43:21.586676 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.292 192.168.1.113:6812/16056 -- osd pg remove(epoch 194300; pg24.254; ) v2 -- ?+0 0x62bc980
  -128> 2014-08-12 17:43:21.586701 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.344 192.168.1.106:6812/6757 -- osd pg remove(epoch 194300; pg24.254; ) v2 -- ?+0 0x62b9880
  -127> 2014-08-12 17:43:21.586831 7f4c5e33a700  0 log [WRN] : pg 24.104 has invalid (post-split) stats; must scrub before tier agent can activate
  -126> 2014-08-12 17:43:21.586875 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[24.104( v 194215'15298 (192718'12373,194215'15298] local-les=194300 n=12 ec=192688 les/c 194300/194235 194298/194299/194299) [306,49,292] r=0 lpr=194299 pi=194232-194298/5 crt=192834'14593 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.368029 12 0.000767
  -125> 2014-08-12 17:43:21.586909 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[24.104( v 194215'15298 (192718'12373,194215'15298] local-les=194300 n=12 ec=192688 les/c 194300/194235 194298/194299/194299) [306,49,292] r=0 lpr=194299 pi=194232-194298/5 crt=192834'14593 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered
  -124> 2014-08-12 17:43:21.586935 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[24.104( v 194215'15298 (192718'12373,194215'15298] local-les=194300 n=12 ec=192688 les/c 194300/194235 194298/194299/194299) [306,49,292] r=0 lpr=194299 pi=194232-194298/5 crt=192834'14593 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000026 0 0.000000
  -123> 2014-08-12 17:43:21.586958 7f4c5e33a700  5 osd.306 pg_epoch: 194300 pg[24.104( v 194215'15298 (192718'12373,194215'15298] local-les=194300 n=12 ec=192688 les/c 194300/194235 194298/194299/194299) [306,49,292] r=0 lpr=194299 pi=194232-194298/5 crt=192834'14593 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean
  -122> 2014-08-12 17:43:21.587005 7f4c5e33a700  1 -- 192.168.1.101:6800/51417 --> osd.49 192.168.1.111:6824/11112 -- pg_info(1 pgs e194300:24.104) v4 -- ?+0 0x6d863c0
  -121> 2014-08-12 17:43:21.587043 7f4c5e33a700  1 -- 192.168.1.101:6800/51417 --> osd.292 192.168.1.113:6812/16056 -- pg_info(1 pgs e194300:24.104) v4 -- ?+0 0x6d86200
  -120> 2014-08-12 17:43:21.587364 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.344 192.168.1.106:6812/6757 -- pg_info(1 pgs e194300:24.58) v4 -- ?+0 0x603a680
  -119> 2014-08-12 17:43:21.587407 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.47 192.168.1.111:6828/12102 -- osd pg remove(epoch 194300; pg24.104; ) v2 -- ?+0 0x62bbb80
  -118> 2014-08-12 17:43:21.587444 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.352 192.168.1.108:6826/4273 -- pg_info(1 pgs e194300:24.58) v4 -- ?+0 0x603aa00
  -117> 2014-08-12 17:43:21.587478 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.295 192.168.1.113:6806/14945 -- osd pg remove(epoch 194300; pg24.104; ) v2 -- ?+0 0x62baa00
  -116> 2014-08-12 17:43:21.587510 7f4c69567700  1 -- 192.168.1.101:6800/51417 --> osd.397 192.168.1.105:6808/41085 -- osd pg remove(epoch 194300; pg24.104; ) v2 -- ?+0 0x62ba4c0
  -115> 2014-08-12 17:43:21.587533 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.58( v 194217'14171 (192718'11167,194217'14171] local-les=194300 n=-1 ec=192688 les/c 194300/193814 194298/194299/194299) [306,352,344]/[306] r=0 lpr=194299 pi=193813-194298/13 bft=344,352 crt=0'0 lcod 0'0 mlcod 0'0 active+degraded+remapped] on activate: bft=344,352 from 0//0//-1
  -114> 2014-08-12 17:43:21.587566 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.58( v 194217'14171 (192718'11167,194217'14171] local-les=194300 n=-1 ec=192688 les/c 194300/193814 194298/194299/194299) [306,352,344]/[306] r=0 lpr=194299 pi=193813-194298/13 bft=344,352 crt=0'0 lcod 0'0 mlcod 0'0 active+degraded+remapped] target shard 344 from 0//0//-1
  -113> 2014-08-12 17:43:21.587584 7f4c5d939700  5 osd.306 pg_epoch: 194300 pg[24.58( v 194217'14171 (192718'11167,194217'14171] local-les=194300 n=-1 ec=192688 les/c 194300/193814 194298/194299/194299) [306,352,344]/[306] r=0 lpr=194299 pi=193813-194298/13 bft=344,352 crt=0'0 lcod 0'0 mlcod 0'0 active+degraded+remapped] target shard 352 from 0//0//-1
  -112> 2014-08-12 17:43:21.587939 7f4c5d939700  0 log [WRN] : pg 24.58 has invalid (post-split) stats; must scrub before tier agent can activate
  -111> 2014-08-12 17:43:21.588814 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.302 192.168.1.113:6810/15731 -- pg_info(1 pgs e194300:24.3f1) v4 -- ?+0 0x603b640
  -110> 2014-08-12 17:43:21.588955 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.290 192.168.1.109:6808/31607 -- pg_info(1 pgs e194300:24.119) v4 -- ?+0 0x6039a40
  -109> 2014-08-12 17:43:21.589098 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.18 192.168.1.107:6805/52903 -- pg_info(1 pgs e194300:24.1c4) v4 -- ?+0 0x603db00
  -108> 2014-08-12 17:43:21.589383 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.313 192.168.1.109:6820/35445 -- pg_info(1 pgs e194300:24.13b) v4 -- ?+0 0x6038540
  -107> 2014-08-12 17:43:21.589673 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.342 192.168.1.110:6802/41438 -- pg_info(1 pgs e194300:24.36d) v4 -- ?+0 0x603d240
  -106> 2014-08-12 17:43:21.589818 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.18 192.168.1.107:6805/52903 13 ==== pg_info(1 pgs e194300:24.1c4) v4 ==== 748+0+0 (2739292217 0 0) 0x603c280 con 0x40cdac0
  -105> 2014-08-12 17:43:21.589857 7f4c65145700  5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589678, event: header_read, request: pg_info(1 pgs e194300:24.1c4) v4
  -104> 2014-08-12 17:43:21.589872 7f4c65145700  5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589680, event: throttled, request: pg_info(1 pgs e194300:24.1c4) v4
  -103> 2014-08-12 17:43:21.589881 7f4c65145700  5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589760, event: all_read, request: pg_info(1 pgs e194300:24.1c4) v4
  -102> 2014-08-12 17:43:21.589888 7f4c65145700  5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589852, event: dispatched, request: pg_info(1 pgs e194300:24.1c4) v4
  -101> 2014-08-12 17:43:21.589898 7f4c65145700  5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589898, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.1c4) v4
  -100> 2014-08-12 17:43:21.589912 7f4c65145700  5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589911, event: started, request: pg_info(1 pgs e194300:24.1c4) v4
   -99> 2014-08-12 17:43:21.589972 7f4c65145700  5 -- op tracker -- , seq: 649, time: 2014-08-12 17:43:21.589972, event: done, request: pg_info(1 pgs e194300:24.1c4) v4
   -98> 2014-08-12 17:43:21.589998 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.302 192.168.1.113:6810/15731 12 ==== pg_info(1 pgs e194300:24.3f1) v4 ==== 744+0+0 (1504649001 0 0) 0x603b640 con 0x48baec0
   -97> 2014-08-12 17:43:21.590017 7f4c65145700  5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.589838, event: header_read, request: pg_info(1 pgs e194300:24.3f1) v4
   -96> 2014-08-12 17:43:21.590028 7f4c65145700  5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.589842, event: throttled, request: pg_info(1 pgs e194300:24.3f1) v4
   -95> 2014-08-12 17:43:21.590037 7f4c65145700  5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.589945, event: all_read, request: pg_info(1 pgs e194300:24.3f1) v4
   -94> 2014-08-12 17:43:21.590046 7f4c65145700  5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.590015, event: dispatched, request: pg_info(1 pgs e194300:24.3f1) v4
   -93> 2014-08-12 17:43:21.590056 7f4c65145700  5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.590055, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.3f1) v4
   -92> 2014-08-12 17:43:21.590067 7f4c65145700  5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.590066, event: started, request: pg_info(1 pgs e194300:24.3f1) v4
   -91> 2014-08-12 17:43:21.590095 7f4c65145700  5 -- op tracker -- , seq: 650, time: 2014-08-12 17:43:21.590094, event: done, request: pg_info(1 pgs e194300:24.3f1) v4
   -90> 2014-08-12 17:43:21.590113 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.290 192.168.1.109:6808/31607 16 ==== pg_info(1 pgs e194300:24.119) v4 ==== 748+0+0 (847792910 0 0) 0x6039a40 con 0x40c9760
   -89> 2014-08-12 17:43:21.590110 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.395 192.168.1.105:6810/41287 -- pg_info(1 pgs e194300:24.354) v4 -- ?+0 0x6039880
   -88> 2014-08-12 17:43:21.590131 7f4c65145700  5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.589950, event: header_read, request: pg_info(1 pgs e194300:24.119) v4
   -87> 2014-08-12 17:43:21.590141 7f4c65145700  5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.589953, event: throttled, request: pg_info(1 pgs e194300:24.119) v4
   -86> 2014-08-12 17:43:21.590148 7f4c65145700  5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590035, event: all_read, request: pg_info(1 pgs e194300:24.119) v4
   -85> 2014-08-12 17:43:21.590154 7f4c65145700  5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590129, event: dispatched, request: pg_info(1 pgs e194300:24.119) v4
   -84> 2014-08-12 17:43:21.590160 7f4c65145700  5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590159, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.119) v4
   -83> 2014-08-12 17:43:21.590167 7f4c65145700  5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590167, event: started, request: pg_info(1 pgs e194300:24.119) v4
   -82> 2014-08-12 17:43:21.590186 7f4c65145700  5 -- op tracker -- , seq: 651, time: 2014-08-12 17:43:21.590186, event: done, request: pg_info(1 pgs e194300:24.119) v4
   -81> 2014-08-12 17:43:21.590199 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.18 192.168.1.107:6805/52903 14 ==== pg_info(1 pgs e194300:24.1c4) v4 ==== 748+0+0 (660960379 0 0) 0x603db00 con 0x40cdac0
   -80> 2014-08-12 17:43:21.590213 7f4c65145700  5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.589972, event: header_read, request: pg_info(1 pgs e194300:24.1c4) v4
   -79> 2014-08-12 17:43:21.590219 7f4c65145700  5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.589974, event: throttled, request: pg_info(1 pgs e194300:24.1c4) v4
   -78> 2014-08-12 17:43:21.590225 7f4c65145700  5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590060, event: all_read, request: pg_info(1 pgs e194300:24.1c4) v4
   -77> 2014-08-12 17:43:21.590230 7f4c65145700  5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590211, event: dispatched, request: pg_info(1 pgs e194300:24.1c4) v4
   -76> 2014-08-12 17:43:21.590240 7f4c65145700  5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590240, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.1c4) v4
   -75> 2014-08-12 17:43:21.590253 7f4c65145700  5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590252, event: started, request: pg_info(1 pgs e194300:24.1c4) v4
   -74> 2014-08-12 17:43:21.590266 7f4c69f68700  1 -- 192.168.1.101:6800/51417 --> osd.308 192.168.1.113:6802/14403 -- pg_info(1 pgs e194300:24.2dc) v4 -- ?+0 0x603c440
   -73> 2014-08-12 17:43:21.590274 7f4c65145700  5 -- op tracker -- , seq: 652, time: 2014-08-12 17:43:21.590274, event: done, request: pg_info(1 pgs e194300:24.1c4) v4
   -72> 2014-08-12 17:43:21.590293 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.313 192.168.1.109:6820/35445 11 ==== pg_info(1 pgs e194300:24.13b) v4 ==== 748+0+0 (1395045469 0 0) 0x6038540 con 0x40ca680
   -71> 2014-08-12 17:43:21.590308 7f4c65145700  5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590201, event: header_read, request: pg_info(1 pgs e194300:24.13b) v4
   -70> 2014-08-12 17:43:21.590315 7f4c65145700  5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590203, event: throttled, request: pg_info(1 pgs e194300:24.13b) v4
   -69> 2014-08-12 17:43:21.590320 7f4c65145700  5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590275, event: all_read, request: pg_info(1 pgs e194300:24.13b) v4
   -68> 2014-08-12 17:43:21.590326 7f4c65145700  5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590306, event: dispatched, request: pg_info(1 pgs e194300:24.13b) v4
   -67> 2014-08-12 17:43:21.590332 7f4c65145700  5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590331, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.13b) v4
   -66> 2014-08-12 17:43:21.590343 7f4c65145700  5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590343, event: started, request: pg_info(1 pgs e194300:24.13b) v4
   -65> 2014-08-12 17:43:21.590366 7f4c65145700  5 -- op tracker -- , seq: 653, time: 2014-08-12 17:43:21.590366, event: done, request: pg_info(1 pgs e194300:24.13b) v4
   -64> 2014-08-12 17:43:21.590795 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.342 192.168.1.110:6802/41438 7 ==== pg_info(1 pgs e194300:24.36d) v4 ==== 748+0+0 (330703328 0 0) 0x2aec0c0 con 0x41d3860
   -63> 2014-08-12 17:43:21.590820 7f4c65145700  5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590687, event: header_read, request: pg_info(1 pgs e194300:24.36d) v4
   -62> 2014-08-12 17:43:21.590832 7f4c65145700  5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590689, event: throttled, request: pg_info(1 pgs e194300:24.36d) v4
   -61> 2014-08-12 17:43:21.590841 7f4c65145700  5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590773, event: all_read, request: pg_info(1 pgs e194300:24.36d) v4
   -60> 2014-08-12 17:43:21.590850 7f4c65145700  5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590817, event: dispatched, request: pg_info(1 pgs e194300:24.36d) v4
   -59> 2014-08-12 17:43:21.590859 7f4c65145700  5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590859, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.36d) v4
   -58> 2014-08-12 17:43:21.590871 7f4c65145700  5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590871, event: started, request: pg_info(1 pgs e194300:24.36d) v4
   -57> 2014-08-12 17:43:21.590900 7f4c65145700  5 -- op tracker -- , seq: 654, time: 2014-08-12 17:43:21.590900, event: done, request: pg_info(1 pgs e194300:24.36d) v4
   -56> 2014-08-12 17:43:21.591093 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.313 192.168.1.109:6820/35445 12 ==== pg_info(1 pgs e194300:24.13b) v4 ==== 748+0+0 (2101955393 0 0) 0x6d82840 con 0x40ca680
   -55> 2014-08-12 17:43:21.591136 7f4c65145700  5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.590938, event: header_read, request: pg_info(1 pgs e194300:24.13b) v4
   -54> 2014-08-12 17:43:21.591152 7f4c65145700  5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.590940, event: throttled, request: pg_info(1 pgs e194300:24.13b) v4
   -53> 2014-08-12 17:43:21.591162 7f4c65145700  5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591008, event: all_read, request: pg_info(1 pgs e194300:24.13b) v4
   -52> 2014-08-12 17:43:21.591171 7f4c65145700  5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591130, event: dispatched, request: pg_info(1 pgs e194300:24.13b) v4
   -51> 2014-08-12 17:43:21.591187 7f4c65145700  5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591187, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.13b) v4
   -50> 2014-08-12 17:43:21.591200 7f4c65145700  5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591200, event: started, request: pg_info(1 pgs e194300:24.13b) v4
   -49> 2014-08-12 17:43:21.591232 7f4c65145700  5 -- op tracker -- , seq: 655, time: 2014-08-12 17:43:21.591231, event: done, request: pg_info(1 pgs e194300:24.13b) v4
   -48> 2014-08-12 17:43:21.591256 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.395 192.168.1.105:6810/41287 7 ==== pg_info(1 pgs e194300:24.354) v4 ==== 748+0+0 (4289877745 0 0) 0x6039880 con 0x40cf0c0
   -47> 2014-08-12 17:43:21.591273 7f4c65145700  5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.590969, event: header_read, request: pg_info(1 pgs e194300:24.354) v4
   -46> 2014-08-12 17:43:21.591280 7f4c65145700  5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.590972, event: throttled, request: pg_info(1 pgs e194300:24.354) v4
   -45> 2014-08-12 17:43:21.591285 7f4c65145700  5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591052, event: all_read, request: pg_info(1 pgs e194300:24.354) v4
   -44> 2014-08-12 17:43:21.591291 7f4c65145700  5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591270, event: dispatched, request: pg_info(1 pgs e194300:24.354) v4
   -43> 2014-08-12 17:43:21.591297 7f4c65145700  5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591296, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.354) v4
   -42> 2014-08-12 17:43:21.591304 7f4c65145700  5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591304, event: started, request: pg_info(1 pgs e194300:24.354) v4
   -41> 2014-08-12 17:43:21.591324 7f4c65145700  5 -- op tracker -- , seq: 656, time: 2014-08-12 17:43:21.591324, event: done, request: pg_info(1 pgs e194300:24.354) v4
   -40> 2014-08-12 17:43:21.591344 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.308 192.168.1.113:6802/14403 11 ==== pg_info(1 pgs e194300:24.2dc) v4 ==== 748+0+0 (1765976911 0 0) 0x603d240 con 0x41d0f20
   -39> 2014-08-12 17:43:21.591365 7f4c65145700  5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591016, event: header_read, request: pg_info(1 pgs e194300:24.2dc) v4
   -38> 2014-08-12 17:43:21.591373 7f4c65145700  5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591019, event: throttled, request: pg_info(1 pgs e194300:24.2dc) v4
   -37> 2014-08-12 17:43:21.591378 7f4c65145700  5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591104, event: all_read, request: pg_info(1 pgs e194300:24.2dc) v4
   -36> 2014-08-12 17:43:21.591384 7f4c65145700  5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591363, event: dispatched, request: pg_info(1 pgs e194300:24.2dc) v4
   -35> 2014-08-12 17:43:21.591390 7f4c65145700  5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591390, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.2dc) v4
   -34> 2014-08-12 17:43:21.591397 7f4c65145700  5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591397, event: started, request: pg_info(1 pgs e194300:24.2dc) v4
   -33> 2014-08-12 17:43:21.591416 7f4c65145700  5 -- op tracker -- , seq: 657, time: 2014-08-12 17:43:21.591416, event: done, request: pg_info(1 pgs e194300:24.2dc) v4
   -32> 2014-08-12 17:43:21.591430 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.290 192.168.1.109:6808/31607 17 ==== pg_info(1 pgs e194300:24.119) v4 ==== 748+0+0 (3829287638 0 0) 0x629dcc0 con 0x40c9760
   -31> 2014-08-12 17:43:21.591445 7f4c65145700  5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591217, event: header_read, request: pg_info(1 pgs e194300:24.119) v4
   -30> 2014-08-12 17:43:21.591451 7f4c65145700  5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591220, event: throttled, request: pg_info(1 pgs e194300:24.119) v4
   -29> 2014-08-12 17:43:21.591457 7f4c65145700  5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591314, event: all_read, request: pg_info(1 pgs e194300:24.119) v4
   -28> 2014-08-12 17:43:21.591462 7f4c65145700  5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591443, event: dispatched, request: pg_info(1 pgs e194300:24.119) v4
   -27> 2014-08-12 17:43:21.591468 7f4c65145700  5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591468, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.119) v4
   -26> 2014-08-12 17:43:21.591475 7f4c65145700  5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591475, event: started, request: pg_info(1 pgs e194300:24.119) v4
   -25> 2014-08-12 17:43:21.591494 7f4c65145700  5 -- op tracker -- , seq: 658, time: 2014-08-12 17:43:21.591494, event: done, request: pg_info(1 pgs e194300:24.119) v4
   -24> 2014-08-12 17:43:21.591509 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.395 192.168.1.105:6810/41287 8 ==== pg_info(1 pgs e194300:24.354) v4 ==== 748+0+0 (4226165281 0 0) 0x6039500 con 0x40cf0c0
   -23> 2014-08-12 17:43:21.591528 7f4c65145700  5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591413, event: header_read, request: pg_info(1 pgs e194300:24.354) v4
   -22> 2014-08-12 17:43:21.591538 7f4c65145700  5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591416, event: throttled, request: pg_info(1 pgs e194300:24.354) v4
   -21> 2014-08-12 17:43:21.591548 7f4c65145700  5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591482, event: all_read, request: pg_info(1 pgs e194300:24.354) v4
   -20> 2014-08-12 17:43:21.591557 7f4c65145700  5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591526, event: dispatched, request: pg_info(1 pgs e194300:24.354) v4
   -19> 2014-08-12 17:43:21.591566 7f4c65145700  5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591566, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.354) v4
   -18> 2014-08-12 17:43:21.591577 7f4c65145700  5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591577, event: started, request: pg_info(1 pgs e194300:24.354) v4
   -17> 2014-08-12 17:43:21.591604 7f4c65145700  5 -- op tracker -- , seq: 659, time: 2014-08-12 17:43:21.591604, event: done, request: pg_info(1 pgs e194300:24.354) v4
   -16> 2014-08-12 17:43:21.592053 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.308 192.168.1.113:6802/14403 12 ==== pg_info(1 pgs e194300:24.2dc) v4 ==== 748+0+0 (2691013303 0 0) 0x6039340 con 0x41d0f20
   -15> 2014-08-12 17:43:21.592096 7f4c65145700  5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.591898, event: header_read, request: pg_info(1 pgs e194300:24.2dc) v4
   -14> 2014-08-12 17:43:21.592112 7f4c65145700  5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.591901, event: throttled, request: pg_info(1 pgs e194300:24.2dc) v4
   -13> 2014-08-12 17:43:21.592122 7f4c65145700  5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.591987, event: all_read, request: pg_info(1 pgs e194300:24.2dc) v4
   -12> 2014-08-12 17:43:21.592131 7f4c65145700  5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.592090, event: dispatched, request: pg_info(1 pgs e194300:24.2dc) v4
   -11> 2014-08-12 17:43:21.592142 7f4c65145700  5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.592142, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.2dc) v4
   -10> 2014-08-12 17:43:21.592155 7f4c65145700  5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.592155, event: started, request: pg_info(1 pgs e194300:24.2dc) v4
    -9> 2014-08-12 17:43:21.592191 7f4c65145700  5 -- op tracker -- , seq: 660, time: 2014-08-12 17:43:21.592191, event: done, request: pg_info(1 pgs e194300:24.2dc) v4
    -8> 2014-08-12 17:43:21.592226 7f4c65145700  1 -- 192.168.1.101:6800/51417 <== osd.342 192.168.1.110:6802/41438 8 ==== pg_info(1 pgs e194300:24.36d) v4 ==== 748+0+0 (2521894262 0 0) 0x629bb80 con 0x41d3860
    -7> 2014-08-12 17:43:21.592247 7f4c65145700  5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.591904, event: header_read, request: pg_info(1 pgs e194300:24.36d) v4
    -6> 2014-08-12 17:43:21.592258 7f4c65145700  5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.591906, event: throttled, request: pg_info(1 pgs e194300:24.36d) v4
    -5> 2014-08-12 17:43:21.592267 7f4c65145700  5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.591989, event: all_read, request: pg_info(1 pgs e194300:24.36d) v4
    -4> 2014-08-12 17:43:21.592275 7f4c65145700  5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.592245, event: dispatched, request: pg_info(1 pgs e194300:24.36d) v4
    -3> 2014-08-12 17:43:21.592285 7f4c65145700  5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.592285, event: waiting_for_osdmap, request: pg_info(1 pgs e194300:24.36d) v4
    -2> 2014-08-12 17:43:21.592296 7f4c65145700  5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.592296, event: started, request: pg_info(1 pgs e194300:24.36d) v4
    -1> 2014-08-12 17:43:21.592322 7f4c65145700  5 -- op tracker -- , seq: 661, time: 2014-08-12 17:43:21.592322, event: done, request: pg_info(1 pgs e194300:24.36d) v4
     0> 2014-08-12 17:43:21.593174 7f4c5d939700 -1 *** Caught signal (Floating point exception) **
 in thread 7f4c5d939700

 ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
 1: /usr/bin/ceph-osd() [0x9ae061]
 2: (()+0xf500) [0x7f4c77a51500]
 3: (ReplicatedPG::agent_choose_mode(bool)+0xa6e) [0x84f57e]
 4: (ReplicatedPG::agent_setup()+0x1d2) [0x866d72]
 5: (ReplicatedPG::on_activate()+0x457) [0x8677a7]
 6: (PG::RecoveryState::Active::react(PG::AllReplicasActivated const&)+0x73) [0x7dd173]
 7: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::QueryState>, boost::statechart::custom_reaction<PG::ActMap>, boost::statechart::custom_reaction<PG::AdvMap>, boost::statechart::custom_reaction<PG::MInfoRec>, boost::statechart::custom_reaction<PG::MNotifyRec>, boost::statechart::custom_reaction<PG::MLogRec>, boost::statechart::custom_reaction<PG::Backfilled>, boost::statechart::custom_reaction<PG::AllReplicasActivated>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x36c) [0x844fec]
 8: (boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x26) [0x845066]
 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x4b) [0x83b4db]
 10: (PG::handle_peering_event(std::tr1::shared_ptr<PG::CephPeeringEvt>, PG::RecoveryCtx*)+0x32f) [0x7f756f]
 11: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x2f2) [0x6475b2]
 12: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x16) [0x69e686]
 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x551) [0xab8561]
 14: (ThreadPool::WorkThread::entry()+0x10) [0xabb5a0]
 15: (()+0x7851) [0x7f4c77a49851]
 16: (clone()+0x6d) [0x7f4c769da94d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.306.log
--- end dump of recent events ---

Files

Download all files
ceph-osd.4.log (42.1 MB) ceph-osd.4.log karan singh, 08/13/2014 08:24 AM
ceph-osd.325.log.zip (6.43 MB) ceph-osd.325.log.zip karan singh, 08/13/2014 08:24 AM
Actions
Copy link

Also available in: Atom PDF