osd.1.log - Ceph - Ceph

Bug #9808 » osd.1.log

tail of the osd.1 log - Loïc Dachary, 10/17/2014 04:13 PM

       ...
 -10-17 16:05:53.764430 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 10.002768 1 0.000237
 -10-17 16:05:53.764483 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] enter Started/ReplicaActive/RepWaitBackfillReserved
 -10-17 16:05:53.764540 7f0aac930700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:53.764548 7f0aac930700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REJECT  pgid: 0.3, query_epoch: 16 v3 -- ?+0 0x518eb40 con 0x4b01b80
 -10-17 16:05:53.764594 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepWaitBackfillReserved 0.000110 0 0.000000
 -10-17 16:05:53.764643 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] enter Started/ReplicaActive/RepNotRecovering
 -10-17 16:05:53.764678 7f0aac930700 10 log is not dirty
 -10-17 16:05:53.797762 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfill
 -10-17 16:05:53.797871 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/NotBackfilling 10.000411 1 0.000064
 -10-17 16:05:53.797928 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/WaitLocalBackfillReserved
 -10-17 16:05:53.798025 7f0aac0df700 10 log is not dirty
 -10-17 16:05:53.798272 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 LocalBackfillReserved
 -10-17 16:05:53.798362 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 0.000434 1 0.000161
 -10-17 16:05:53.798411 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved
 -10-17 16:05:53.798474 7f0aac930700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REQUEST  pgid: 0.2, query_epoch: 16, prio: 201 v3 -- ?+0 0x5191de0 con 0x4b01b80
 -10-17 16:05:53.798552 7f0aac930700 10 log is not dirty
 -10-17 16:05:53.800388 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 151 ==== MBackfillReserve REJECT  pgid: 0.2, query_epoch: 16 v3 ==== 30+0+0 (3553360764 0 0) 0x5191de0 con 0x4b01b80
 -10-17 16:05:53.800445 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:53.800451 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:53.800454 7f0ab45ef700 20 osd.1 16 _dispatch 0x5191de0 MBackfillReserve REJECT  pgid: 0.2, query_epoch: 16 v3
 -10-17 16:05:53.800513 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x5191de0
 -10-17 16:05:53.800611 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:53.800615 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:53.800635 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RemoteReservationRejected
 -10-17 16:05:53.800751 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.002340 1 0.000196
 -10-17 16:05:53.800861 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling
 -10-17 16:05:53.800904 7f0aac0df700 10 log is not dirty
 -10-17 16:05:54.131861 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 152 ==== MBackfillReserve REQUEST  pgid: 0.5, query_epoch: 16, prio: 201 v3 ==== 30+0+0 (56951451 0 0) 0x518eb40 con 0x4b01b80
 -10-17 16:05:54.131927 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:54.131933 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:54.131936 7f0ab45ef700 20 osd.1 16 _dispatch 0x518eb40 MBackfillReserve REQUEST  pgid: 0.5, query_epoch: 16, prio: 201 v3
 -10-17 16:05:54.132002 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x518eb40
 -10-17 16:05:54.132111 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:54.132115 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:54.132136 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfillPrio: priority 201
 -10-17 16:05:54.132225 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] state<Started/ReplicaActive/RepNotRecovering>: backfill reservation rejected: full ratio is 0.896762, which is greater than max allowed ratio 0.8
 -10-17 16:05:54.132303 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 10.002531 1 0.000144
 -10-17 16:05:54.132355 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] enter Started/ReplicaActive/RepWaitBackfillReserved
 -10-17 16:05:54.132419 7f0aac930700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:54.132435 7f0aac930700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REJECT  pgid: 0.5, query_epoch: 16 v3 -- ?+0 0x518e780 con 0x4b01b80
 -10-17 16:05:54.132479 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepWaitBackfillReserved 0.000124 0 0.000000
 -10-17 16:05:54.132538 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] enter Started/ReplicaActive/RepNotRecovering
 -10-17 16:05:54.132574 7f0aac930700 10 log is not dirty
 -10-17 16:05:54.157738 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfill
 -10-17 16:05:54.157846 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/NotBackfilling 10.000368 1 0.000141
 -10-17 16:05:54.157906 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/WaitLocalBackfillReserved
 -10-17 16:05:54.158003 7f0aac0df700 10 log is not dirty
 -10-17 16:05:54.158241 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 LocalBackfillReserved
 -10-17 16:05:54.158328 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 0.000422 1 0.000164
 -10-17 16:05:54.158378 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved
 -10-17 16:05:54.158440 7f0aac930700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REQUEST  pgid: 0.6, query_epoch: 16, prio: 201 v3 -- ?+0 0x5191c00 con 0x4b01b80
 -10-17 16:05:54.158495 7f0aac930700 10 log is not dirty
 -10-17 16:05:54.160344 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 153 ==== MBackfillReserve REJECT  pgid: 0.6, query_epoch: 16 v3 ==== 30+0+0 (3890351304 0 0) 0x5191c00 con 0x4b01b80
 -10-17 16:05:54.160413 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:54.160419 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:54.160422 7f0ab45ef700 20 osd.1 16 _dispatch 0x5191c00 MBackfillReserve REJECT  pgid: 0.6, query_epoch: 16 v3
 -10-17 16:05:54.160481 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x5191c00
 -10-17 16:05:54.160587 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:54.160591 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:54.160607 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RemoteReservationRejected
 -10-17 16:05:54.160706 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.002329 1 0.000173
 -10-17 16:05:54.160767 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling
 -10-17 16:05:54.160846 7f0aac0df700 10 log is not dirty
 -10-17 16:05:54.234385 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:05:54.234439 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:51.788971 last_rx_back 2014-10-17 16:05:51.788971 last_rx_front 2014-10-17 16:05:51.788971
 -10-17 16:05:54.234486 7f0abd1ef700 20 osd.1 16 scrub_random_backoff lost coin flip, randomly backing off
 -10-17 16:05:54.234495 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:54.234499 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:54.328097 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfill
 -10-17 16:05:54.328207 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/NotBackfilling 10.000372 1 0.000148
 -10-17 16:05:54.328264 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/WaitLocalBackfillReserved
 -10-17 16:05:54.328358 7f0aac930700 10 log is not dirty
 -10-17 16:05:54.328579 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 LocalBackfillReserved
 -10-17 16:05:54.328649 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 0.000385 1 0.000162
 -10-17 16:05:54.328694 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved
 -10-17 16:05:54.328753 7f0aac0df700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REQUEST  pgid: 0.7, query_epoch: 16, prio: 201 v3 -- ?+0 0x4c67480 con 0x4b01b80
 -10-17 16:05:54.328900 7f0aac0df700 10 log is not dirty
 -10-17 16:05:54.330442 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 154 ==== MBackfillReserve REJECT  pgid: 0.7, query_epoch: 16 v3 ==== 30+0+0 (3941101861 0 0) 0x4c67480 con 0x4b01b80
 -10-17 16:05:54.330497 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:54.330502 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:54.330505 7f0ab45ef700 20 osd.1 16 _dispatch 0x4c67480 MBackfillReserve REJECT  pgid: 0.7, query_epoch: 16 v3
 -10-17 16:05:54.330575 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x4c67480
 -10-17 16:05:54.330670 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:54.330674 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:54.330691 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RemoteReservationRejected
 -10-17 16:05:54.330797 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.002105 1 0.000253
 -10-17 16:05:54.330855 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling
 -10-17 16:05:54.330897 7f0aac930700 10 log is not dirty
 -10-17 16:05:55.234652 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:05:55.234691 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:51.788971 last_rx_back 2014-10-17 16:05:51.788971 last_rx_front 2014-10-17 16:05:51.788971
 -10-17 16:05:55.234710 7f0abd1ef700 20 osd.1 16 scrub_random_backoff lost coin flip, randomly backing off
 -10-17 16:05:55.234723 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:55.234725 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:55.359667 7f0abb8fc700 20 filestore(/home/loic/software/ceph/ceph/src/dev/osd1) sync_entry woke after 5.000135
 -10-17 16:05:55.359692 7f0abb8fc700 10 journal commit_start max_applied_seq 145, open_ops 0
 -10-17 16:05:55.359703 7f0abb8fc700 10 journal commit_start blocked, all open_ops have completed
 -10-17 16:05:55.359704 7f0abb8fc700 10 journal commit_start nothing to do
 -10-17 16:05:55.359706 7f0abb8fc700 10 journal commit_start
 -10-17 16:05:55.359712 7f0abb8fc700 20 filestore(/home/loic/software/ceph/ceph/src/dev/osd1) sync_entry waiting for max_interval 5.000000
 -10-17 16:05:56.165967 7f0aad9d2700 10 monclient: tick
 -10-17 16:05:56.166017 7f0aad9d2700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-10-17 16:05:26.166013)
 -10-17 16:05:56.166039 7f0aad9d2700 10 monclient: renew subs? (now: 2014-10-17 16:05:56.166038; renew after: 2014-10-17 16:06:56.163640) -- no
 -10-17 16:05:56.234903 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:05:56.234957 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:51.788971 last_rx_back 2014-10-17 16:05:51.788971 last_rx_front 2014-10-17 16:05:51.788971
 -10-17 16:05:56.235005 7f0abd1ef700 20 osd.1 16 scrub_random_backoff lost coin flip, randomly backing off
 -10-17 16:05:56.235014 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:56.235018 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:56.884215 7f0ab0367700  1 -- 127.0.0.1:6806/2360 <== osd.2 127.0.0.1:0/2640 75 ==== osd_ping(ping e16 stamp 2014-10-17 16:05:56.883722) v2 ==== 47+0+0 (46135967 0 0) 0x5191480 con 0x4b01080
 -10-17 16:05:56.884204 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 <== osd.2 127.0.0.1:0/2640 75 ==== osd_ping(ping e16 stamp 2014-10-17 16:05:56.883722) v2 ==== 47+0+0 (46135967 0 0) 0x496f2c0 con 0x4b014a0
 -10-17 16:05:56.884321 7f0ab0367700  1 -- 127.0.0.1:6806/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:05:56.883722) v2 -- ?+0 0x5190760 con 0x4b01080
 -10-17 16:05:56.884380 7f0ab0367700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:56.884456 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:05:56.883722) v2 -- ?+0 0x496e5a0 con 0x4b014a0
 -10-17 16:05:56.884575 7f0ab1c5a700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:57.089376 7f0aa4c71700 20 osd.1 16 update_osd_stat osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:05:57.089429 7f0aa4c71700  5 osd.1 16 heartbeat: osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:05:57.089460 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6810/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:05:57.089446) v2 -- ?+0 0x4c67de0 con 0x4f8ec00
 -10-17 16:05:57.089532 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6812/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:05:57.089446) v2 -- ?+0 0x4c672a0 con 0x4f8eec0
 -10-17 16:05:57.089561 7f0aa4c71700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:57.089446 last_rx_back 2014-10-17 16:05:51.788971 last_rx_front 2014-10-17 16:05:51.788971
 -10-17 16:05:57.090521 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6812/2640 63 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:05:57.089446) v2 ==== 47+0+0 (1263760565 0 0) 0x496e780 con 0x4f8eec0
 -10-17 16:05:57.090602 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:57.089446 last_rx_back 2014-10-17 16:05:51.788971 last_rx_front 2014-10-17 16:05:51.788971 -> 2014-10-17 16:05:57.089446
 -10-17 16:05:57.090645 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:57.090678 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6810/2640 63 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:05:57.089446) v2 ==== 47+0+0 (1263760565 0 0) 0x518ef00 con 0x4f8ec00
 -10-17 16:05:57.090701 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:57.089446 last_rx_back 2014-10-17 16:05:51.788971 -> 2014-10-17 16:05:57.089446 last_rx_front 2014-10-17 16:05:57.089446
 -10-17 16:05:57.090725 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:57.235209 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:05:57.235261 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:57.089446 last_rx_back 2014-10-17 16:05:57.089446 last_rx_front 2014-10-17 16:05:57.089446
 -10-17 16:05:57.235291 7f0abd1ef700  7 osd.1 16 do_mon_report
 -10-17 16:05:57.235295 7f0abd1ef700 10 osd.1 16 send_alive up_thru currently 15 want 15
 -10-17 16:05:57.235301 7f0abd1ef700 20 osd.1 16 send_pg_stats
 -10-17 16:05:57.235330 7f0abd1ef700 20 osd.1 16 scrub_random_backoff lost coin flip, randomly backing off
 -10-17 16:05:57.235338 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:57.235341 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:58.168962 7f0abeae2700 10 osd.1.objecter tick
 -10-17 16:05:58.235477 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:05:58.235543 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:57.089446 last_rx_back 2014-10-17 16:05:57.089446 last_rx_front 2014-10-17 16:05:57.089446
 -10-17 16:05:58.235638 7f0abd1ef700 20 osd.1 16 scrub_random_backoff lost coin flip, randomly backing off
 -10-17 16:05:58.235665 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:58.235674 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:58.584746 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 <== osd.2 127.0.0.1:0/2640 76 ==== osd_ping(ping e16 stamp 2014-10-17 16:05:58.584252) v2 ==== 47+0+0 (756060908 0 0) 0x49703a0 con 0x4b014a0
 -10-17 16:05:58.584758 7f0ab0367700  1 -- 127.0.0.1:6806/2360 <== osd.2 127.0.0.1:0/2640 76 ==== osd_ping(ping e16 stamp 2014-10-17 16:05:58.584252) v2 ==== 47+0+0 (756060908 0 0) 0x4c65860 con 0x4b01080
 -10-17 16:05:58.584898 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:05:58.584252) v2 -- ?+0 0x496f2c0 con 0x4b014a0
 -10-17 16:05:58.584974 7f0ab1c5a700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:58.585059 7f0ab0367700  1 -- 127.0.0.1:6806/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:05:58.584252) v2 -- ?+0 0x5191480 con 0x4b01080
 -10-17 16:05:58.585159 7f0ab0367700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:58.789821 7f0aa4c71700 20 osd.1 16 update_osd_stat osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:05:58.789870 7f0aa4c71700  5 osd.1 16 heartbeat: osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:05:58.789911 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6810/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:05:58.789897) v2 -- ?+0 0x4c67a20 con 0x4f8ec00
 -10-17 16:05:58.789986 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6812/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:05:58.789897) v2 -- ?+0 0x4c654a0 con 0x4f8eec0
 -10-17 16:05:58.790033 7f0aa4c71700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:58.789897 last_rx_back 2014-10-17 16:05:57.089446 last_rx_front 2014-10-17 16:05:57.089446
 -10-17 16:05:58.790897 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6810/2640 64 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:05:58.789897) v2 ==== 47+0+0 (231322297 0 0) 0x518fa40 con 0x4f8ec00
 -10-17 16:05:58.790975 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:58.789897 last_rx_back 2014-10-17 16:05:57.089446 -> 2014-10-17 16:05:58.789897 last_rx_front 2014-10-17 16:05:57.089446
 -10-17 16:05:58.791038 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:58.791090 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6812/2640 64 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:05:58.789897) v2 ==== 47+0+0 (231322297 0 0) 0x4970580 con 0x4f8eec0
 -10-17 16:05:58.791159 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:58.789897 last_rx_back 2014-10-17 16:05:58.789897 last_rx_front 2014-10-17 16:05:57.089446 -> 2014-10-17 16:05:58.789897
 -10-17 16:05:58.791212 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:59.235828 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:05:59.235861 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:58.789897 last_rx_back 2014-10-17 16:05:58.789897 last_rx_front 2014-10-17 16:05:58.789897
 -10-17 16:05:59.235917 7f0abd1ef700 20 osd.1 16 scrub_should_schedule loadavg 0.44 < max 5 = yes
 -10-17 16:05:59.235932 7f0abd1ef700 20 osd.1 16 sched_scrub load_is_low=1
 -10-17 16:05:59.235948 7f0abd1ef700 10 osd.1 16 sched_scrub 0.2 at 2014-10-17 16:01:48.864210: 250.372 < min (86400 seconds)
 -10-17 16:05:59.235955 7f0abd1ef700 20 osd.1 16 sched_scrub done
 -10-17 16:05:59.235957 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:05:59.235958 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:05:59.890367 7f0aa4c71700 20 osd.1 16 update_osd_stat osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:05:59.890422 7f0aa4c71700  5 osd.1 16 heartbeat: osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:05:59.890506 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6810/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:05:59.890447) v2 -- ?+0 0x51affe0 con 0x4f8ec00
 -10-17 16:05:59.890591 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6812/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:05:59.890447) v2 -- ?+0 0x51afe00 con 0x4f8eec0
 -10-17 16:05:59.890691 7f0aa4c71700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:59.890447 last_rx_back 2014-10-17 16:05:58.789897 last_rx_front 2014-10-17 16:05:58.789897
 -10-17 16:05:59.891586 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6810/2640 65 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:05:59.890447) v2 ==== 47+0+0 (639503031 0 0) 0x518ffe0 con 0x4f8ec00
 -10-17 16:05:59.891662 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:59.890447 last_rx_back 2014-10-17 16:05:58.789897 -> 2014-10-17 16:05:59.890447 last_rx_front 2014-10-17 16:05:58.789897
 -10-17 16:05:59.891723 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:05:59.891772 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6812/2640 65 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:05:59.890447) v2 ==== 47+0+0 (639503031 0 0) 0x4971480 con 0x4f8eec0
 -10-17 16:05:59.891835 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:59.890447 last_rx_back 2014-10-17 16:05:59.890447 last_rx_front 2014-10-17 16:05:58.789897 -> 2014-10-17 16:05:59.890447
 -10-17 16:05:59.891886 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:00.236097 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:06:00.236129 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:59.890447 last_rx_back 2014-10-17 16:05:59.890447 last_rx_front 2014-10-17 16:05:59.890447
 -10-17 16:06:00.236186 7f0abd1ef700 20 osd.1 16 scrub_should_schedule loadavg 0.44 < max 5 = yes
 -10-17 16:06:00.236196 7f0abd1ef700 20 osd.1 16 sched_scrub load_is_low=1
 -10-17 16:06:00.236198 7f0abd1ef700 10 osd.1 16 sched_scrub 0.2 at 2014-10-17 16:01:48.864210: 251.372 < min (86400 seconds)
 -10-17 16:06:00.236205 7f0abd1ef700 20 osd.1 16 sched_scrub done
 -10-17 16:06:00.236207 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:00.236208 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:00.284965 7f0ab0367700  1 -- 127.0.0.1:6806/2360 <== osd.2 127.0.0.1:0/2640 77 ==== osd_ping(ping e16 stamp 2014-10-17 16:06:00.284651) v2 ==== 47+0+0 (3988830398 0 0) 0x4c670c0 con 0x4b01080
 -10-17 16:06:00.284974 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 <== osd.2 127.0.0.1:0/2640 77 ==== osd_ping(ping e16 stamp 2014-10-17 16:06:00.284651) v2 ==== 47+0+0 (3988830398 0 0) 0x496ffe0 con 0x4b014a0
 -10-17 16:06:00.285085 7f0ab0367700  1 -- 127.0.0.1:6806/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:06:00.284651) v2 -- ?+0 0x4c65860 con 0x4b01080
 -10-17 16:06:00.285151 7f0ab0367700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:00.285208 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:06:00.284651) v2 -- ?+0 0x49703a0 con 0x4b014a0
 -10-17 16:06:00.285303 7f0ab1c5a700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:00.359871 7f0abb8fc700 20 filestore(/home/loic/software/ceph/ceph/src/dev/osd1) sync_entry woke after 5.000156
 -10-17 16:06:00.359933 7f0abb8fc700 10 journal commit_start max_applied_seq 145, open_ops 0
 -10-17 16:06:00.359941 7f0abb8fc700 10 journal commit_start blocked, all open_ops have completed
 -10-17 16:06:00.359945 7f0abb8fc700 10 journal commit_start nothing to do
 -10-17 16:06:00.359950 7f0abb8fc700 10 journal commit_start
 -10-17 16:06:00.359976 7f0abb8fc700 20 filestore(/home/loic/software/ceph/ceph/src/dev/osd1) sync_entry waiting for max_interval 5.000000
 -10-17 16:06:01.236345 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:06:01.236377 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:05:59.890447 last_rx_back 2014-10-17 16:05:59.890447 last_rx_front 2014-10-17 16:05:59.890447
 -10-17 16:06:01.236434 7f0abd1ef700 20 osd.1 16 scrub_should_schedule loadavg 0.4 < max 5 = yes
 -10-17 16:06:01.236444 7f0abd1ef700 20 osd.1 16 sched_scrub load_is_low=1
 -10-17 16:06:01.236446 7f0abd1ef700 10 osd.1 16 sched_scrub 0.2 at 2014-10-17 16:01:48.864210: 252.372 < min (86400 seconds)
 -10-17 16:06:01.236453 7f0abd1ef700 20 osd.1 16 sched_scrub done
 -10-17 16:06:01.236455 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:01.236456 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:01.590944 7f0aa4c71700 20 osd.1 16 update_osd_stat osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:06:01.590987 7f0aa4c71700  5 osd.1 16 heartbeat: osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:06:01.590999 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6810/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:06:01.590994) v2 -- ?+0 0x51afc20 con 0x4f8ec00
 -10-17 16:06:01.591020 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6812/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:06:01.590994) v2 -- ?+0 0x51afa40 con 0x4f8eec0
 -10-17 16:06:01.591043 7f0aa4c71700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:05:59.890447 last_rx_front 2014-10-17 16:05:59.890447
 -10-17 16:06:01.591416 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6810/2640 66 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:06:01.590994) v2 ==== 47+0+0 (526584437 0 0) 0x518f680 con 0x4f8ec00
 -10-17 16:06:01.591454 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:05:59.890447 -> 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:05:59.890447
 -10-17 16:06:01.591489 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:01.591499 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6812/2640 66 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:06:01.590994) v2 ==== 47+0+0 (526584437 0 0) 0x4970940 con 0x4f8eec0
 -10-17 16:06:01.591520 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:05:59.890447 -> 2014-10-17 16:06:01.590994
 -10-17 16:06:01.591540 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:02.236592 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:06:02.236625 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:06:01.590994
 -10-17 16:06:02.236638 7f0abd1ef700  7 osd.1 16 do_mon_report
 -10-17 16:06:02.236640 7f0abd1ef700 10 osd.1 16 send_alive up_thru currently 15 want 15
 -10-17 16:06:02.236642 7f0abd1ef700 20 osd.1 16 send_pg_stats
 -10-17 16:06:02.236693 7f0abd1ef700 20 osd.1 16 scrub_should_schedule loadavg 0.4 < max 5 = yes
 -10-17 16:06:02.236701 7f0abd1ef700 20 osd.1 16 sched_scrub load_is_low=1
 -10-17 16:06:02.236703 7f0abd1ef700 10 osd.1 16 sched_scrub 0.2 at 2014-10-17 16:01:48.864210: 253.372 < min (86400 seconds)
 -10-17 16:06:02.236709 7f0abd1ef700 20 osd.1 16 sched_scrub done
 -10-17 16:06:02.236711 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:02.236712 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:03.169183 7f0abeae2700 10 osd.1.objecter tick
 -10-17 16:06:03.236892 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:06:03.236973 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:06:01.590994
 -10-17 16:06:03.237057 7f0abd1ef700 20 osd.1 16 scrub_random_backoff lost coin flip, randomly backing off
 -10-17 16:06:03.237067 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:03.237071 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:03.767116 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 155 ==== MBackfillReserve REQUEST  pgid: 0.3, query_epoch: 16, prio: 201 v3 ==== 30+0+0 (760745717 0 0) 0x518e780 con 0x4b01b80
 -10-17 16:06:03.767181 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:03.767187 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:03.767190 7f0ab45ef700 20 osd.1 16 _dispatch 0x518e780 MBackfillReserve REQUEST  pgid: 0.3, query_epoch: 16, prio: 201 v3
 -10-17 16:06:03.767254 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x518e780
 -10-17 16:06:03.767364 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:03.767368 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:03.767390 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfillPrio: priority 201
 -10-17 16:06:03.767479 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] state<Started/ReplicaActive/RepNotRecovering>: backfill reservation rejected: full ratio is 0.896762, which is greater than max allowed ratio 0.8
 -10-17 16:06:03.767537 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 10.002894 1 0.000308
 -10-17 16:06:03.767589 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] enter Started/ReplicaActive/RepWaitBackfillReserved
 -10-17 16:06:03.767661 7f0aac0df700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:03.767679 7f0aac0df700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REJECT  pgid: 0.3, query_epoch: 16 v3 -- ?+0 0x4c67c00 con 0x4b01b80
 -10-17 16:06:03.767723 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepWaitBackfillReserved 0.000135 0 0.000000
 -10-17 16:06:03.767777 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.3( v 11'9 (11'4,11'9] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/10) [2,1]/[2] r=-1 lpr=16 pi=10-14/3 luod=0'0 crt=0'0 lcod 0'0 active] enter Started/ReplicaActive/RepNotRecovering
 -10-17 16:06:03.767813 7f0aac0df700 10 log is not dirty
 -10-17 16:06:03.801028 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfill
 -10-17 16:06:03.801137 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/NotBackfilling 10.000277 1 0.000183
 -10-17 16:06:03.801199 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/WaitLocalBackfillReserved
 -10-17 16:06:03.801285 7f0aac930700 10 log is not dirty
 -10-17 16:06:03.801481 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 LocalBackfillReserved
 -10-17 16:06:03.801569 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 0.000370 1 0.000157
 -10-17 16:06:03.801642 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved
 -10-17 16:06:03.801760 7f0aac0df700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REQUEST  pgid: 0.2, query_epoch: 16, prio: 201 v3 -- ?+0 0x4c66b20 con 0x4b01b80
 -10-17 16:06:03.801843 7f0aac0df700 10 log is not dirty
 -10-17 16:06:03.803485 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 156 ==== MBackfillReserve REJECT  pgid: 0.2, query_epoch: 16 v3 ==== 30+0+0 (3553360764 0 0) 0x4c66b20 con 0x4b01b80
 -10-17 16:06:03.803544 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:03.803552 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:03.803557 7f0ab45ef700 20 osd.1 16 _dispatch 0x4c66b20 MBackfillReserve REJECT  pgid: 0.2, query_epoch: 16 v3
 -10-17 16:06:03.803651 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x4c66b20
 -10-17 16:06:03.803791 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:03.803807 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:03.803793 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RemoteReservationRejected
 -10-17 16:06:03.803893 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.002252 1 0.000279
 -10-17 16:06:03.803966 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.2( v 11'11 (11'5,11'11] local-les=16 n=11 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'5 lcod 11'10 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling
 -10-17 16:06:03.804004 7f0aac930700 10 log is not dirty
 -10-17 16:06:04.134794 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 157 ==== MBackfillReserve REQUEST  pgid: 0.5, query_epoch: 16, prio: 201 v3 ==== 30+0+0 (56951451 0 0) 0x4c67c00 con 0x4b01b80
 -10-17 16:06:04.134855 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:04.134863 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:04.134873 7f0ab45ef700 20 osd.1 16 _dispatch 0x4c67c00 MBackfillReserve REQUEST  pgid: 0.5, query_epoch: 16, prio: 201 v3
 -10-17 16:06:04.134987 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x4c67c00
 -10-17 16:06:04.135155 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:04.135169 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:04.135138 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfillPrio: priority 201
 -10-17 16:06:04.135213 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] state<Started/ReplicaActive/RepNotRecovering>: backfill reservation rejected: full ratio is 0.896762, which is greater than max allowed ratio 0.8
 -10-17 16:06:04.135265 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepNotRecovering 10.002727 1 0.000352
 -10-17 16:06:04.135307 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] enter Started/ReplicaActive/RepWaitBackfillReserved
 -10-17 16:06:04.135374 7f0aac0df700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:04.135383 7f0aac0df700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REJECT  pgid: 0.5, query_epoch: 16 v3 -- ?+0 0x4c64780 con 0x4b01b80
 -10-17 16:06:04.135428 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] exit Started/ReplicaActive/RepWaitBackfillReserved 0.000120 0 0.000000
 -10-17 16:06:04.135475 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.5( v 11'18 (11'13,11'18] lb 0//0//-1 local-les=0 n=0 ec=1 les/c 16/11 14/15/12) [2,1]/[2] r=-1 lpr=16 pi=8-14/4 luod=0'0 crt=0'0 lcod 0'0 active] enter Started/ReplicaActive/RepNotRecovering
 -10-17 16:06:04.135511 7f0aac0df700 10 log is not dirty
 -10-17 16:06:04.161021 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfill
 -10-17 16:06:04.161127 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/NotBackfilling 10.000360 1 0.000127
 -10-17 16:06:04.161185 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/WaitLocalBackfillReserved
 -10-17 16:06:04.161279 7f0aac930700 10 log is not dirty
 -10-17 16:06:04.161418 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 LocalBackfillReserved
 -10-17 16:06:04.161479 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 0.000295 1 0.000160
 -10-17 16:06:04.161522 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved
 -10-17 16:06:04.161582 7f0aac0df700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REQUEST  pgid: 0.6, query_epoch: 16, prio: 201 v3 -- ?+0 0x518ed20 con 0x4b01b80
 -10-17 16:06:04.161638 7f0aac0df700 10 log is not dirty
 -10-17 16:06:04.163305 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 158 ==== MBackfillReserve REJECT  pgid: 0.6, query_epoch: 16 v3 ==== 30+0+0 (3890351304 0 0) 0x518ed20 con 0x4b01b80
 -10-17 16:06:04.163365 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:04.163372 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:04.163377 7f0ab45ef700 20 osd.1 16 _dispatch 0x518ed20 MBackfillReserve REJECT  pgid: 0.6, query_epoch: 16 v3
 -10-17 16:06:04.163487 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x518ed20
 -10-17 16:06:04.163626 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:04.163646 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:04.163628 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RemoteReservationRejected
 -10-17 16:06:04.163728 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.002207 1 0.000163
 -10-17 16:06:04.163791 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.6( v 11'8 (11'1,11'8] local-les=16 n=8 ec=1 les/c 16/13 14/15/12) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'1 lcod 11'7 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling
 -10-17 16:06:04.163827 7f0aac930700 10 log is not dirty
 -10-17 16:06:04.237211 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:06:04.237245 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:06:01.590994
 -10-17 16:06:04.237285 7f0abd1ef700 20 osd.1 16 scrub_random_backoff lost coin flip, randomly backing off
 -10-17 16:06:04.237293 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:04.237297 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:04.331050 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RequestBackfill
 -10-17 16:06:04.331141 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/NotBackfilling 10.000286 1 0.000128
 -10-17 16:06:04.331188 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/WaitLocalBackfillReserved
 -10-17 16:06:04.331265 7f0aac0df700 10 log is not dirty
 -10-17 16:06:04.331376 7f0aac930700 10 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 LocalBackfillReserved
 -10-17 16:06:04.331429 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] exit Started/Primary/Active/WaitLocalBackfillReserved 0.000240 1 0.000130
 -10-17 16:06:04.331468 7f0aac930700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] enter Started/Primary/Active/WaitRemoteBackfillReserved
 -10-17 16:06:04.331521 7f0aac930700  1 -- 127.0.0.1:6805/2360 --> 127.0.0.1:6809/2640 -- MBackfillReserve REQUEST  pgid: 0.7, query_epoch: 16, prio: 201 v3 -- ?+0 0x5191a20 con 0x4b01b80
 -10-17 16:06:04.331573 7f0aac930700 10 log is not dirty
 -10-17 16:06:04.333109 7f0ab45ef700  1 -- 127.0.0.1:6805/2360 <== osd.2 127.0.0.1:6809/2640 159 ==== MBackfillReserve REJECT  pgid: 0.7, query_epoch: 16 v3 ==== 30+0+0 (3941101861 0 0) 0x5191a20 con 0x4b01b80
 -10-17 16:06:04.333168 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:04.333176 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:04.333208 7f0ab45ef700 20 osd.1 16 _dispatch 0x5191a20 MBackfillReserve REJECT  pgid: 0.7, query_epoch: 16 v3
 -10-17 16:06:04.333304 7f0ab45ef700 15 osd.1 16 require_same_or_newer_map 16 (i am 16) 0x5191a20
 -10-17 16:06:04.333444 7f0ab45ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:04.333461 7f0ab45ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:04.333444 7f0aac0df700 10 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill+backfill_toofull] handle_peering_event: epoch_sent: 16 epoch_requested: 16 RemoteReservationRejected
 -10-17 16:06:04.333549 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.002081 1 0.000146
 -10-17 16:06:04.333620 7f0aac0df700  5 osd.1 pg_epoch: 16 pg[0.7( v 11'16 (11'10,11'16] local-les=16 n=16 ec=1 les/c 16/13 14/15/8) [1,2]/[1] r=0 lpr=15 pi=12-14/2 bft=2 crt=11'10 lcod 11'15 mlcod 0'0 active+undersized+degraded+remapped+backfill_toofull] enter Started/Primary/Active/NotBackfilling
 -10-17 16:06:04.333658 7f0aac0df700 10 log is not dirty
 -10-17 16:06:04.385462 7f0ab0367700  1 -- 127.0.0.1:6806/2360 <== osd.2 127.0.0.1:0/2640 78 ==== osd_ping(ping e16 stamp 2014-10-17 16:06:04.385034) v2 ==== 47+0+0 (3435835715 0 0) 0x4c64000 con 0x4b01080
 -10-17 16:06:04.385484 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 <== osd.2 127.0.0.1:0/2640 78 ==== osd_ping(ping e16 stamp 2014-10-17 16:06:04.385034) v2 ==== 47+0+0 (3435835715 0 0) 0x496eb40 con 0x4b014a0
 -10-17 16:06:04.385588 7f0ab0367700  1 -- 127.0.0.1:6806/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:06:04.385034) v2 -- ?+0 0x4c670c0 con 0x4b01080
 -10-17 16:06:04.385654 7f0ab0367700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:04.385711 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:06:04.385034) v2 -- ?+0 0x496ffe0 con 0x4b014a0
 -10-17 16:06:04.385825 7f0ab1c5a700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:05.237449 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:06:05.237502 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:06:01.590994
 -10-17 16:06:05.237623 7f0abd1ef700 20 osd.1 16 scrub_should_schedule loadavg 0.4 < max 5 = yes
 -10-17 16:06:05.237666 7f0abd1ef700 20 osd.1 16 sched_scrub load_is_low=1
 -10-17 16:06:05.237677 7f0abd1ef700 10 osd.1 16 sched_scrub 0.2 at 2014-10-17 16:01:48.864210: 256.373 < min (86400 seconds)
 -10-17 16:06:05.237712 7f0abd1ef700 20 osd.1 16 sched_scrub done
 -10-17 16:06:05.237720 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:05.237723 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:05.360142 7f0abb8fc700 20 filestore(/home/loic/software/ceph/ceph/src/dev/osd1) sync_entry woke after 5.000164
 -10-17 16:06:05.360204 7f0abb8fc700 10 journal commit_start max_applied_seq 145, open_ops 0
 -10-17 16:06:05.360217 7f0abb8fc700 10 journal commit_start blocked, all open_ops have completed
 -10-17 16:06:05.360221 7f0abb8fc700 10 journal commit_start nothing to do
 -10-17 16:06:05.360227 7f0abb8fc700 10 journal commit_start
 -10-17 16:06:05.360253 7f0abb8fc700 20 filestore(/home/loic/software/ceph/ceph/src/dev/osd1) sync_entry waiting for max_interval 5.000000
 -10-17 16:06:05.485966 7f0ab0367700  1 -- 127.0.0.1:6806/2360 <== osd.2 127.0.0.1:0/2640 79 ==== osd_ping(ping e16 stamp 2014-10-17 16:06:05.485443) v2 ==== 47+0+0 (415244331 0 0) 0x4c65e00 con 0x4b01080
 -10-17 16:06:05.485992 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 <== osd.2 127.0.0.1:0/2640 79 ==== osd_ping(ping e16 stamp 2014-10-17 16:06:05.485443) v2 ==== 47+0+0 (415244331 0 0) 0x4970b20 con 0x4b014a0
 -10-17 16:06:05.486070 7f0ab0367700  1 -- 127.0.0.1:6806/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:06:05.485443) v2 -- ?+0 0x4c64000 con 0x4b01080
 -10-17 16:06:05.486136 7f0ab0367700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:05.486194 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:06:05.485443) v2 -- ?+0 0x496eb40 con 0x4b014a0
 -10-17 16:06:05.486301 7f0ab1c5a700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:06.166198 7f0aad9d2700 10 monclient: tick
 -10-17 16:06:06.166232 7f0aad9d2700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-10-17 16:05:36.166231)
 -10-17 16:06:06.166242 7f0aad9d2700 10 monclient: renew subs? (now: 2014-10-17 16:06:06.166242; renew after: 2014-10-17 16:06:56.163640) -- no
 -10-17 16:06:06.237872 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:06:06.237905 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:06:01.590994
 -10-17 16:06:06.237962 7f0abd1ef700 20 osd.1 16 scrub_should_schedule loadavg 0.37 < max 5 = yes
 -10-17 16:06:06.237972 7f0abd1ef700 20 osd.1 16 sched_scrub load_is_low=1
 -10-17 16:06:06.237975 7f0abd1ef700 10 osd.1 16 sched_scrub 0.2 at 2014-10-17 16:01:48.864210: 257.374 < min (86400 seconds)
 -10-17 16:06:06.237980 7f0abd1ef700 20 osd.1 16 sched_scrub done
 -10-17 16:06:06.237983 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:06.237984 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:07.186067 7f0ab0367700  1 -- 127.0.0.1:6806/2360 <== osd.2 127.0.0.1:0/2640 80 ==== osd_ping(ping e16 stamp 2014-10-17 16:06:07.185863) v2 ==== 47+0+0 (2749626086 0 0) 0x4c66580 con 0x4b01080
 -10-17 16:06:07.186086 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 <== osd.2 127.0.0.1:0/2640 80 ==== osd_ping(ping e16 stamp 2014-10-17 16:06:07.185863) v2 ==== 47+0+0 (2749626086 0 0) 0x4970ee0 con 0x4b014a0
 -10-17 16:06:07.186115 7f0ab0367700  1 -- 127.0.0.1:6806/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:06:07.185863) v2 -- ?+0 0x4c65e00 con 0x4b01080
 -10-17 16:06:07.186144 7f0ab0367700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:07.186174 7f0ab1c5a700  1 -- 127.0.0.1:6807/2360 --> 127.0.0.1:0/2640 -- osd_ping(ping_reply e16 stamp 2014-10-17 16:06:07.185863) v2 -- ?+0 0x4970b20 con 0x4b014a0
 -10-17 16:06:07.186214 7f0ab1c5a700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:07.238130 7f0abd1ef700  5 osd.1 16 tick
 -10-17 16:06:07.238163 7f0abd1ef700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:01.590994 last_rx_back 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:06:01.590994
 -10-17 16:06:07.238175 7f0abd1ef700  7 osd.1 16 do_mon_report
 -10-17 16:06:07.238177 7f0abd1ef700 10 osd.1 16 send_alive up_thru currently 15 want 15
 -10-17 16:06:07.238178 7f0abd1ef700 20 osd.1 16 send_pg_stats
 -10-17 16:06:07.238191 7f0abd1ef700 20 osd.1 16 scrub_random_backoff lost coin flip, randomly backing off
 -10-17 16:06:07.238194 7f0abd1ef700 10 osd.1 16 do_waiters -- start
 -10-17 16:06:07.238205 7f0abd1ef700 10 osd.1 16 do_waiters -- finish
 -10-17 16:06:07.491260 7f0aa4c71700 20 osd.1 16 update_osd_stat osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:06:07.491302 7f0aa4c71700  5 osd.1 16 heartbeat: osd_stat(163 GB used, 19277 MB avail, 182 GB total, peers [2]/[] op hist [])
 -10-17 16:06:07.491315 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6810/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:06:07.491309) v2 -- ?+0 0x51af860 con 0x4f8ec00
 -10-17 16:06:07.491336 7f0aa4c71700  1 -- 127.0.0.1:0/2360 --> 127.0.0.1:6812/2640 -- osd_ping(ping e16 stamp 2014-10-17 16:06:07.491309) v2 -- ?+0 0x51af680 con 0x4f8eec0
 -10-17 16:06:07.491358 7f0aa4c71700 25 osd.1 16 heartbeat_check osd.2 first_tx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:07.491309 last_rx_back 2014-10-17 16:06:01.590994 last_rx_front 2014-10-17 16:06:01.590994
 -10-17 16:06:07.491741 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6810/2640 67 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:06:07.491309) v2 ==== 47+0+0 (2726995711 0 0) 0x496f860 con 0x4f8ec00
 -10-17 16:06:07.491778 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:07.491309 last_rx_back 2014-10-17 16:06:01.590994 -> 2014-10-17 16:06:07.491309 last_rx_front 2014-10-17 16:06:01.590994
 -10-17 16:06:07.491795 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16
 -10-17 16:06:07.491814 7f0ab2cfc700  1 -- 127.0.0.1:0/2360 <== osd.2 127.0.0.1:6812/2640 67 ==== osd_ping(ping_reply e16 stamp 2014-10-17 16:06:07.491309) v2 ==== 47+0+0 (2726995711 0 0) 0x4c67840 con 0x4f8eec0
 -10-17 16:06:07.491835 7f0ab2cfc700 25 osd.1 16 handle_osd_ping got reply from osd.2 first_rx 2014-10-17 16:02:21.262553 last_tx 2014-10-17 16:06:07.491309 last_rx_back 2014-10-17 16:06:07.491309 last_rx_front 2014-10-17 16:06:01.590994 -> 2014-10-17 16:06:07.491309
 -10-17 16:06:07.491846 7f0ab2cfc700 20 osd.1 16 share_map_peer 0x4b01b80 already has epoch 16

« Previous
1
2
Next »

(2-2/2)

Project

General

Profile

Ceph

Bug #9808 » osd.1.log