Ceph

i reproduced this issue on master and on 7b1c77a643516c12de4df558f89924b8c0fe45e7 by manually repeating the steps in recovery-preemption.yaml with a vstart cluster with memstore. but after the recovery/backfill finished, none of the backfill/recovery reported being preempt: no (defer backfill|defer recovery) showed up in the osd log.

debug-reserver=10 shows that there are victims get preempted. but following patch does not print any "DeferBackfill" or "DeferRecovery" in the log message. will keep looking

diff --git a/src/osd/PG.cc b/src/osd/PG.cc
index 005333530f..f8523ad5c3 100644
--- a/src/osd/PG.cc
+++ b/src/osd/PG.cc
@@ -5810,6 +5810,13 @@ void PG::process_peering_event(RecoveryCtx *rctx)

 void PG::queue_peering_event(CephPeeringEvtRef evt)
 {
+
+  if (evt->get_event().dynamic_type() == PG::DeferBackfill::static_type()) {
+    derr << __func__ << "DeferBackfill" << dendl;
+  } else if (evt->get_event().dynamic_type() == PG::DeferRecovery::static_type()) {
+    derr << __func__ << "DeferRecovery" << dendl;
+  }
+
   if (old_peering_evt(evt))
     return;
   peering_queue.push_back(evt);

mostly passes? http://pulpito.ceph.com/sage-2017-11-20_20:12:00-rados-master-distro-basic-smithi/

3 out of 10 failed: http://pulpito.ceph.com/kchai-2017-11-28_03:41:37-rados-master-distro-basic-mira/

my wip-22043 helped, see http://pulpito.ceph.com/sage-2017-12-05_21:42:54-rados:singleton-master-distro-basic-mira/

the problem seems to be that the timing of the test isn't reliably triggering backfill when the osd is marked out. i think we need to (1) set pg logs short, (2) do initial long bench run, (3) mark osd out. (now guarnateed to have backfill.) then (4) set pg logs long again, and (5) do the short run to dirty the pgs, then (6) let the down osd recovery (should reliably do log recovery). that's what leads to the preemption message in the log.