Ceph

main reason caused this is the Op order is wrong because osd_op_queue,osd_op_queue_cut_off configured as debug_random.
And I also tried to read PrioritizedQueue.h code, I think the reason caused this is:
1. enqueue_op repop(osd_op) this can cause a cow, priority is 127
2. enqueue_op MOSDPGPush ( this will use the snap generated by 1), priority is 63
if osd_op_queue_cut_off = high, then the up two op will be all put into queue, but queue has starvation strategy,
this may cause the Op order wrong, and lead to assert failed.

code like following:

// if there are multiple buckets/subqueues with sufficient tokens,
// we behave like a strict priority queue among all subqueues that
// are eligible to run.
for (typename SubQueues::iterator i = queue.begin();
i != queue.end();
++i) {
assert(!(i->second.empty()));
if (i->second.front().first < i->second.num_tokens()) {
T ret = i->second.front().second;
unsigned cost = i->second.front().first;
i->second.take_tokens(cost);
i->second.pop_front();
if (i->second.empty()) {
remove_queue(i->first);
}
distribute_tokens(cost);
return ret;
}
}

If I understand this correctly, the bug is that with the strict cutoff at 127, it's possible for a push on a newly created object during backfill to end up ordered ahead of the repop which creates it. I expect that the bug is that
1) Issue repop which creates clone due to cow
2) Backfill catches up to that object and sends a push on another clone using the clone created in 1) as a source for clone_range for some common extents

The bug ultimately is that we are sending a push with clone sources which we are not holding ObjectContext locks on. The interface between ReplicatedBackend and ReplicatedPG will need to be extended a bit so that ReplicatedBackend won't use any source objects it can't lock. Those obcs will then need to be added to the set to unlock when we finish the push.