Ceph

In the attached image you'll see graphs of a set of nine Windows guests running video surveillance over the duration of the Ceph recovery process. Under normal conditions (when Ceph isn't recovering/backfilling), the blue line on each graph should track closely with the red dashed baseline. The drops indicate i/o issues due to the Ceph recovery and backfilling process.

As shown in the image, the guests don't halt or resume i/o at the same time. At some points during the recovery, i/o proceeds normally, then falls off. Sometimes the fall off will go to roughly 50% the expected bandwidth. Other times, it will fall off to 0mbps.

Added a graph of the same time period from last night / this morning for a comparison of what the graphs should look like when RBD is working properly. As you can see, i/o should be roughly consistent and track close to the red dashed baseline. The tall spikes can be ignored, because they only occur during school passing periods when students flood the hallways and create heavier writes.

We had an osd process die unexpectedly a few days ago. I run a cronjob that 'service ceph start's on all nodes for this very reason. It restarted the osd daemon and recovery started. During the time, at least on other OSD flapped and was incorrectly marked down with status like:

HEALTH_WARN 128 pgs backfill; 267 pgs backfilling; 3355 pgs degraded; 3 pgs down; 32 pgs peering; 92 pgs recovering; 94 pgs recovery_wait; 12 pgs stuck inactive; 2172 pgs stuck unclean; 27 requests are blocked > 32 sec; 5 osds have slow requests; recovery 2116832/32281104 degraded (6.557%); 2/64 in osds are down; noscrub,nodeep-scrub flag(s) set

The whole event took ~1.5 hours. Attached are graphs showing three of the guests' traffic versus their baseline during this event (ceph-issue-6333-recovery.jpg) plus four graphs of various rbd perf dumps for the same three guests. You'll notice reads seem to pile up, then get processed in an abnormally large spike. And you'll notice extremely high read latency.

Could this be a situation where we're attempting to read from the primary object, but it is quite busy? If that is the case, is it possible to detect this issue, and remap the request to another copy that could respond faster?

Hey-

Here is what I think we need to fully understand what is happening:

- debug ms = 1, debug osd = 20 on the osds during recovery/backfill. (hopefully this is doable without torpedoing performance so much)
- wait for a guest to stall
- while guest is stalled, query the admin socket and dump the objecter requests

That will let us see the specific request(s) that are blocked (along with how long they ahve been blocked), and map that back to the osd logs where we can see why they are stalled out.

You may be right that it is reads, but I'm not certain.. in general, an individual object that is stuck should not stay that way for long. I have some ideas about what we can do to help, but I'm not sure the status quo should be as bad as it sounds like it is, so we should address that first.

Hey, any update on this one?

See #6278 for teh snaptrimmer portion of this. the rest is somewhere between resolved, need more info, and can't reproduce.