Ceph

two theories:

- deep scrub is slowing things down. can you try 'ceph osd set nodeepscrub' and/or 'ceph osd set noscrub', wait for scrubs to finish, and then repeat?

- one or a handful of osds are slow peering. can you do something like

while sleep 5 ; do date ; ceph pg dump | grep peering ; done | tee peering.txt

while an osd is restarted, stopping only after everything is healthy again?

One interesting observation is that when I tried restarting an OSD a few minutes after it has been restarted and cluster settled, peering finished up in seconds. I then tried restarting osd.0, which I had restarted yesterday (so it was running 0.56.6); this resulted in slow peering again.

Another observation is that I've seen a few times is "active" (not active+clean) pgs right after peering and for a few minutes. These came with slow requests, so not sure if it's normal either.

This could be a result of writing out pg epochs on each map change. I have a branch which should greatly reduce the problem.

wip_bobtail_pg_epoch 2c5a9f0e178843e7ed514708bab137def840ab89

I just pushed that version, so verify the sha1 before using (the gitbuilder might take a little while to get to it).

It's based on current bobtail. Let us know if it helps.
-Sam

Something that's not clear to me: does this need to be in all peers to have an effect? Or in other words, to fix this would I need to upgrade all of my OSDs to a version containing this to have a faster peering?

Also, this doesn't affect Cuttlefish at all, right?

As we can assume client ops are waiting for new OSD map that is issued only after peering finishes.
It seems that reasonable way to fix the problem is to speed up issuing of new OSD map, is it possible? Thank you.
P.S. ceph and iostat logs are attached, 2 osds (placed on sda and sdb) are marked as 'out' during the test.

After some testing want to make a little correction to the last point.
Actually, ops to a specific PG start to be processed after the one becomes active, not after peering completely finishes. Assuming entire peering takes T (T is in order of minutes), then avg latency of a single read op would be T/2 and avg latency of a single write op would be 3/4T (due to waiting for replicas). So a typical latency during peering takes tens of seconds, that is inappropriate.

I guess it's unlikely to speed up peering by an order of magnitude, but may be it's ok to prioritize peering of PGs awaited by clients? Though docs say "peering messages must not be reordered"...

We wrote a test that sequentially reads 1M blocks spreaded by 64Mb offsets (i.e. from different placement groups) and prints each op latency, and started it simultaneously with peering. Here is what timings we've got:
2.384s
9.328s
2.034s
0.001s
1.179s
0.001s
0.797s
56.005s
23.488s
2.903s
38.236s
1.930s
7.875s
0.001s
3.760s
0.001s
4.768s
0.001s

Briefly, peering took 140 sec, and during that time cluster resolved about 20 sequential client ops with avg latency 7 sec.
We tried to fix that behavior by prioritizing peering of placement groups awaited by clients (see patch http://pastebin.com/PR8DmAR1) but for some unknown reasons it totally broke peering and caused marking all placement groups as staled. Will be very grateful for any suggestions for fixing the peering problem.

Igur, a main problem is that we are writing out the pg epoch to the filestore when we don't need to. The second problem is that peering involves talking to peers. Prioritizing peering messages for pgs with waiting clients will have a side effect of delaying messages on stray pgs which may be blocking the primary on another OSD from going active.

Because your comparer depends on mutable state under the qlock without holding the qlock, it may reorder the list in surprising ways.

Also, that list is only the list to be processed in a single _process() call, the larger peering_wq is what you would need to prioritize to see a difference.

I think the best approach though is to decrease peering cost over all.

Faidon, it shouldn't affect cuttlefish as much, though that is not clear. That patch would need to be installed on all OSDs. It's not well tested yet, so you probably only want to use it on a test cluster at this time.

I updated the branch to sha1: 99831601474eddd15da72aaff421be83c02eaf46.

I just want to add that I am definitely seeing this behavior on Cuttlefish. We run a number of VM's atop RBD. Any time peering is triggered it stalls I/O on all the VM's. Just now I restarted a single OSD (out of 16) and had I/O stall for over 6 minutes:

[WRN] 7 slow requests, 1 included below; oldest blocked for > 401.472296 secs

Once peering is (mostly?) complete, I/O to the VM's continues as normal. Recovery of degraded objects doesn't have nearly the same impact, it proceeds without problem once the big peering delay is over.

This makes the RBD-backed VM's seem fragile since any time we do any maintenance they go out to lunch. Would love to see a solution for this in a future Cuttlefish patch.

Some more details about my setup:
Hosts are CentOS 6.4 + elrepo kernel-ml. Ceph is cuttlefish (0.61.2) from official packages. OSD's are on rotating disks using btrfs, with journals on LVM volumes on SSD.
The majority of traffic is from VM servers using libvirt + qemu + KVM + librbd.
This cluster was built with bobtail originally, and I recall seeing similar behavior before the upgrade to cuttlefish, just not as pronounced.
Anecdotally, it seems like the delays are growing with the amount of data in the cluster or pool.
Let me know what other information I can provide that would be helpful.

I've uploaded slowpeer-ceph-osd.2.log.bz2 (--debug-ms=1 --debug-filestore=5 --debug-osd=20) & slowpeer-osd2-ceph.log from an OSD restart to cephdrop. The peering phase didn't last long this time (30s) but there were a slow requests with pgs in "active" (not active+clean) & active+recovery_wait.

I updated my cluster from 0.61.2 to 0.61.3 and can tell a noticeable improvement. There are still some I/O stalls while PG's are peering but it doesn't take nearly as long so the impact is much less. I'm not afraid to restart hosts or daemons any more, at least. Further improvement to the time it takes to peer and/or the ability to process I/O (especially writes) while peering would still be welcome.

Just for the record:

We did a troubleshooting/log collecting session with Sam last week. It seems that the issue is apparent when an OSD has been running for a while, i.e. the first restart always restarts in slow peering (ranging from 40s to 2min+), while the subsequent ones peer instanteously. We tested a few ideas and the outcome (to my understanding) is that it is a local issue and non-pagecache-related. Sam had a log theory, which would be solved in a refactoring -- I'm unsure if this is still the prevalent theory.