Ceph » RADOS

This morning, I have :
PG_NOT_DEEP_SCRUBBED: 11 pgs not deep-scrubbed in time
Never had before Pacific.

Could it be those scrubs/deep scrubs who create latency ? And being slower than before ?

Gilles Mocellin wrote:

This morning, I have :
PG_NOT_DEEP_SCRUBBED: 11 pgs not deep-scrubbed in time
Never had before Pacific.

Could it be those scrubs/deep scrubs who create latency ? And being slower than before ?

Definitely it could be but that's just speculating so far.

Could you please grep osd logs for "timed out" and "slow operation observed" patterns and share relevant log if found.

Zero occurrence of "timed out" in all my ceph-osds logs for 2 days. But, as I have increased bluestore_prefer_deferred_size_hdd like I say in #56488,
I have no more halts, just some non-visible impacts with increase in latency.

Ah, something I have not said : during the high latencies, when they are impacting, I also lose many metrics throught the MGR prometheus exporter.
Which can also be tied to the MGR freezes I had while tweaking bluestore_prefer_deferred_size_hdd :
The service (systemd) and process (ceph-mgr) were still UP, but no more logs and no more dashboard an prometheus metrics.
And ceph -s showed "no mgrs".

To get some more insight on the issue I would suggest to do the following once the issue is faced again:
1) For OSD-in-question dump and store current performance counters via: ceph tell osd.N perf dump > original.perf
2) Reset that OSD perf counters: ceph tell osd.N perf reset all
3) Start collecting iostats output for relevant OSD drives, both main and DB ones.
4) Leave cluster running for e.g. 1 min
5) Dump and save counters again.
6) Share both dumps, iostat report and preferably monitor and osd logs.

Just a follow-up.

Finally, what's helping us the best is increasing osd_scrub_sleep to 0.4.

I've just let Mark and Ronen know about this issue.