Bug #21897
closedOSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled
0%
Description
OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled
Version-Release number of selected component (if applicable):
jewel
How reproducible:
Always in the RH scale lab
Additional info:
Testing in the scale lab uncovered an area where osds were
hitting heartbeat timeouts during backfill.
This turned out to be due to too many omap entries read in each 'chunk'
being backfilled, and only resetting the heartbeat once an entire chunk was assembled. We should ask users to tune this down for their
bucket index osds - setting:
Max number of omap entries per chunk:
osd_recovery_max_omap_entries_per_chunk = 1024
Josh is planning to add a patch for luminous and jewel to reset the heartbeat in middle of gathering a chunk, which should make the current default of 64000 entries per backfill chunk work even with large stores.
Updated by Ken Dreyer over 6 years ago
- Description updated (diff)
- Backport set to luminous, jewel
Updated by Kefu Chai over 6 years ago
- Status changed from New to Fix Under Review
- Assignee set to Josh Durgin
Updated by Josh Durgin over 6 years ago
- Copied to Backport #22670: jewel: OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled added
Updated by Josh Durgin over 6 years ago
- Copied to Backport #22671: luminous: OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled added
Updated by Yuri Weinstein over 6 years ago
https://github.com/ceph/ceph/pull/19910 was merged
Updated by Nathan Cutler about 6 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler about 6 years ago
- Status changed from Pending Backport to Resolved