Project

General

Profile

Bug #21897

OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled

Added by Vikhyat Umrao over 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Josh Durgin
Category:
OSD
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
luminous, jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v10.2.7
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled

Version-Release number of selected component (if applicable):
jewel

How reproducible:
Always in the RH scale lab

Additional info:

Testing in the scale lab uncovered an area where osds were
hitting heartbeat timeouts during backfill.

This turned out to be due to too many omap entries read in each 'chunk'
being backfilled, and only resetting the heartbeat once an entire chunk was assembled. We should ask users to tune this down for their
bucket index osds - setting:

Max number of omap entries per chunk:

osd_recovery_max_omap_entries_per_chunk = 1024

Josh is planning to add a patch for luminous and jewel to reset the heartbeat in middle of gathering a chunk, which should make the current default of 64000 entries per backfill chunk work even with large stores.

Related issues

Copied to Ceph - Backport #22670: jewel: OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled Resolved
Copied to Ceph - Backport #22671: luminous: OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled Resolved

History

#1 Updated by Vikhyat Umrao over 5 years ago

  • Description updated (diff)

#2 Updated by Vikhyat Umrao over 5 years ago

  • Description updated (diff)

#3 Updated by Ken Dreyer over 5 years ago

  • Description updated (diff)
  • Backport set to luminous, jewel

#4 Updated by Kefu Chai about 5 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Josh Durgin

#5 Updated by Josh Durgin about 5 years ago

  • Copied to Backport #22670: jewel: OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled added

#6 Updated by Josh Durgin about 5 years ago

  • Copied to Backport #22671: luminous: OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled added

#8 Updated by Nathan Cutler about 5 years ago

  • Status changed from Fix Under Review to Pending Backport

#9 Updated by Nathan Cutler almost 5 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF