Project

General

Profile

Bug #39437

osd: PriorityCache.cc: 265: FAILED ceph_assert(mem_avail >= 0)

Added by Patrick Donnelly about 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:

Description

2019-04-20T18:30:11.748 INFO:tasks.ceph.osd.4.smithi079.stderr:/build/ceph-15.0.0-427-g609b64d/src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f2025d78700 time 2019-04-20 18:30:11.747518
2019-04-20T18:30:11.748 INFO:tasks.ceph.osd.4.smithi079.stderr:/build/ceph-15.0.0-427-g609b64d/src/common/PriorityCache.cc: 265: FAILED ceph_assert(mem_avail >= 0)
2019-04-20T18:30:11.750 INFO:tasks.ceph.osd.4.smithi079.stderr: ceph version 15.0.0-427-g609b64d (609b64dbba750855b8c0d27110f536669da3205f) octopus (dev)
2019-04-20T18:30:11.750 INFO:tasks.ceph.osd.4.smithi079.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x8504d0]
2019-04-20T18:30:11.750 INFO:tasks.ceph.osd.4.smithi079.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x8506ab]
2019-04-20T18:30:11.750 INFO:tasks.ceph.osd.4.smithi079.stderr: 3: (PriorityCache::Manager::balance()+0x417) [0xec69c7]
2019-04-20T18:30:11.750 INFO:tasks.ceph.osd.4.smithi079.stderr: 4: (BlueStore::MempoolThread::entry()+0x4ca) [0xdf1daa]
2019-04-20T18:30:11.751 INFO:tasks.ceph.osd.4.smithi079.stderr: 5: (()+0x76ba) [0x7f2033c196ba]
2019-04-20T18:30:11.751 INFO:tasks.ceph.osd.4.smithi079.stderr: 6: (clone()+0x6d) [0x7f203322041d]

/ceph/teuthology-archive/pdonnell-2019-04-17_06:07:08-multimds-wip-pdonnell-testing-20190417.032809-distro-basic-smithi/3857201/teuthology.log

Perhaps this can be resolved through tuning the OSD cache.

History

#1 Updated by Mark Nelson about 2 months ago

Patrick Donnelly wrote:

[...]

/ceph/teuthology-archive/pdonnell-2019-04-17_06:07:08-multimds-wip-pdonnell-testing-20190417.032809-distro-basic-smithi/3857201/teuthology.log

Perhaps this can be resolved through tuning the OSD cache.

That assert is over-aggressive and ends up being thrown when there's not enough memory available to even add a single "chunk" of memory per cache for growth. Instead I'll just allow it and set mem_avail to zero. This will allow the OSD to run but may result in higher than osd_memory_target memory usage in severely memory constrained scenarios.

Building a fix now.

Mark

#2 Updated by Patrick Donnelly about 2 months ago

  • Status changed from New to In Progress
  • Assignee set to Mark Nelson

Mark Nelson wrote:

Patrick Donnelly wrote:

[...]

/ceph/teuthology-archive/pdonnell-2019-04-17_06:07:08-multimds-wip-pdonnell-testing-20190417.032809-distro-basic-smithi/3857201/teuthology.log

Perhaps this can be resolved through tuning the OSD cache.

That assert is over-aggressive and ends up being thrown when there's not enough memory available to even add a single "chunk" of memory per cache for growth. Instead I'll just allow it and set mem_avail to zero. This will allow the OSD to run but may result in higher than osd_memory_target memory usage in severely memory constrained scenarios.

Building a fix now.

Mark

Great! Thanks Mark.

#4 Updated by Patrick Donnelly about 2 months ago

  • Status changed from In Progress to Need Review
  • Pull request ID set to 27763

#5 Updated by Patrick Donnelly about 2 months ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF