Bug #50368
closedcommon/PriorityCache.cc: FAILED ceph_assert(mem_avail >= 0) in radosbench_omap_write perf tests
0%
Description
2021-04-14T02:24:55.475+0000 7f8392c7b700 10 rocksdb: request_cache_bytes Priority: 0 Request: 0 2021-04-14T02:24:55.475+0000 7f8392c7b700 10 rocksdb: request_cache_bytes Priority: 0 Request: 0 2021-04-14T02:24:55.475+0000 7f8392c7b700 10 rocksdb: request_cache_bytes Priority: 1 Request: 83597768 2021-04-14T02:24:55.475+0000 7f8392c7b700 10 rocksdb: request_cache_bytes Priority: 1 Request: 585163408 2021-04-14T02:24:55.475+0000 7f8392c7b700 10 rocksdb: request_cache_bytes Priority: 1 Request: 83597768 2021-04-14T02:24:55.475+0000 7f8392c7b700 10 rocksdb: request_cache_bytes Priority: 1 Request: 585163408 2021-04-14T02:24:55.487+0000 7f8392c7b700 -1 /build/ceph-17.0.0-3132-g8f57c508/src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f8392c7b700 time 2021-04-14T02:24:55.490010+0000 /build/ceph-17.0.0-3132-g8f57c508/src/common/PriorityCache.cc: 301: FAILED ceph_assert(mem_avail >= 0) ceph version 17.0.0-3132-g8f57c508 (8f57c508df7255bcecbb072d622570c337fdedaa) quincy (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x5588d54fd256] 2: ceph-osd(+0xc6842e) [0x5588d54fd42e] 3: (PriorityCache::Manager::balance()+0x444) [0x5588d614bc34] 4: (BlueStore::MempoolThread::entry()+0x831) [0x5588d5b8ea51] 5: (Thread::_entry_func(void*)+0x9) [0x5588d5cec609] 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f83a21096db] 7: clone()
rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_18.04 workloads/radosbench_omap_write}
/a/nojha-2021-04-14_00:54:53-rados-master-distro-basic-smithi/6044327
rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_18.04 workloads/radosbench_omap_write}
/a/gregf-2021-04-14_03:14:48-rados-wip-mon-quorum-leader-413-distro-basic-smithi/6044843
Updated by Neha Ojha about 3 years ago
- Has duplicate Bug #50397: src/common/PriorityCache.cc: 301: FAILED ceph_assert(mem_avail >= 0) added
Updated by Neha Ojha about 3 years ago
- Priority changed from Normal to High
rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_18.04 workloads/radosbench_omap_write}
/a/sage-2021-04-15_22:06:53-rados-wip-sage-testing-2021-04-15-1314-distro-basic-smithi/6049958
Updated by Neha Ojha about 3 years ago
- Priority changed from High to Urgent
rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_18.04 workloads/radosbench_omap_write}
/a/nojha-2021-04-15_20:05:27-rados-wip-50217-distro-basic-smithi/6049629
Updated by Neha Ojha about 3 years ago
The tests are consistently failing in master - https://pulpito.ceph.com/nojha-2021-04-16_15:33:38-rados:perf-wip-50217-distro-basic-smithi/
Good d9497684f7e3838c088ffdc11675bc530a49a72c - https://pulpito.ceph.com/nojha-2021-04-16_19:40:45-rados:perf-master-distro-basic-gibba/
Bad 8f57c508df7255bcecbb072d622570c337fdedaa - https://pulpito.ceph.com/nojha-2021-04-14_00:54:53-rados-master-distro-basic-smithi/6044327/
Updated by Neha Ojha about 3 years ago
- Status changed from New to Triaged
Tests are passing with fdb4f834486, the commit before https://github.com/ceph/ceph/pull/40731 merged - https://pulpito.ceph.com/nojha-2021-04-16_22:43:02-rados:perf-wip-neha-fdb4f834486-distro-basic-smithi/.
The same failure was seen during the testing of 40731 - https://pulpito.ceph.com/kchai-2021-04-12_08:01:43-rados-wip-kefu-testing-2021-04-12-1256-distro-basic-smithi/6039328/.
Hopefully, https://github.com/ceph/ceph/pull/40896 will fix it.
Updated by Neha Ojha about 3 years ago
- Status changed from Triaged to Fix Under Review
- Pull request ID set to 40896
Neha Ojha wrote:
Tests are passing with fdb4f834486, the commit before https://github.com/ceph/ceph/pull/40731 merged - https://pulpito.ceph.com/nojha-2021-04-16_22:43:02-rados:perf-wip-neha-fdb4f834486-distro-basic-smithi/.
The same failure was seen during the testing of 40731 - https://pulpito.ceph.com/kchai-2021-04-12_08:01:43-rados-wip-kefu-testing-2021-04-12-1256-distro-basic-smithi/6039328/.
Hopefully, https://github.com/ceph/ceph/pull/40896 will fix it.
Updated by Neha Ojha about 3 years ago
- Status changed from Fix Under Review to Resolved