Project

General

Profile

Bug #57605

rbd/test_librbd_python.sh: cluster [WRN] pool 'test-librbd-smithi137-24673-7' is full (reached quota's max_bytes: 10 MiB)

Added by Laura Flores 4 months ago. Updated 4 months ago.

Status:
Fix Under Review
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Caught during a quincy p2p upgrade test:
https://pulpito.ceph.com/yuriw-2022-09-15_18:01:12-upgrade:quincy-p2p:quincy-p2p-stress-split-quincy-release-distro-default-smithi

2022-09-16T01:40:25.340 INFO:teuthology.orchestra.run.smithi111.stdout:2022-09-16T00:36:16.065933+0000 mon.a (mon.0) 7199 : cluster [WRN] pool 'test-librbd-smithi137-24673-7' is full (reached quota's max_bytes: 10 MiB)

Fails during rbd/test_librbd_python.sh 7-workload.

History

#1 Updated by Ilya Dryomov 4 months ago

  • Status changed from New to In Progress
  • Assignee set to Christopher Hoffman

#2 Updated by Christopher Hoffman 4 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 48288

#3 Updated by Ilya Dryomov 4 months ago

Chris, could you please add your timestamp analysis pointing away from test_librbd_python.sh and to test_librbd.sh here?

#4 Updated by Christopher Hoffman 4 months ago

The librbd api test "RemoveFullTry" runs here:

2022-09-16T00:35:56.053 INFO:tasks.workunit.client.0.smithi137.stdout:[ RUN ] TestLibRBD.RemoveFullTry
2022-09-16T00:36:17.094 INFO:tasks.workunit.client.0.smithi137.stdout:[ OK ] TestLibRBD.RemoveFullTry (21059 ms)

Then the task ceph that fails is here:
2022-09-16T01:40:25.248 INFO:tasks.ceph:Checking cluster log for badness...
2022-09-16T01:40:25.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(MGR_DOWN\)' | egrep -v 'but it is still running' | egrep -v 'wrongly marked me down' | egrep -v 'objects unfound and apparently lost' | egrep -v 'log bound mismatch' | egrep -v 'failed to encode map' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | head -n 1
2022-09-16T01:40:25.340 INFO:teuthology.orchestra.run.smithi111.stdout:2022-09-16T00:36:16.065933+0000 mon.a (mon.0) 7199 : cluster [WRN] pool 'test-librbd-smithi137-24673-7' is full (reached quota's max_bytes: 10 MiB)

The timestamp for WRN is at 2022-09-16T00:36:16.065933. RemoveFullTry via remove_full_try() sets max_bytes to 10M during this test.

Also available in: Atom PDF