Bug #57605
openrbd/test_librbd_python.sh: cluster [WRN] pool 'test-librbd-smithi137-24673-7' is full (reached quota's max_bytes: 10 MiB)
0%
Description
Caught during a quincy p2p upgrade test:
https://pulpito.ceph.com/yuriw-2022-09-15_18:01:12-upgrade:quincy-p2p:quincy-p2p-stress-split-quincy-release-distro-default-smithi
2022-09-16T01:40:25.340 INFO:teuthology.orchestra.run.smithi111.stdout:2022-09-16T00:36:16.065933+0000 mon.a (mon.0) 7199 : cluster [WRN] pool 'test-librbd-smithi137-24673-7' is full (reached quota's max_bytes: 10 MiB)
Fails during rbd/test_librbd_python.sh 7-workload.
Updated by Ilya Dryomov over 1 year ago
- Status changed from New to In Progress
- Assignee set to Christopher Hoffman
Updated by Christopher Hoffman over 1 year ago
- Status changed from In Progress to Fix Under Review
- Pull request ID set to 48288
Updated by Ilya Dryomov over 1 year ago
Chris, could you please add your timestamp analysis pointing away from test_librbd_python.sh and to test_librbd.sh here?
Updated by Christopher Hoffman over 1 year ago
The librbd api test "RemoveFullTry" runs here:
2022-09-16T00:35:56.053 INFO:tasks.workunit.client.0.smithi137.stdout:[ RUN ] TestLibRBD.RemoveFullTry
2022-09-16T00:36:17.094 INFO:tasks.workunit.client.0.smithi137.stdout:[ OK ] TestLibRBD.RemoveFullTry (21059 ms)
Then the task ceph that fails is here:
2022-09-16T01:40:25.248 INFO:tasks.ceph:Checking cluster log for badness...
2022-09-16T01:40:25.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(MGR_DOWN\)' | egrep -v 'but it is still running' | egrep -v 'wrongly marked me down' | egrep -v 'objects unfound and apparently lost' | egrep -v 'log bound mismatch' | egrep -v 'failed to encode map' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | head -n 1
2022-09-16T01:40:25.340 INFO:teuthology.orchestra.run.smithi111.stdout:2022-09-16T00:36:16.065933+0000 mon.a (mon.0) 7199 : cluster [WRN] pool 'test-librbd-smithi137-24673-7' is full (reached quota's max_bytes: 10 MiB)
The timestamp for WRN is at 2022-09-16T00:36:16.065933. RemoveFullTry via remove_full_try() sets max_bytes to 10M during this test.