Bug #65604: dbench.sh workload times out after 3h when run with-quiescer - CephFS - Ceph

Actions

Copy link

Bug #65604

open

dbench.sh workload times out after 3h when run with-quiescer

Added by Leonid Usov 13 days ago. Updated 2 days ago.

Status:

New

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

MDS, quiesce

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

https://pulpito.ceph.com/leonidus-2024-04-21_11:37:13-fs-wip-lusov-quiescer-distro-default-smithi/7666604/

No quiesce errors were seen in the run

2024-04-21T12:08:40.015 INFO:tasks.workunit:Running workunit suites/dbench.sh...
2024-04-21T12:08:40.016 DEBUG:teuthology.orchestra.run.smithi060:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=
44cf288680846a2951ce69e38174a7d3845faa8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/hom
e/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh
...
2024-04-21T15:08:39.211 INFO:tasks.workunit.client.0.smithi060.stdout:   1  cleanup 10679 sec
2024-04-21T15:08:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 124
2024-04-21T15:08:40.072 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0...

dbench has apparently locked up:

2024-04-21T12:08:40.077 INFO:tasks.workunit.client.0.smithi060.stderr:+ dbench 1
2024-04-21T12:08:40.083 INFO:tasks.workunit.client.0.smithi060.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004
2024-04-21T12:08:40.083 INFO:tasks.workunit.client.0.smithi060.stdout:
2024-04-21T12:08:40.083 INFO:tasks.workunit.client.0.smithi060.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs
2024-04-21T12:08:40.083 INFO:tasks.workunit.client.0.smithi060.stdout:0 of 1 processes prepared for launch   0 sec
2024-04-21T12:08:40.184 INFO:tasks.workunit.client.0.smithi060.stdout:1 of 1 processes prepared for launch   0 sec
2024-04-21T12:08:40.184 INFO:tasks.workunit.client.0.smithi060.stdout:releasing clients
2024-04-21T12:08:41.184 INFO:tasks.workunit.client.0.smithi060.stdout:   1      3209    34.28 MB/sec  warmup   1 sec  latency 7.321 ms
2024-04-21T12:08:42.184 INFO:tasks.workunit.client.0.smithi060.stdout:   1      5976    23.61 MB/sec  warmup   2 sec  latency 16.565 ms
2024-04-21T12:08:43.184 INFO:tasks.workunit.client.0.smithi060.stdout:   1      8537    21.42 MB/sec  warmup   3 sec  latency 18.683 ms
2024-04-21T12:08:44.184 INFO:tasks.workunit.client.0.smithi060.stdout:   1     11366    19.64 MB/sec  warmup   4 sec  latency 13.370 ms
2024-04-21T12:08:45.184 INFO:tasks.workunit.client.0.smithi060.stdout:   1     14583    19.49 MB/sec  warmup   5 sec  latency 17.577 ms
2024-04-21T12:08:46.184 INFO:tasks.workunit.client.0.smithi060.stdout:   1     17872    19.20 MB/sec  warmup   6 sec  latency 16.217 ms
...
2024-04-21T12:20:37.255 INFO:tasks.workunit.client.0.smithi060.stdout:   1     39146     0.00 MB/sec  execute 597 sec  latency 672709.544 ms
2024-04-21T12:20:38.255 INFO:tasks.workunit.client.0.smithi060.stdout:   1     39146     0.00 MB/sec  execute 598 sec  latency 673709.694 ms
2024-04-21T12:20:39.255 INFO:tasks.workunit.client.0.smithi060.stdout:   1     39146     0.00 MB/sec  execute 599 sec  latency 674709.786 ms
2024-04-21T12:20:40.255 INFO:tasks.workunit.client.0.smithi060.stdout:   1  cleanup 600 sec
2024-04-21T12:20:41.255 INFO:tasks.workunit.client.0.smithi060.stdout:   1  cleanup 601 sec
2024-04-21T12:20:42.255 INFO:tasks.workunit.client.0.smithi060.stdout:   1  cleanup 602 sec