Actions
Bug #19698
closedcephtool/test.sh error on full tests
Status:
Closed
Priority:
Immediate
Assignee:
David Zafman
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2017-04-19T18:02:20.376 INFO:tasks.workunit.client.0.smithi132.stdout:HEALTH_ERR Full ratio(s) out of order 2017-04-19T18:02:20.376 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1430: test_mon_pg: ceph health detail 2017-04-19T18:02:20.377 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1430: test_mon_pg: grep 'backfill_ratio (0.912) < nearfull_ratio (0.913), increased' 2017-04-19T18:02:20.671 INFO:tasks.workunit.client.0.smithi132.stdout:backfill_ratio (0.912) < nearfull_ratio (0.913), increased 2017-04-19T18:02:20.671 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1431: test_mon_pg: ceph osd set-nearfull-ratio .892 2017-04-19T18:02:22.316 INFO:tasks.workunit.client.0.smithi132.stderr:osd set-nearfull-ratio 0.892 2017-04-19T18:02:22.336 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1432: test_mon_pg: ceph osd set-backfillfull-ratio .963 2017-04-19T18:02:24.873 INFO:tasks.workunit.client.0.smithi132.stderr:osd set-backfillfull-ratio 0.963 2017-04-19T18:02:24.889 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1433: test_mon_pg: ceph health detail 2017-04-19T18:02:24.889 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1433: test_mon_pg: grep 'full_ratio (0.962) < backfillfull_ratio (0.963), increased' 2017-04-19T18:02:25.193 INFO:tasks.workunit.client.0.smithi132.stdout:full_ratio (0.962) < backfillfull_ratio (0.963), increased 2017-04-19T18:02:25.193 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1434: test_mon_pg: ceph osd set-backfillfull-ratio .912 2017-04-19T18:02:27.417 INFO:tasks.workunit.client.0.smithi132.stderr:osd set-backfillfull-ratio 0.912 2017-04-19T18:02:27.432 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1437: test_mon_pg: WAITFORFULL=10 2017-04-19T18:02:27.432 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1438: test_mon_pg: ceph --admin-daemon /osd.0.asok injectfull nearfull 2017-04-19T18:02:27.523 INFO:tasks.workunit.client.0.smithi132.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2017-04-19T18:02:27.533 INFO:tasks.workunit.client.0.smithi132.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1: test_mon_pg: rm -fr /tmp/cephtool.ot9 2017-04-19T18:02:27.535 INFO:tasks.workunit:Stopping ['cephtool'] on client.0... 2017-04-19T18:02:27.536 INFO:teuthology.orchestra.run.smithi132:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2017-04-19T18:02:27.709 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing2_2017_4_20_2/qa/tasks/workunit.py", line 186, in task config.get('subdir'), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing2_2017_4_20_2/qa/tasks/workunit.py", line 334, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing2_2017_4_20_2/qa/tasks/workunit.py", line 450, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 193, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 414, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 149, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 171, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi132 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=wip-sage-testing2_2017_4_20_2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2017-04-19T18:02:27.739 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=c7f6f9ff57504bab81b759fc0f1bbe35/a/yuriw-2017-04-19_16:55:52-rados-wip-sage-testing2_2017_4_20_2-distro-basic-smithi/1045082
Updated by David Zafman about 7 years ago
- Status changed from New to In Progress
- Assignee set to David Zafman
Updated by David Zafman almost 7 years ago
- Status changed from In Progress to 7
Updated by David Zafman almost 7 years ago
- Status changed from 7 to Pending Backport
- Backport set to jewel, kraken
Updated by David Zafman almost 7 years ago
- Related to Bug #15912: An OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed added
Updated by David Zafman almost 7 years ago
- Related to Bug #19682: Additional full fixes added
Updated by David Zafman almost 7 years ago
- Status changed from Pending Backport to Closed
This only needs to be backported if we decide to do #19682.
Updated by David Zafman almost 7 years ago
- Related to deleted (Bug #15912: An OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed)
Actions