Actions
Bug #48772
openqa: pjd: not ok 9, 44, 80
Status:
Need More Info
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
kceph, qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2021-01-06T01:34:10.018 INFO:tasks.workunit.client.0.smithi169.stderr:+ ntest=80 2021-01-06T01:34:10.019 INFO:tasks.workunit.client.0.smithi169.stderr:+ expect u::r--,u:root:r-x,u:65531:rwx,g::r--,g:root:rw-,g:65532:rwx,m::rwx,o::r-- getfacl fstest_f71e28af0da52e932b49b4529262bdc3/fstest_517a816e470f636c5b43cc0e37eced1e access 2021-01-06T01:34:10.019 INFO:tasks.workunit.client.0.smithi169.stderr:+ e=u::r--,u:root:r-x,u:65531:rwx,g::r--,g:root:rw-,g:65532:rwx,m::rwx,o::r-- 2021-01-06T01:34:10.019 INFO:tasks.workunit.client.0.smithi169.stderr:+ shift 2021-01-06T01:34:10.022 INFO:tasks.workunit.client.0.smithi169.stderr:++ /home/ubuntu/cephtest/mnt.0/client.0/tmp/tmp/../pjd-fstest-20090130-RC/tests/xacl/../../fstest getfacl fstest_f71e28af0da52e932b49b4529262bdc3/fstest_517a816e470f636c5b43cc0e37eced1e access 2021-01-06T01:34:10.023 INFO:tasks.workunit.client.0.smithi169.stderr:++ tail -1 2021-01-06T01:34:10.027 INFO:tasks.workunit.client.0.smithi169.stderr:+ r=u::r--,u:root:r-x,g::r--,g:root:rw-,m::r-x,o::r-- 2021-01-06T01:34:10.028 INFO:tasks.workunit.client.0.smithi169.stderr:+ echo u::r--,u:root:r-x,g::r--,g:root:rw-,m::r-x,o::r-- 2021-01-06T01:34:10.029 INFO:tasks.workunit.client.0.smithi169.stderr:+ egrep '^u::r--,u:root:r-x,u:65531:rwx,g::r--,g:root:rw-,g:65532:rwx,m::rwx,o::r--$' 2021-01-06T01:34:10.036 INFO:tasks.workunit.client.0.smithi169.stderr:+ '[' 1 -eq 0 ']' 2021-01-06T01:34:10.037 INFO:tasks.workunit.client.0.smithi169.stderr:+ echo 'not ok 80' 2021-01-06T01:34:10.037 INFO:tasks.workunit.client.0.smithi169.stderr:++ expr 80 + 1 2021-01-06T01:34:10.041 INFO:tasks.workunit.client.0.smithi169.stderr:+ ntest=81 2021-01-06T01:34:10.041 INFO:tasks.workunit.client.0.smithi169.stderr:+ rm -rf fstest_f71e28af0da52e932b49b4529262bdc3
From: /ceph/teuthology-archive/pdonnell-2021-01-06_00:07:44-fs:workload-wip-pdonnell-testing-20210105.221014-distro-basic-smithi/5758001/teuthology.log
and
2021-01-06T02:26:04.667 INFO:tasks.workunit.client.0.smithi132.stderr:+ ntest=9 2021-01-06T02:26:04.667 INFO:tasks.workunit.client.0.smithi132.stderr:+ expect u::r--,u:65531:r-x,g::r--,g:65532:rw-,m::r-x,o::r-- getfacl fstest_7937197fb603abc921a23974e436a365/fstest_57353644066348182b4c987fbc698b61 access 2021-01-06T02:26:04.667 INFO:tasks.workunit.client.0.smithi132.stderr:+ e=u::r--,u:65531:r-x,g::r--,g:65532:rw-,m::r-x,o::r-- 2021-01-06T02:26:04.667 INFO:tasks.workunit.client.0.smithi132.stderr:+ shift 2021-01-06T02:26:04.670 INFO:tasks.workunit.client.0.smithi132.stderr:++ /home/ubuntu/cephtest/mnt.0/client.0/tmp/tmp/../pjd-fstest-20090130-RC/tests/xacl/../../fstest getfacl fstest_7937197fb603abc921a23974e436a365/fstest_57353644066348182b4c987fbc698b61 access 2021-01-06T02:26:04.670 INFO:tasks.workunit.client.0.smithi132.stderr:++ tail -1 2021-01-06T02:26:04.673 INFO:tasks.workunit.client.0.smithi132.stderr:+ r=u::r--,g::r-x,o::r-- 2021-01-06T02:26:04.675 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo u::r--,g::r-x,o::r-- 2021-01-06T02:26:04.675 INFO:tasks.workunit.client.0.smithi132.stderr:+ egrep '^u::r--,u:65531:r-x,g::r--,g:65532:rw-,m::r-x,o::r--$' 2021-01-06T02:26:04.682 INFO:tasks.workunit.client.0.smithi132.stderr:+ '[' 1 -eq 0 ']' 2021-01-06T02:26:04.683 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 'not ok 9' ... 2021-01-06T02:26:44.670 INFO:tasks.workunit.client.0.smithi132.stderr:+ ntest=44 2021-01-06T02:26:44.670 INFO:tasks.workunit.client.0.smithi132.stderr:+ expect u::r--,u:root:r-x,g::r--,g:root:rw-,m::r-x,o::r-- getfacl fstest_7937197fb603abc921a23974e436a365/fstest_57353644066348182b4c987fbc698b61 access 2021-01-06T02:26:44.671 INFO:tasks.workunit.client.0.smithi132.stderr:+ e=u::r--,u:root:r-x,g::r--,g:root:rw-,m::r-x,o::r-- 2021-01-06T02:26:44.671 INFO:tasks.workunit.client.0.smithi132.stderr:+ shift 2021-01-06T02:26:44.673 INFO:tasks.workunit.client.0.smithi132.stderr:++ /home/ubuntu/cephtest/mnt.0/client.0/tmp/tmp/../pjd-fstest-20090130-RC/tests/xacl/../../fstest getfacl fstest_7937197fb603abc921a23974e436a365/fstest_57353644066348182b4c987fbc698b61 access 2021-01-06T02:26:44.673 INFO:tasks.workunit.client.0.smithi132.stderr:++ tail -1 2021-01-06T02:26:44.681 INFO:tasks.workunit.client.0.smithi132.stderr:+ r=u::r--,u:65531:r-x,g::r--,g:65532:rw-,m::r-x,o::r-- 2021-01-06T02:26:44.682 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo u::r--,u:65531:r-x,g::r--,g:65532:rw-,m::r-x,o::r-- 2021-01-06T02:26:44.683 INFO:tasks.workunit.client.0.smithi132.stderr:+ egrep '^u::r--,u:root:r-x,g::r--,g:root:rw-,m::r-x,o::r--$' 2021-01-06T02:26:44.690 INFO:tasks.workunit.client.0.smithi132.stderr:+ '[' 1 -eq 0 ']' 2021-01-06T02:26:44.691 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 'not ok 44' ... 2021-01-06T02:27:24.717 INFO:tasks.workunit.client.0.smithi132.stderr:+ ntest=80 2021-01-06T02:27:24.717 INFO:tasks.workunit.client.0.smithi132.stderr:+ expect u::r--,u:root:r-x,u:65531:rwx,g::r--,g:root:rw-,g:65532:rwx,m::rwx,o::r-- getfacl fstest_7937197fb603abc921a23974e436a365/fstest_57353644066348182b4c987fbc698b61 access 2021-01-06T02:27:24.717 INFO:tasks.workunit.client.0.smithi132.stderr:+ e=u::r--,u:root:r-x,u:65531:rwx,g::r--,g:root:rw-,g:65532:rwx,m::rwx,o::r-- 2021-01-06T02:27:24.717 INFO:tasks.workunit.client.0.smithi132.stderr:+ shift 2021-01-06T02:27:24.720 INFO:tasks.workunit.client.0.smithi132.stderr:++ /home/ubuntu/cephtest/mnt.0/client.0/tmp/tmp/../pjd-fstest-20090130-RC/tests/xacl/../../fstest getfacl fstest_7937197fb603abc921a23974e436a365/fstest_57353644066348182b4c987fbc698b61 access 2021-01-06T02:27:24.720 INFO:tasks.workunit.client.0.smithi132.stderr:++ tail -1 2021-01-06T02:27:24.725 INFO:tasks.workunit.client.0.smithi132.stderr:+ r=u::r--,u:root:r-x,g::r--,g:root:rw-,m::r-x,o::r-- 2021-01-06T02:27:24.726 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo u::r--,u:root:r-x,g::r--,g:root:rw-,m::r-x,o::r-- 2021-01-06T02:27:24.727 INFO:tasks.workunit.client.0.smithi132.stderr:+ egrep '^u::r--,u:root:r-x,u:65531:rwx,g::r--,g:root:rw-,g:65532:rwx,m::rwx,o::r--$' 2021-01-06T02:27:24.734 INFO:tasks.workunit.client.0.smithi132.stderr:+ '[' 1 -eq 0 ']' 2021-01-06T02:27:24.734 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 'not ok 80'
From: /ceph/teuthology-archive/pdonnell-2021-01-06_00:07:44-fs:workload-wip-pdonnell-testing-20210105.221014-distro-basic-smithi/5758099/teuthology.log
Exception for easy googling:
2021-01-06T02:27:29.722 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210105.221014/qa/tasks/workunit.py", line 147, in task
cleanup=cleanup)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210105.221014/qa/tasks/workunit.py", line 297, in _spawn_on_all_clients
timeout=timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210105.221014/qa/tasks/workunit.py", line 425, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/pjd.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0ed162b51928c50f20cee111f8292828eda755e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2021-01-06T02:27:29.751 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7fe7154b2af7469e9c31a4f9b3837336
Common config for the two test is in kernel testing branch, which may be related.
Updated by Patrick Donnelly over 3 years ago
- Status changed from New to Triaged
Both tests are with the kernel's testing branch.
Updated by Patrick Donnelly over 3 years ago
Also k-stock:
Failure: Command failed (workunit test suites/pjd.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e9d58097ffdbebcb80f58de46bac71db7077a422 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2 jobs: ['5773725', '5773757']
suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/pjd}}']
suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/pjd}}']
Updated by Patrick Donnelly over 3 years ago
- Target version changed from v16.0.0 to v17.0.0
- Backport set to pacific,octopus,nautilus
Updated by Ramana Raja over 2 years ago
Latest instance of failure here, https://pulpito.ceph.com/yuriw-2021-08-31_15:01:19-fs-wip-yuri8-testing-2021-08-30-0930-pacific-distro-basic-smithi/6368826/
Updated by Patrick Donnelly over 2 years ago
- Status changed from Triaged to Need More Info
/ceph/teuthology-archive/pdonnell-2021-10-19_04:32:14-fs-wip-pdonnell-testing-20211019.013028-distro-basic-smithi/6450334/teuthology.log
Actions