Project

General

Profile

Bug #24177

qa: fsstress workunit does not execute in parallel on same host without clobbering files

Added by Patrick Donnelly almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
High
Category:
Testing
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
mimic,luminous
Regression:
No
Severity:
4 - irritation
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa, task(easy)
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2018-05-18T02:15:04.481 INFO:tasks.workunit:Running workunit suites/fsstress.sh...
2018-05-18T02:15:04.481 INFO:teuthology.orchestra.run.smithi138:Running (workunit test suites/fsstress.sh): 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=wip-pdonnell-testing-20180516.034822 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2018-05-18T02:15:04.552 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1...
2018-05-18T02:15:04.552 INFO:tasks.workunit:Running workunit suites/fsstress.sh...
2018-05-18T02:15:04.553 INFO:teuthology.orchestra.run.smithi138:Running (workunit test suites/fsstress.sh): 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=wip-pdonnell-testing-20180516.034822 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'
2018-05-18T02:15:07.632 INFO:tasks.workunit.client.1.smithi138.stdout:cc  -DNO_XFS -I/home/ubuntu/cephtest/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib  fsstress.c   -o fsstress
2018-05-18T02:15:07.791 INFO:tasks.workunit.client.0.smithi138.stderr:rm: cannot remove ‘/home/ubuntu/cephtest/fsstress/ltp-full.tgz’: No such file or directory
2018-05-18T02:15:07.798 INFO:tasks.workunit.client.0.smithi138.stdout:cc  -DNO_XFS -I/home/ubuntu/cephtest/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib  fsstress.c   -o fsstress
2018-05-18T02:15:07.853 INFO:tasks.workunit.client.1.smithi138.stdout:Starting fsstress /home/ubuntu/cephtest/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d fsstress-smithi13818909 -l 1 -n 1000 -p 10 -v
2018-05-18T02:15:07.934 INFO:tasks.workunit.client.1.smithi138.stderr:/home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh: line 20: /home/ubuntu/cephtest/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress: Permission denied
2018-05-18T02:15:07.968 INFO:tasks.workunit.client.0.smithi138.stdout:Starting fsstress /home/ubuntu/cephtest/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d fsstress-smithi13818915 -l 1 -n 1000 -p 10 -v
2018-05-18T02:15:08.294 INFO:tasks.workunit.client.0.smithi138.stdout:seed = 1526373927
2018-05-18T02:15:08.318 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1...
2018-05-18T02:15:08.319 INFO:teuthology.orchestra.run.smithi138:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1'
2018-05-18T02:15:08.531 INFO:tasks.workunit.client.0.smithi138.stdout:0/0: rename - no filename

From: /ceph/teuthology-archive/pdonnell-2018-05-18_01:20:14-multimds-wip-pdonnell-testing-20180516.034822-testing-basic-smithi/2545327/teuthology.log

I'm thinking the right approach here is to have workunit.py allocate a temporary directory as qa/workunits/suites/fsstress.sh already tries to operate in ${TESTDIR}.


Related issues

Related to CephFS - Bug #35829: qa: workunits/fs/misc/acl.sh failure from unexpected system.posix_acl_default attribute Rejected 09/06/2018
Related to CephFS - Bug #36165: qa: Command failed on smithi189 with status 1: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 /home/ubuntu/cephtest/mnt.0/client.0/tmp' Resolved 09/24/2018
Copied to CephFS - Backport #36152: luminous: qa: fsstress workunit does not execute in parallel on same host without clobbering files Resolved
Copied to CephFS - Backport #36153: mimic: qa: fsstress workunit does not execute in parallel on same host without clobbering files Resolved

History

#1 Updated by Patrick Donnelly almost 6 years ago

  • Description updated (diff)

#2 Updated by Jeff Layton almost 6 years ago

I suspect the problem is in unpacking and building ltp. The fsstress commands already use a pid-specific directory. Hmm also, it looks like the mkdir command has a non-matching directory name. Might be best to make a $fsstress_dir variable and pass it to both?

command="${BIN_PATH} -d fsstress-`hostname`$$ -l 1 -n 1000 -p 10 -v" 
echo "Starting fsstress $command" 
mkdir fsstress`hostname`-$$
$command

#3 Updated by Patrick Donnelly over 5 years ago

/ceph/teuthology-archive/yuriw-2018-08-04_04:21:34-multimds-wip-yuri5-testing-2018-08-03-2359-luminous-testing-basic-smithi/2864120/teuthology.log

#4 Updated by Patrick Donnelly over 5 years ago

  • Related to Bug #35829: qa: workunits/fs/misc/acl.sh failure from unexpected system.posix_acl_default attribute added

#5 Updated by Patrick Donnelly over 5 years ago

  • Status changed from New to Fix Under Review

#6 Updated by Patrick Donnelly over 5 years ago

  • Status changed from Fix Under Review to Pending Backport

#7 Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #36152: luminous: qa: fsstress workunit does not execute in parallel on same host without clobbering files added

#8 Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #36153: mimic: qa: fsstress workunit does not execute in parallel on same host without clobbering files added

#9 Updated by Patrick Donnelly over 5 years ago

  • Related to Bug #36165: qa: Command failed on smithi189 with status 1: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 /home/ubuntu/cephtest/mnt.0/client.0/tmp' added

#10 Updated by Nathan Cutler over 5 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF