Project

General

Profile

Bug #3752

fsync-tester script need to be fixed to run in the nightlies

Added by Tamilarasi muthamizhan about 11 years ago. Updated about 11 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

log: ubuntu@teuthology:/a/teuthology-2013-01-05_22:28:52-regression-next-testing-basic/35949

35949: (190s) collection:kernel-cephfs clusters:fixed-3.yaml fs:btrfs.yaml tasks:kclient_workunit_suites_fsync.yaml
Command failed with status 1: 'mkdir p - /tmp/cephtest/mnt.0/client.0/tmp && cd -- /tmp/cephtest/mnt.0/client.0/tmp && CEPH_REF=ce49968938ca3636f48fe543111aa219f36914d8 PATH="$PATH:/tmp/cephtest/binary/usr/local/bin" LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/cephtest/binary/usr/local/lib" CEPH_CONF="/tmp/cephtest/ceph.conf" CEPH_SECRET_FILE="/tmp/cephtest/data/client.0.secret" CEPH_ID="0" PYTHONPATH="$PYTHONPATH:/tmp/cephtest/binary/usr/local/lib/python2.7/dist-packages:/tmp/cephtest/binary/usr/local/lib/python2.6/dist-packages" /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/workunit.client.0/suites/fsync-tester.sh'

2013-01-06T04:31:01.782 INFO:teuthology.task.workunit.client.0.err:+ ./fsync-tester
2013-01-06T04:31:01.785 INFO:teuthology.task.workunit.client.0.err:open: Permission denied
2013-01-06T04:31:01.786 INFO:teuthology.task.workunit:Stopping suites/fsync-tester.sh on client.0...
2013-01-06T04:31:01.787 DEBUG:teuthology.orchestra.run:Running: 'rm -rf -- /tmp/cephtest/workunits.list /tmp/cephtest/workunit.client.0'
2013-01-06T04:31:01.805 ERROR:teuthology.run_tasks:Saw exception from tasks
Traceback (most recent call last):
  File "/var/lib/teuthworker/teuthology-next/teuthology/run_tasks.py", line 25, in run_tasks
    manager = _run_one_task(taskname, ctx=ctx, config=config)
  File "/var/lib/teuthworker/teuthology-next/teuthology/run_tasks.py", line 14, in _run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/teuthology-next/teuthology/task/workunit.py", line 90, in task
    _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), config.get('subdir'))
  File "/var/lib/teuthworker/teuthology-next/teuthology/task/workunit.py", line 130, in _spawn_on_all_clients
    p.spawn(_run_tests, ctx, refspec, role, [unit], env, subdir)
  File "/var/lib/teuthworker/teuthology-next/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/var/lib/teuthworker/teuthology-next/teuthology/parallel.py", line 100, in next
    resurrect_traceback(result)
  File "/var/lib/teuthworker/teuthology-next/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/teuthology-next/teuthology/task/workunit.py", line 222, in _run_tests
    args=args,
  File "/var/lib/teuthworker/teuthology-next/teuthology/orchestra/remote.py", line 40, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/var/lib/teuthworker/teuthology-next/teuthology/orchestra/run.py", line 257, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/var/lib/teuthworker/teuthology-next/teuthology/orchestra/run.py", line 253, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status)
CommandFailedError: Command failed with status 1: 'mkdir -p -- /tmp/cephtest/mnt.0/client.0/tmp && cd -- /tmp/cephtest/mnt.0/client.0/tmp && CEPH_REF=ce49968938ca3636f48fe543111aa219f36914d8 PATH="$PATH:/tmp/cephtest/binary/usr/local/bin" LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/cephtest/binary/usr/local/lib" CEPH_CONF="/tmp/cephtest/ceph.conf" CEPH_SECRET_FILE="/tmp/cephtest/data/client.0.secret" CEPH_ID="0" PYTHONPATH="$PYTHONPATH:/tmp/cephtest/binary/usr/local/lib/python2.7/dist-packages:/tmp/cephtest/binary/usr/local/lib/python2.6/dist-packages" /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/workunit.client.0/suites/fsync-tester.sh'
2013-01-06T04:31:01.807 DEBUG:teuthology.run_tasks:Unwinding manager <contextlib.GeneratorContextManager object at 0x1af6210>
2013-01-06T04:31:01.807 INFO:teuthology.task.kclient:Unmounting kernel clients...
2013-01-06T04:31:01.807 DEBUG:teuthology.task.kclient:Unmounting client client.0...
2013-01-06T04:31:01.808 DEBUG:teuthology.orchestra.run:Running: 'sudo umount /tmp/cephtest/mnt.0'
2013-01-06T04:31:04.485 DEBUG:teuthology.orchestra.run:Running: 'rmdir -- /tmp/cephtest/mnt.0'
2013-01-06T04:31:04.493 DEBUG:teuthology.run_tasks:Unwinding manager <contextlib.GeneratorContextManager object at 0x1a36a90>
2013-01-06T04:31:04.493 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/var/lib/teuthworker/teuthology-next/teuthology/contextutil.py", line 27, in nested
    yield vars
  File "/var/lib/teuthworker/teuthology-next/teuthology/task/ceph.py", line 1091, in task
    yield
  File "/var/lib/teuthworker/teuthology-next/teuthology/run_tasks.py", line 25, in run_tasks
    manager = _run_one_task(taskname, ctx=ctx, config=config)
  File "/var/lib/teuthworker/teuthology-next/teuthology/run_tasks.py", line 14, in _run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/teuthology-next/teuthology/task/workunit.py", line 90, in task
    _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), config.get('subdir'))
  File "/var/lib/teuthworker/teuthology-next/teuthology/task/workunit.py", line 130, in _spawn_on_all_clients
    p.spawn(_run_tests, ctx, refspec, role, [unit], env, subdir)
  File "/var/lib/teuthworker/teuthology-next/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/var/lib/teuthworker/teuthology-next/teuthology/parallel.py", line 100, in next
    resurrect_traceback(result)
  File "/var/lib/teuthworker/teuthology-next/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/teuthology-next/teuthology/task/workunit.py", line 222, in _run_tests
    args=args,
  File "/var/lib/teuthworker/teuthology-next/teuthology/orchestra/remote.py", line 40, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/var/lib/teuthworker/teuthology-next/teuthology/orchestra/run.py", line 257, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/var/lib/teuthworker/teuthology-next/teuthology/orchestra/run.py", line 253, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status)
CommandFailedError: Command failed with status 1: 'mkdir -p -- /tmp/cephtest/mnt.0/client.0/tmp && cd -- /tmp/cephtest/mnt.0/client.0/tmp && CEPH_REF=ce49968938ca3636f48fe543111aa219f36914d8 PATH="$PATH:/tmp/cephtest/binary/usr/local/bin" LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/cephtest/binary/usr/local/lib" CEPH_CONF="/tmp/cephtest/ceph.conf" CEPH_SECRET_FILE="/tmp/cephtest/data/client.0.secret" CEPH_ID="0" PYTHONPATH="$PYTHONPATH:/tmp/cephtest/binary/usr/local/lib/python2.7/dist-packages:/tmp/cephtest/binary/usr/local/lib/python2.6/dist-packages" /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/workunit.client.0/suites/fsync-tester.sh'

History

#1 Updated by Sam Lang about 11 years ago

I made a change to the cfuse task to chmod 1777 the ceph root dir after its mounted. I think we should do the same for the kclient task to resolve issues like this.

#2 Updated by Greg Farnum about 11 years ago

  • Status changed from New to 7

This is weird. Tamil says this one has never passed, but we can both run it locally fine and it passes in the ceph-fuse variant (this is a kernel mount that's failing).
I don't see how it can be a straightforward permissions issue since the task is doing a wget and an executable build before it fails while running the executable, but I've changed the task to do the same permissions settings as the FUSE task does (f17847e537802671c6f90bd1a0cdaa0e9d1e6f7a) and we'll see if that helps.

#3 Updated by Sage Weil about 11 years ago

There was a bug in the kernel for o_creat permissions checking for non root users.. Its fixed in the testing branch. There were several days where testing wasn't getting built due to gitbyilder issues.. But I think it should work now with master Ceph.git and testing kclient

#4 Updated by Greg Farnum about 11 years ago

Should we revert that teuthology commit, then?

#5 Updated by Sage Weil about 11 years ago

I think so.

But first let's verify it passes.

#6 Updated by Greg Farnum about 11 years ago

  • Status changed from 7 to Resolved

This appears to be passing now, right Tamil?

Since I'm not seeing anything else breaking I'm inclined to leave the chmod patch in.

#7 Updated by Tamilarasi muthamizhan about 11 years ago

Yes, Greg. The test passed in the recent runs.

#8 Updated by Greg Farnum about 11 years ago

Josh just pinged me that there was a typo in the chmod patch, and nobody's noticed so apparently it still hasn't been run.
Given Sage's belief we don't need it, though, I just reverted the patch. If we do see issues with this again at some point, please re-open and ping me.

Also available in: Atom PDF