Project

General

Profile

Bug #9423

failure in client_recovery task

Added by Greg Farnum almost 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
09/10/2014
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:

Description

2014-09-09T06:36:33.282 INFO:teuthology.orchestra.run.plana89.stderr:Traceback (most recent call last):
2014-09-09T06:36:33.283 INFO:teuthology.orchestra.run.plana89.stderr:  File "<string>", line 4, in <module>
2014-09-09T06:36:33.283 INFO:teuthology.orchestra.run.plana89.stderr:IOError: [Errno 107] Transport endpoint is not connected: '/home/ubuntu/cephtest/mnt.1/background_file'
2014-09-09T06:36:33.296 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0
2014-09-09T06:36:33.297 INFO:teuthology.orchestra.run.plana89:Running: 'sudo daemon-helper kill python -c \'\nimport time\n\nf = open("/home/ubuntu/cephtest/mnt.0/background_file", \'"\'"\'w\'"\'"\')\nf.write(\'"\'"\'content\'"\'"\')\nf.close()\n\''
2014-09-09T06:36:33.456 INFO:teuthology.orchestra.run.plana89.stderr:daemon-helper: command failed with exit status 1
2014-09-09T06:36:37.821 INFO:tasks.mds_client_recovery:cap_waiter waited 4.4567129612s
2014-09-09T06:36:37.840 INFO:tasks.mds_client_recovery:test_stale_caps (tasks.mds_client_recovery.TestClientRecovery) ... FAIL
2014-09-09T06:36:37.841 INFO:teuthology.orchestra.run.plana76:Running: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 
2014-09-09T06:36:37.843 INFO:teuthology.orchestra.run.plana89:Running: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 
2014-09-09T06:36:37.846 INFO:tasks.cephfs.mount:Terminating background process
2014-09-09T06:36:37.846 INFO:tasks.cephfs.mount:Terminating background process
2014-09-09T06:36:37.846 INFO:tasks.cephfs.mount:Terminating background process
2014-09-09T06:36:37.846 INFO:teuthology.orchestra.run.plana89:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.1" 
2014-09-09T06:36:37.867 INFO:teuthology.orchestra.run.plana89.stderr:stat: cannot read file system information for `/home/ubuntu/cephtest/mnt.1': Transport endpoint is not connected
2014-09-09T06:36:37.868 INFO:teuthology.orchestra.run.plana89:Running: 'rm -rf /home/ubuntu/cephtest/mnt.1'
2014-09-09T06:36:37.940 INFO:teuthology.orchestra.run.plana89.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1': Is a directory
2014-09-09T06:36:37.952 INFO:tasks.mds_client_recovery:ERROR
2014-09-09T06:36:37.953 INFO:tasks.mds_client_recovery:
2014-09-09T06:36:37.953 INFO:tasks.mds_client_recovery:======================================================================
2014-09-09T06:36:37.953 INFO:tasks.mds_client_recovery:ERROR: test_stale_caps (tasks.mds_client_recovery.TestClientRecovery)
2014-09-09T06:36:37.953 INFO:tasks.mds_client_recovery:----------------------------------------------------------------------
2014-09-09T06:36:37.954 INFO:tasks.mds_client_recovery:Traceback (most recent call last):
2014-09-09T06:36:37.954 INFO:tasks.mds_client_recovery:  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 45, in tearDown
2014-09-09T06:36:37.954 INFO:tasks.mds_client_recovery:    self.mount_a.teardown()
2014-09-09T06:36:37.954 INFO:tasks.mds_client_recovery:  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/cephfs/fuse_mount.py", line 221, in teardown
2014-09-09T06:36:37.955 INFO:tasks.mds_client_recovery:    self.mountpoint,
2014-09-09T06:36:37.955 INFO:tasks.mds_client_recovery:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 114, in run
2014-09-09T06:36:37.955 INFO:tasks.mds_client_recovery:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2014-09-09T06:36:37.955 INFO:tasks.mds_client_recovery:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 401, in run
2014-09-09T06:36:37.956 INFO:tasks.mds_client_recovery:    r.wait()
2014-09-09T06:36:37.956 INFO:tasks.mds_client_recovery:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 102, in wait
2014-09-09T06:36:37.956 INFO:tasks.mds_client_recovery:    exitstatus=status, node=self.hostname)
2014-09-09T06:36:37.956 INFO:tasks.mds_client_recovery:CommandFailedError: Command failed on plana89 with status 1: 'rm -rf /home/ubuntu/cephtest/mnt.1'
2014-09-09T06:36:37.957 INFO:tasks.mds_client_recovery:
2014-09-09T06:36:37.957 INFO:tasks.mds_client_recovery:======================================================================
2014-09-09T06:36:37.957 INFO:tasks.mds_client_recovery:FAIL: test_stale_caps (tasks.mds_client_recovery.TestClientRecovery)
2014-09-09T06:36:37.957 INFO:tasks.mds_client_recovery:----------------------------------------------------------------------
2014-09-09T06:36:37.958 INFO:tasks.mds_client_recovery:Traceback (most recent call last):
2014-09-09T06:36:37.958 INFO:tasks.mds_client_recovery:  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 196, in test_stale_caps
2014-09-09T06:36:37.958 INFO:tasks.mds_client_recovery:    cap_waited, self.mds_session_timeout
2014-09-09T06:36:37.958 INFO:tasks.mds_client_recovery:AssertionError: Capability handover took 4.4567129612, expected approx 60
2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery:
2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery:----------------------------------------------------------------------
2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery:Ran 7 tests in 606.279s
2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery:
2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery:FAILED (failures=1, errors=1)
2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:
2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:======================================================================
2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:ERROR: test_stale_caps (tasks.mds_client_recovery.TestClientRecovery)
2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:----------------------------------------------------------------------
2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:Traceback (most recent call last):
2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 45, in tearDown
2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery:    self.mount_a.teardown()
2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery:  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/cephfs/fuse_mount.py", line 221, in teardown
2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery:    self.mountpoint,
2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 114, in run
2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2014-09-09T06:36:37.962 INFO:tasks.mds_client_recovery:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 401, in run
2014-09-09T06:36:37.962 INFO:tasks.mds_client_recovery:    r.wait()
2014-09-09T06:36:37.962 INFO:tasks.mds_client_recovery:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 102, in wait
2014-09-09T06:36:37.962 INFO:tasks.mds_client_recovery:    exitstatus=status, node=self.hostname)
2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:CommandFailedError: Command failed on plana89 with status 1: 'rm -rf /home/ubuntu/cephtest/mnt.1'
2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:
2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:======================================================================
2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:FAIL: test_stale_caps (tasks.mds_client_recovery.TestClientRecovery)
2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:----------------------------------------------------------------------
2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:Traceback (most recent call last):
2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 196, in test_stale_caps
2014-09-09T06:36:37.964 INFO:tasks.mds_client_recovery:    cap_waited, self.mds_session_timeout
2014-09-09T06:36:37.964 INFO:tasks.mds_client_recovery:AssertionError: Capability handover took 4.4567129612, expected approx 60
2014-09-09T06:36:37.964 INFO:tasks.mds_client_recovery:
2014-09-09T06:36:37.964 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 408, in task
    raise RuntimeError("Test failure.")
RuntimeError: Test failure.
2014-09-09T06:36:37.964 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse
2014-09-09T06:36:37.964 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients...
2014-09-09T06:36:37.964 DEBUG:teuthology.run_tasks:Unwinding manager ceph

There are no crash logs in the archive, and while we have a bunch of client debug logs they're pretty much empty.

Associated revisions

Revision 09beb4c7
Added by John Spray almost 5 years ago

Merge pull request #141 from ceph/wip-9423

Fix #9423 - race in mds_client_recovery

History

#2 Updated by John Spray almost 5 years ago

Had seen this previously at http://pulpito.ceph.com/teuthology-2014-09-05_23:04:02-fs-master-testing-basic-multi/470163/ -- fix is queued up on ceph-qa-suite wip-9284

#3 Updated by Greg Farnum almost 5 years ago

Can we pull out the fix so we can merge it and have it run against giant going forward?

#4 Updated by John Spray almost 5 years ago

  • Status changed from New to Testing

#5 Updated by John Spray almost 5 years ago

Fixed merged to giant.

commit 09beb4c7c6cdfbb2b2db674956f040842bc87311
Merge: b1365a8 1d9101c
Author: John Spray <jcspray@gmail.com>
Date:   Mon Sep 15 18:13:14 2014 +0100

    Merge pull request #141 from ceph/wip-9423

    Fix #9423 - race in mds_client_recovery

#6 Updated by John Spray almost 5 years ago

  • Status changed from Testing to Resolved

Also available in: Atom PDF