Bug #9423
failure in client_recovery task
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2014-09-09T06:36:33.282 INFO:teuthology.orchestra.run.plana89.stderr:Traceback (most recent call last): 2014-09-09T06:36:33.283 INFO:teuthology.orchestra.run.plana89.stderr: File "<string>", line 4, in <module> 2014-09-09T06:36:33.283 INFO:teuthology.orchestra.run.plana89.stderr:IOError: [Errno 107] Transport endpoint is not connected: '/home/ubuntu/cephtest/mnt.1/background_file' 2014-09-09T06:36:33.296 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2014-09-09T06:36:33.297 INFO:teuthology.orchestra.run.plana89:Running: 'sudo daemon-helper kill python -c \'\nimport time\n\nf = open("/home/ubuntu/cephtest/mnt.0/background_file", \'"\'"\'w\'"\'"\')\nf.write(\'"\'"\'content\'"\'"\')\nf.close()\n\'' 2014-09-09T06:36:33.456 INFO:teuthology.orchestra.run.plana89.stderr:daemon-helper: command failed with exit status 1 2014-09-09T06:36:37.821 INFO:tasks.mds_client_recovery:cap_waiter waited 4.4567129612s 2014-09-09T06:36:37.840 INFO:tasks.mds_client_recovery:test_stale_caps (tasks.mds_client_recovery.TestClientRecovery) ... FAIL 2014-09-09T06:36:37.841 INFO:teuthology.orchestra.run.plana76:Running: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 2014-09-09T06:36:37.843 INFO:teuthology.orchestra.run.plana89:Running: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 2014-09-09T06:36:37.846 INFO:tasks.cephfs.mount:Terminating background process 2014-09-09T06:36:37.846 INFO:tasks.cephfs.mount:Terminating background process 2014-09-09T06:36:37.846 INFO:tasks.cephfs.mount:Terminating background process 2014-09-09T06:36:37.846 INFO:teuthology.orchestra.run.plana89:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.1" 2014-09-09T06:36:37.867 INFO:teuthology.orchestra.run.plana89.stderr:stat: cannot read file system information for `/home/ubuntu/cephtest/mnt.1': Transport endpoint is not connected 2014-09-09T06:36:37.868 INFO:teuthology.orchestra.run.plana89:Running: 'rm -rf /home/ubuntu/cephtest/mnt.1' 2014-09-09T06:36:37.940 INFO:teuthology.orchestra.run.plana89.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1': Is a directory 2014-09-09T06:36:37.952 INFO:tasks.mds_client_recovery:ERROR 2014-09-09T06:36:37.953 INFO:tasks.mds_client_recovery: 2014-09-09T06:36:37.953 INFO:tasks.mds_client_recovery:====================================================================== 2014-09-09T06:36:37.953 INFO:tasks.mds_client_recovery:ERROR: test_stale_caps (tasks.mds_client_recovery.TestClientRecovery) 2014-09-09T06:36:37.953 INFO:tasks.mds_client_recovery:---------------------------------------------------------------------- 2014-09-09T06:36:37.954 INFO:tasks.mds_client_recovery:Traceback (most recent call last): 2014-09-09T06:36:37.954 INFO:tasks.mds_client_recovery: File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 45, in tearDown 2014-09-09T06:36:37.954 INFO:tasks.mds_client_recovery: self.mount_a.teardown() 2014-09-09T06:36:37.954 INFO:tasks.mds_client_recovery: File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/cephfs/fuse_mount.py", line 221, in teardown 2014-09-09T06:36:37.955 INFO:tasks.mds_client_recovery: self.mountpoint, 2014-09-09T06:36:37.955 INFO:tasks.mds_client_recovery: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 114, in run 2014-09-09T06:36:37.955 INFO:tasks.mds_client_recovery: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2014-09-09T06:36:37.955 INFO:tasks.mds_client_recovery: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 401, in run 2014-09-09T06:36:37.956 INFO:tasks.mds_client_recovery: r.wait() 2014-09-09T06:36:37.956 INFO:tasks.mds_client_recovery: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 102, in wait 2014-09-09T06:36:37.956 INFO:tasks.mds_client_recovery: exitstatus=status, node=self.hostname) 2014-09-09T06:36:37.956 INFO:tasks.mds_client_recovery:CommandFailedError: Command failed on plana89 with status 1: 'rm -rf /home/ubuntu/cephtest/mnt.1' 2014-09-09T06:36:37.957 INFO:tasks.mds_client_recovery: 2014-09-09T06:36:37.957 INFO:tasks.mds_client_recovery:====================================================================== 2014-09-09T06:36:37.957 INFO:tasks.mds_client_recovery:FAIL: test_stale_caps (tasks.mds_client_recovery.TestClientRecovery) 2014-09-09T06:36:37.957 INFO:tasks.mds_client_recovery:---------------------------------------------------------------------- 2014-09-09T06:36:37.958 INFO:tasks.mds_client_recovery:Traceback (most recent call last): 2014-09-09T06:36:37.958 INFO:tasks.mds_client_recovery: File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 196, in test_stale_caps 2014-09-09T06:36:37.958 INFO:tasks.mds_client_recovery: cap_waited, self.mds_session_timeout 2014-09-09T06:36:37.958 INFO:tasks.mds_client_recovery:AssertionError: Capability handover took 4.4567129612, expected approx 60 2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery: 2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery:---------------------------------------------------------------------- 2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery:Ran 7 tests in 606.279s 2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery: 2014-09-09T06:36:37.959 INFO:tasks.mds_client_recovery:FAILED (failures=1, errors=1) 2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery: 2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:====================================================================== 2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:ERROR: test_stale_caps (tasks.mds_client_recovery.TestClientRecovery) 2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:---------------------------------------------------------------------- 2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery:Traceback (most recent call last): 2014-09-09T06:36:37.960 INFO:tasks.mds_client_recovery: File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 45, in tearDown 2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery: self.mount_a.teardown() 2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery: File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/cephfs/fuse_mount.py", line 221, in teardown 2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery: self.mountpoint, 2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 114, in run 2014-09-09T06:36:37.961 INFO:tasks.mds_client_recovery: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2014-09-09T06:36:37.962 INFO:tasks.mds_client_recovery: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 401, in run 2014-09-09T06:36:37.962 INFO:tasks.mds_client_recovery: r.wait() 2014-09-09T06:36:37.962 INFO:tasks.mds_client_recovery: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 102, in wait 2014-09-09T06:36:37.962 INFO:tasks.mds_client_recovery: exitstatus=status, node=self.hostname) 2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:CommandFailedError: Command failed on plana89 with status 1: 'rm -rf /home/ubuntu/cephtest/mnt.1' 2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery: 2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:====================================================================== 2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:FAIL: test_stale_caps (tasks.mds_client_recovery.TestClientRecovery) 2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:---------------------------------------------------------------------- 2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery:Traceback (most recent call last): 2014-09-09T06:36:37.963 INFO:tasks.mds_client_recovery: File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 196, in test_stale_caps 2014-09-09T06:36:37.964 INFO:tasks.mds_client_recovery: cap_waited, self.mds_session_timeout 2014-09-09T06:36:37.964 INFO:tasks.mds_client_recovery:AssertionError: Capability handover took 4.4567129612, expected approx 60 2014-09-09T06:36:37.964 INFO:tasks.mds_client_recovery: 2014-09-09T06:36:37.964 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/mds_client_recovery.py", line 408, in task raise RuntimeError("Test failure.") RuntimeError: Test failure. 2014-09-09T06:36:37.964 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2014-09-09T06:36:37.964 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2014-09-09T06:36:37.964 DEBUG:teuthology.run_tasks:Unwinding manager ceph
There are no crash logs in the archive, and while we have a bunch of client debug logs they're pretty much empty.
Associated revisions
History
#2 Updated by John Spray over 9 years ago
Had seen this previously at http://pulpito.ceph.com/teuthology-2014-09-05_23:04:02-fs-master-testing-basic-multi/470163/ -- fix is queued up on ceph-qa-suite wip-9284
#3 Updated by Greg Farnum over 9 years ago
Can we pull out the fix so we can merge it and have it run against giant going forward?
#4 Updated by John Spray over 9 years ago
- Status changed from New to 7
#5 Updated by John Spray over 9 years ago
Fixed merged to giant.
commit 09beb4c7c6cdfbb2b2db674956f040842bc87311 Merge: b1365a8 1d9101c Author: John Spray <jcspray@gmail.com> Date: Mon Sep 15 18:13:14 2014 +0100 Merge pull request #141 from ceph/wip-9423 Fix #9423 - race in mds_client_recovery
#6 Updated by John Spray over 9 years ago
- Status changed from 7 to Resolved