Ceph » CephFS

2020-05-13 04:04:28,374.374 INFO:tasks.cephfs.test_snapshots:testing snapserver mds_kill_mdstable_at=1
2020-05-13 04:04:28,374.374 INFO:__main__:Running ['./bin/ceph', 'fs', 'dump', '--format=json']
2020-05-13 04:04:29,142.142 INFO:__main__:Running ['./bin/ceph', 'mds', 'freeze', '44:0', 'true']
2020-05-13 04:04:30,924.924 INFO:__main__:Running ['./bin/ceph', 'daemon', 'mds.b', 'config', 'set', 'mds_kill_mdstable_at', '1']
2020-05-13 04:04:31,373.373 INFO:tasks.cephfs.filesystem:_json_asok output: b'{\n    "success": "mds_kill_mdstable_at = \'1\' (not observed, change may require restart) "\n}\n'
2020-05-13 04:04:31,375.375 INFO:__main__:Running ['sudo', 'mkdir', 'd1/dir/.snap/s11']
2020-05-13 04:04:31,396.396 INFO:__main__:Running ['./bin/ceph', 'fs', 'dump', '--format=json']
2020-05-13 04:04:37,144.144 INFO:__main__:Running ['./bin/ceph', 'fs', 'dump', '--format=json']
2020-05-13 04:04:42,929.929 INFO:__main__:Running ['./bin/ceph', 'fs', 'dump', '--format=json']
2020-05-13 04:04:48,715.715 INFO:__main__:Running ['./bin/ceph', 'fs', 'dump', '--format=json']
2020-05-13 04:04:49,477.477 INFO:__main__:Running ['sudo', 'sysctl', '-n', 'kernel.core_pattern']
2020-05-13 04:04:49,665.665 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: |/usr/libexec
2020-05-13 04:04:49,667.667 INFO:__main__:Running ['cd', '|/usr/libexec', Raw('&&'), 'sudo', 'ls', Raw('|'), 'sudo', 'xargs', 'file']
/bin/sh: line 0: cd: |/usr/libexec: No such file or directory
2020-05-13 04:04:49,701.701 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on local...
2020-05-13 04:04:49,701.701 INFO:__main__:Running ['sudo', 'fusermount', '-u', '/tmp/tmpxtz80q5f/mnt.0']
fusermount: failed to unmount /tmp/tmpxtz80q5f/mnt.0: Device or resource busy
2020-05-13 04:04:49,851.851 INFO:tasks.cephfs.fuse_mount:Failed to unmount ceph-fuse on local, aborting...
2020-05-13 04:04:49,852.852 INFO:__main__:Running ['sudo', Raw('PATH=/usr/sbin:$PATH'), 'lsof', Raw(';'), 'ps', 'auxf']
2020-05-13 04:04:50,169.169 INFO:__main__:Running ['python3', '-c', '\nimport os\npath = "/sys/fs/fuse/connections/46/abort"\nif os.path.exists(path):\n    open(path, "w").write("1")\n']
2020-05-13 04:04:50,267.267 INFO:__main__:Running ['sudo', 'umount', '-l', '-f', '/tmp/tmpxtz80q5f/mnt.0']
2020-05-13 04:04:50,459.459 INFO:__main__:kill 
2020-05-13 04:04:50,461.461 INFO:__main__:kill: killing pid 723 (['nsenter', '--net=/var/run/netns/ceph-ns--tmp-tmpxtz80q5f-mnt.0', './bin/ceph-fuse', '-f', '--name', 'client.0', '/tmp/tmpxtz80q5f/mnt.0'])
2020-05-13T04:04:07.578-0400 7f2aef9c2f80 -1 WARNING: all dangerous and experimental features are enabled.
2020-05-13T04:04:07.605-0400 7f2aef9c2f80 -1 WARNING: all dangerous and experimental features are enabled.
2020-05-13T04:04:07.616-0400 7f2aef9c2f80 -1 WARNING: all dangerous and experimental features are enabled.
2020-05-13T04:04:07.620-0400 7f2aef9c2f80 -1 init, newargv = 0x55e2537362a0 newargc=9
ceph-fuse[723]: starting ceph client
ceph-fuse[723]: starting fuse
2020-05-13 04:04:50,476.476 INFO:tasks.cephfs.mount:Cleaning the 'ceph-ns--tmp-tmpxtz80q5f-mnt.0' netns for '/tmp/tmpxtz80q5f/mnt.0'
2020-05-13 04:04:50,476.476 INFO:tasks.cephfs.mount:Removing the netns 'ceph-ns--tmp-tmpxtz80q5f-mnt.0'
2020-05-13 04:04:50,477.477 INFO:__main__:Running ['sudo', 'bash', '-c', 'ip link set brx.0 down']
2020-05-13 04:04:50,632.632 INFO:__main__:Running ['sudo', 'bash', '-c', 'ip link delete brx.0']
2020-05-13 04:04:50,836.836 INFO:__main__:Running ['sudo', 'bash', '-c', 'ip netns delete ceph-ns--tmp-tmpxtz80q5f-mnt.0']
2020-05-13 04:04:51,011.011 INFO:__main__:Running ['ip', 'addr']
2020-05-13 04:04:51,040.040 INFO:__main__:Running ['sudo', 'bash', '-c', 'ip link show']
2020-05-13 04:04:51,176.176 INFO:__main__:Running ['rm', '-rf', '/tmp/tmpxtz80q5f/mnt.0']
2020-05-13 04:04:51,208.208 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.cephfs.test_snapshots.TestSnapshots.test_kill_mdstable']
2020-05-13 04:04:53,014.014 INFO:__main__:test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) ... ERROR
2020-05-13 04:04:53,014.014 INFO:__main__:Stopped test: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) in 74.268789s
2020-05-13 04:04:53,016.016 INFO:__main__:
2020-05-13 04:04:53,016.016 INFO:__main__:======================================================================
2020-05-13 04:04:53,016.016 INFO:__main__:ERROR: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots)
2020-05-13 04:04:53,016.016 INFO:__main__:----------------------------------------------------------------------
2020-05-13 04:04:53,017.017 INFO:__main__:Traceback (most recent call last):
2020-05-13 04:04:53,017.017 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/test_snapshots.py", line 76, in test_kill_mdstable
2020-05-13 04:04:53,017.017 INFO:__main__:    self.delete_mds_coredump(rank0['name']);
2020-05-13 04:04:53,017.017 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/cephfs_test_case.py", line 272, in delete_mds_coredump
2020-05-13 04:04:53,017.017 INFO:__main__:    "sudo", "ls", run.Raw('|'), "sudo", "xargs", "file" 
2020-05-13 04:04:53,017.017 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 429, in sh
2020-05-13 04:04:53,018.018 INFO:__main__:    proc = self.run(**kwargs)
2020-05-13 04:04:53,018.018 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 348, in run
2020-05-13 04:04:53,018.018 INFO:__main__:    return self._do_run(**kwargs)
2020-05-13 04:04:53,018.018 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 412, in _do_run
2020-05-13 04:04:53,018.018 INFO:__main__:    proc.wait()
2020-05-13 04:04:53,018.018 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 198, in wait
2020-05-13 04:04:53,018.018 INFO:__main__:    raise CommandFailedError(self.args, self.exitstatus)
2020-05-13 04:04:53,018.018 INFO:__main__:teuthology.exceptions.CommandFailedError: Command failed with status 1: ['cd', '|/usr/libexec', Raw('&&'), 'sudo', 'ls', Raw('|'), 'sudo', 'xargs', 'file']
2020-05-13 04:04:53,019.019 INFO:__main__:
2020-05-13 04:04:53,019.019 INFO:__main__:----------------------------------------------------------------------
2020-05-13 04:04:53,019.019 INFO:__main__:Ran 1 test in 74.270s
2020-05-13 04:04:53,019.019 INFO:__main__:
2020-05-13 04:04:53,019.019 INFO:__main__:FAILED (errors=1)
2020-05-13 04:04:53,019.019 INFO:__main__:
2020-05-13 04:04:53,019.019 INFO:__main__:======================================================================
2020-05-13 04:04:53,020.020 INFO:__main__:ERROR: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots)
2020-05-13 04:04:53,020.020 INFO:__main__:----------------------------------------------------------------------
2020-05-13 04:04:53,020.020 INFO:__main__:Traceback (most recent call last):
2020-05-13 04:04:53,020.020 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/test_snapshots.py", line 76, in test_kill_mdstable
2020-05-13 04:04:53,020.020 INFO:__main__:    self.delete_mds_coredump(rank0['name']);
2020-05-13 04:04:53,020.020 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/cephfs_test_case.py", line 272, in delete_mds_coredump
2020-05-13 04:04:53,020.020 INFO:__main__:    "sudo", "ls", run.Raw('|'), "sudo", "xargs", "file" 
2020-05-13 04:04:53,020.020 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 429, in sh
2020-05-13 04:04:53,020.020 INFO:__main__:    proc = self.run(**kwargs)
2020-05-13 04:04:53,021.021 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 348, in run
2020-05-13 04:04:53,021.021 INFO:__main__:    return self._do_run(**kwargs)
2020-05-13 04:04:53,021.021 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 412, in _do_run
2020-05-13 04:04:53,021.021 INFO:__main__:    proc.wait()
2020-05-13 04:04:53,021.021 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 198, in wait
2020-05-13 04:04:53,021.021 INFO:__main__:    raise CommandFailedError(self.args, self.exitstatus)
2020-05-13 04:04:53,021.021 INFO:__main__:teuthology.exceptions.CommandFailedError: Command failed with status 1: ['cd', '|/usr/libexec', Raw('&&'), 'sudo', 'ls', Raw('|'), 'sudo', 'xargs', 'file']
2020-05-13 04:04:53,021.021 INFO:__main__: