Ceph » CephFS

2022-09-18T03:29:20.394 INFO:teuthology.orchestra.run.smithi093.stderr:Traceback (most recent call last):
2022-09-18T03:29:20.395 INFO:teuthology.orchestra.run.smithi093.stderr:  File "<string>", line 39, in <module>
2022-09-18T03:29:20.395 INFO:teuthology.orchestra.run.smithi093.stderr:BrokenPipeError: [Errno 108] Cannot send after transport endpoint shutdown
...
2022-09-18T03:29:29.438 INFO:tasks.cephfs_test_runner:======================================================================
2022-09-18T03:29:29.439 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
2022-09-18T03:29:29.439 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-09-18T03:29:29.439 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-09-18T03:29:29.440 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_b72a2ce17ad60da863086f3b9f8648dc49f11b79/qa/tasks/cephfs/test_client_recovery.py", line 725, in test_reconnect_after_blocklisted
2022-09-18T03:29:29.440 INFO:tasks.cephfs_test_runner:    rproc.wait()
2022-09-18T03:29:29.440 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
2022-09-18T03:29:29.441 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2022-09-18T03:29:29.441 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-09-18T03:29:29.441 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2022-09-18T03:29:29.442 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi093 with status 1: 'python3 -c \'\nimport os\nimport sys\nimport fcntl\nimport errno\nimport time\n\nfd1 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDWR | os.O_CREAT, 0O666)\nfd2 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDONLY)\nfd3 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDWR | os.O_CREAT, 0O666)\nfd4 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDONLY)\n\nos.write(fd1, b\'"\'"\'content\'"\'"\')\nos.read(fd2, 1);\n\nos.write(fd3, b\'"\'"\'content\'"\'"\')\nos.read(fd4, 1);\nfcntl.flock(fd4, fcntl.LOCK_SH | fcntl.LOCK_NB)\n\nprint("blocklist")\nsys.stdout.flush()\n\nsys.stdin.readline()\n\n# wait for mds to close session\ntime.sleep(10);\n\n# trigger \'"\'"\'open session\'"\'"\' message. kclient relies on \'"\'"\'session reject\'"\'"\' message\n# to detect if itself is blocklisted\ntry:\n    os.stat("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1")\nexcept:\n    pass\n\n# wait for auto reconnect\ntime.sleep(10);\n\ntry:\n    os.write(fd1, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n    if e.errno != errno.EBADF:\n        raise\nelse:\n    raise RuntimeError("write() failed to raise error")\n\nos.read(fd2, 1);\n\ntry:\n    os.read(fd4, 1)\nexcept OSError as e:\n    if e.errno != errno.EIO:\n        raise\nelse:\n    raise RuntimeError("read() failed to raise error")\n\''

From: /ceph/teuthology-archive/teuthology-2022-09-13_03:15:04-fs-quincy-distro-default-smithi/7029857/teuthology.log

2022-09-18T03:29:20.392+0000 7fad6b7fe700 10 client.9507 waiting for caps 0x10000000000.head(faked_ino=0 nref=5 ll_ref=2 cap_refs={4=0,1024=0,2048=0,4096=0,8192=0} open={1=1,3=1} mode=100664 size=7/4194304 nlink=1 btime=2022-09-18T03:28:58.828547+0000 mtime=2022-09-18T03:28:58.836968+0000 ctime=2022-09-18T03:28:58.836968+0000 caps=- objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 0] parents=0x1.head["testfile_reconnect_after_blocklisted.1"] 0x7fad7c0097c0) need AsFw want Fb
2022-09-18T03:29:20.392+0000 7fad6b7fe700 20 client.9507 put_inode on 0x10000000000.head(faked_ino=0 nref=6 ll_ref=2 cap_refs={4=0,1024=0,2048=0,4096=0,8192=0} open={1=1,3=1} mode=100664 size=7/4194304 nlink=1 btime=2022-09-18T03:28:58.828547+0000 mtime=2022-09-18T03:28:58.836968+0000 ctime=2022-09-18T03:28:58.836968+0000 caps=- objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 0] parents=0x1.head["testfile_reconnect_after_blocklisted.1"] 0x7fad7c0097c0) n = 1
2022-09-18T03:29:20.392+0000 7fad6b7fe700  3 client.9507 ll_write 0x7fad64003140 7~7 = -108

From: /ceph/teuthology-archive/teuthology-2022-09-13_03:15:04-fs-quincy-distro-default-smithi/7029857/remote/smithi093/log/ceph-client.0.165556.log.gz

Seen in many other runs:

/repdonnell-2022-09-23_19:31:41-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7042346/teuthology.log:2022-09-24T06:44:10.444 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-23_19:31:41-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7042346/teuthology.log:2022-09-24T06:44:10.457 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-25_13:20:24-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044004/teuthology.log:2022-09-25T14:56:02.280 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-25_13:20:24-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044004/teuthology.log:2022-09-25T14:56:02.298 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044462/teuthology.log:2022-09-26T17:16:48.337 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-26_16:35:43-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044462/teuthology.log:2022-09-26T17:16:48.349 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-26_19:11:10-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044502/teuthology.log:2022-09-26T19:43:49.312 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-26_19:11:10-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044502/teuthology.log:2022-09-26T19:43:49.321 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-27_02:27:48-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044558/teuthology.log:2022-09-27T03:00:24.834 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
pdonnell-2022-09-27_02:27:48-fs-wip-pdonnell-testing-20220923.171109-distro-default-smithi/7044558/teuthology.log:2022-09-27T03:00:24.841 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-13_03:15:04-fs-quincy-distro-default-smithi/7029857/teuthology.log:2022-09-18T03:29:29.439 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-13_03:15:04-fs-quincy-distro-default-smithi/7029857/teuthology.log:2022-09-18T03:29:29.444 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-13_04:17:04-fs-pacific-distro-default-smithi/7030309/teuthology.log:2022-09-17T13:46:04.269 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-13_04:17:04-fs-pacific-distro-default-smithi/7030309/teuthology.log:2022-09-17T13:46:04.286 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-27_04:17:03-fs-pacific-distro-default-smithi/7045022/teuthology.log:2022-09-27T07:43:33.096 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
teuthology-2022-09-27_04:17:03-fs-pacific-distro-default-smithi/7045022/teuthology.log:2022-09-27T07:43:33.102 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
vshankar-2022-09-06_06:13:42-fs-wip-vshankar-testing-20220901-100101-testing-default-smithi/7014009/teuthology.log:2022-09-06T11:55:08.106 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)
vshankar-2022-09-06_06:13:42-fs-wip-vshankar-testing-20220901-100101-testing-default-smithi/7014009/teuthology.log:2022-09-06T11:55:08.112 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)