Ceph » CephFS

My ceph version is 15.2.8 and deployment with kubernetes.

The kubernetes version is 1.18.8.

The kubernetes's cni network plugin is flannel.

This problem has plagued me for days and I hope the community will give me some guidance,Thank you so much!

the mgr log

debug 2021-02-08T04:44:19.388+0000 ffff6144c700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):

File "/usr/share/ceph/mgr/volumes/fs/operations/volume.py", line 165, in get_fs_handle

conn.connect()

File "/usr/share/ceph/mgr/volumes/fs/operations/volume.py", line 88, in connect

self.fs.mount(filesystem_name=self.fs_name.encode('utf-8'))

File "cephfs.pyx", line 739, in cephfs.LibCephFS.mount

cephfs.Error: error calling ceph_mount: Connection timed out [Errno 110]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/share/ceph/mgr/mgr_module.py", line 1177, in _handle_command

return self.handle_command(inbuf, cmd)

File "/usr/share/ceph/mgr/volumes/module.py", line 426, in handle_command

return handler(inbuf, cmd)

File "/usr/share/ceph/mgr/volumes/module.py", line 34, in wrap

return f(self, inbuf, cmd)

File "/usr/share/ceph/mgr/volumes/module.py", line 480, in _cmd_fs_subvolume_create

namespace_isolated=cmd.get('namespace_isolated', False))

File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 157, in create_subvolume

with open_volume(self, volname) as fs_handle:

File "/lib64/python3.6/contextlib.py", line 81, in __enter__

return next(self.gen)

File "/usr/share/ceph/mgr/volumes/fs/operations/volume.py", line 316, in open_volume

fs_handle = vc.connection_pool.get_fs_handle(volname)

File "/usr/share/ceph/mgr/volumes/fs/operations/volume.py", line 171, in get_fs_handle

raise VolumeException(-e.args[0], e.args[1])

TypeError: bad operand type for unary -: 'str'

the mds log

debug 2021-02-08T05:08:27.821+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:08:37.822+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:08:47.812+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:08:57.832+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:09:07.792+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:09:17.792+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:09:27.812+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:09:37.852+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:09:47.802+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:09:57.812+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:10:07.822+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:10:17.782+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:10:27.833+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:10:37.803+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:10:47.833+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:10:57.803+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:11:07.773+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:11:17.833+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:11:27.793+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:11:37.793+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:11:47.793+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:11:57.783+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

debug 2021-02-08T05:12:07.764+0000 ffff823ed300 1 mds.myfs-a asok_command: status {prefix=status} (starting...)

the csi-provisioner log

W0208 02:05:17.483167 1 controller.go:943] Retrying syncing claim "e760d43a-eb8e-443b-8461-b38d3b6ce1b1", failure 10

E0208 02:05:17.483207 1 controller.go:966] error syncing claim "e760d43a-eb8e-443b-8461-b38d3b6ce1b1": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded

I0208 02:05:17.483228 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"rook-ceph", Name:"cephfs-test", UID:"e760d43a-eb8e-443b-8461-b38d3b6ce1b1", APIVersion:"v1", ResourceVersion:"13313707", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded

I0208 02:10:17.483438 1 controller.go:1317] provision "rook-ceph/cephfs-test" class "rook-cephfs": started

I0208 02:10:17.483610 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"rook-ceph", Name:"cephfs-test", UID:"e760d43a-eb8e-443b-8461-b38d3b6ce1b1", APIVersion:"v1", ResourceVersion:"13313707", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "rook-ceph/cephfs-test" 

W0208 02:10:17.488237 1 controller.go:943] Retrying syncing claim "e760d43a-eb8e-443b-8461-b38d3b6ce1b1", failure 11

E0208 02:10:17.488277 1 controller.go:966] error syncing claim "e760d43a-eb8e-443b-8461-b38d3b6ce1b1": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = InvalidArgument desc = volume not found

I0208 02:10:17.488317 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"rook-ceph", Name:"cephfs-test", UID:"e760d43a-eb8e-443b-8461-b38d3b6ce1b1", APIVersion:"v1", ResourceVersion:"13313707", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": rpc error: code = InvalidArgument desc = volume not found

E0208 02:15:17.488410 1 controller.go:1035] claim "e760d43a-eb8e-443b-8461-b38d3b6ce1b1" in work queue no longer exists