Ceph » CephFS

need add _netdev mount option

Patrick Donnelly wrote:

You probably have the wrong mons in the mount configuration.

I have only one mon.

Can someone please check the logs, we are having this issue 50% of the time in sanity runs.

In messages_ceph-sshreeka-run379-node5-client

Mar 14 11:02:55 ceph-sshreeka-run379-node5-client mount: mount error 1 = Operation not permitted
Mar 14 11:02:55 ceph-sshreeka-run379-node5-client kernel: libceph: auth method 'x' error -1

looks like fstab didn't include correct secret

Zheng Yan wrote:

In messages_ceph-sshreeka-run379-node5-client
[...]

looks like fstab didn't include correct secret

command mount -a works with same entries in fstab.

Actually, the first failure seems to be a bit before:

Mar 14 08:10:00 ceph-sshreeka-run379-node5-client systemd: Stopping User Slice of root.
Mar 14 08:10:42 ceph-sshreeka-run379-node5-client kernel: ceph: mds1 caps stale
Mar 14 08:11:02 ceph-sshreeka-run379-node5-client kernel: ceph: mds1 caps stale
Mar 14 08:14:43 ceph-sshreeka-run379-node5-client kernel: libceph: mds0 172.16.115.100:6800 socket closed (con state OPEN)
Mar 14 08:14:46 ceph-sshreeka-run379-node5-client chronyd[499]: Source 45.76.244.202 replaced with 64.113.44.54
Mar 14 08:14:47 ceph-sshreeka-run379-node5-client kernel: libceph: mds0 172.16.115.100:6800 connection reset
Mar 14 08:14:47 ceph-sshreeka-run379-node5-client kernel: libceph: reset on mds0
Mar 14 08:14:47 ceph-sshreeka-run379-node5-client kernel: ceph: mds0 closed our session
Mar 14 08:14:47 ceph-sshreeka-run379-node5-client kernel: ceph: mds0 reconnect start
Mar 14 08:14:47 ceph-sshreeka-run379-node5-client kernel: ceph: mds0 reconnect denied
Mar 14 08:15:20 ceph-sshreeka-run379-node5-client systemd-logind: Removed session 20.
Mar 14 08:15:20 ceph-sshreeka-run379-node5-client systemd: Removed slice User Slice of cephuser.
Mar 14 08:15:20 ceph-sshreeka-run379-node5-client systemd: Stopping User Slice of cephuser.
Mar 14 08:23:05 ceph-sshreeka-run379-node5-client kernel: libceph: auth method 'x' error -22
Mar 14 08:23:15 ceph-sshreeka-run379-node5-client kernel: libceph: auth method 'x' error -22
Mar 14 08:23:25 ceph-sshreeka-run379-node5-client kernel: libceph: auth method 'x' error -22
...

It's possible that this has been fixed in recent kernels and the fix hasn't been backported to your distro. Maybe it's worth trying a recent kernel and try to reproduce it.

Luis Henriques wrote:

Actually, the first failure seems to be a bit before:
[...]
The client seems to be trying to renew caps while connection is being closed.

It's possible that this has been fixed in recent kernels and the fix hasn't been backported to your distro. Maybe it's worth trying a recent kernel and try to reproduce it.

I've checked with the latest kernel version, still result was same

kexec in dmesgs looks suspicious. client mounted cephfs, then used kexec to load kernel image again. All issues happened after kexec

I still don't think this is kernel issue. please path kernel with below change and try again.

diff --git a/net/ceph/ceph_common.c b/net/ceph/ceph_common.c
index 584fdbef2088..3ca2d0c65d20 100644
--- a/net/ceph/ceph_common.c
+++ b/net/ceph/ceph_common.c
@@ -432,6 +432,7 @@ ceph_parse_options(char *options, const char *dev_name,
                                err = -ENOMEM;
                                goto out;
                        }
+                       printk("name: %s\n", opt->name);
                        break;
                case Opt_secret:
                        ceph_crypto_key_destroy(opt->key);
@@ -442,6 +443,7 @@ ceph_parse_options(char *options, const char *dev_name,
                                err = -ENOMEM;
                                goto out;
                        }
+                       printk("secret: %s\n", argstr[0].from);
                        err = ceph_crypto_key_unarmor(opt->key, argstr[0].from);
                        if (err < 0)
                                goto out;