Ceph » CephFS

it hung at

    def test_reconnect_timeout(self):
        # Reconnect timeout
        # =================
        # Check that if I stop an MDS and a client goes away, the MDS waits
        # for the reconnect period
        self.fs.mds_stop()
        self.fs.mds_fail()

        mount_a_client_id = self.mount_a.get_global_id()
        self.mount_a.umount_wait(force=True)

umount_wait() hung at _mountpoint_exists()

    def umount(self):
        try:
            log.info('Running fusermount -u on {name}...'.format(name=self.client_remote.name))
            self.client_remote.run(
                args=[
                    'sudo',
                    'fusermount',
                    '-u',
                    self.mountpoint,
                ],
            )
        except run.CommandFailedError:
            log.info('Failed to unmount ceph-fuse on {name}, aborting...'.format(name=self.client_remote.name))
            # abort the fuse mount, killing all hung processes
            self.client_remote.run(
                args=[
                    'if', 'test', '-e', '/sys/fs/fuse/connections/*/abort',
                    run.Raw(';'), 'then',
                    'echo',
                    '1',
                    run.Raw('>'),
                    run.Raw('/sys/fs/fuse/connections/*/abort'),
                    run.Raw(';'), 'fi',
                ],
            )
            # make sure its unmounted
            if self._mountpoint_exists():

_mountpoint_exists() executes 'ls -d /home/ubuntu/cephtest/mnt.1' but the MDS have already stopped.

the hung happen only when umount fails.

failed to unmount /home/ubuntu/cephtest/mnt.1: Device or resource busy

Isn't the first umount supposed to fail, and then the "except run.CommandFailedError:" block forcibly cleans it up?

Although I think we can probably switch up mountpoint_exists to be based off of "df" or "mount" parsing instead of an "ls", which will help avoid the hangs.

no, the first umount supposed to success. but it fails with error "Device or resource busy".

I think the simplest fix is remove the self._mountpoint_exists() check.

It's weird that the fusermount u is failing - usually if you have an offline MDS and an idle fuse mount and you fusermount -u it, then the fusermount call will succeed, but the ceph-fuse process won't terminate until it can talk to the MDS.

That's the flow that this code usually sees, anyway. I don't know why that isn't happening here :-/

I agree with Zheng that the _mountpoint_exists thing needs to not be there, because when fusermount fails we'll block like this, but we may need to also add an exception handler around the umount call so that if the mount point doesn't exist we don't bail out.

Ah, having seen this fail in an interactive run, I now also realise that the /sys/fs/fuse/connections/*/abort' is bogus in the case of multiple FUSE mounts, it needs to be a while loop over any connections that exist.