Bug #47591: TestNFS: test_exports_on_mgr_restart: command failed with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt' - CephFS - Ceph

Actions

Copy link

Bug #47591

closed

TestNFS: test_exports_on_mgr_restart: command failed with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt'

Added by Michael Fritch over 3 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Varsha Rao

Category:

Testing

Target version:

Ceph - v16.0.0

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

rados

Component(FS):

Ganesha FSAL, mgr/nfs, qa-suite

Labels (FS):

NFS-cluster

Pull request ID:

37571

Crash signature (v1):

Crash signature (v2):

Description

a/mgfritch-2020-09-21_20:24:35-rados:cephadm-wip-mgfritch-testing-2020-09-21-1034-distro-basic-smithi/5457554/teuthology.log

2020-09-22T02:00:26.889 INFO:tasks.mgr.mgr_test_case:Restarted after module load (new active a/14671)
2020-09-22T02:00:26.890 DEBUG:tasks.ceph_test_case:wait_until_true: success in 10s
2020-09-22T02:00:26.890 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early orch set backend cephadm
2020-09-22T02:00:27.234 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early nfs export ls test
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout:[
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout:  "/cephfs" 
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout:]
2020-09-22T02:00:27.625 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early nfs cluster info test
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout:{
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout:    "test": [
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout:        {
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:            "hostname": "smithi077",
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:            "ip": [
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:                "172.21.15.77" 
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:            ],
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:            "port": 2049
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:        }
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:    ]
2020-09-22T02:00:27.982 INFO:teuthology.orchestra.run.smithi077.stdout:}
2020-09-22T02:00:27.996 INFO:teuthology.orchestra.run.smithi077:> sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt
2020-09-22T02:00:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:mount.nfs: Protocol not supported
2020-09-22T02:00:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 32
2020-09-22T02:00:28.104 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_exports_on_mgr_restart'
2020-09-22T02:00:29.307 INFO:tasks.cephfs_test_runner:test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:======================================================================
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgfritch-testing-2020-09-21-1034/qa/tasks/cephfs/test_nfs.py", line 368, in test_exports_on_mgr_restart
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgfritch-testing-2020-09-21-1034/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner:    r.wait()
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi077 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt'
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:
2020-09-22T02:00:29.313 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------

Actions

Copy link

Updated by Patrick Donnelly over 3 years ago

Description updated (diff)

Actions

Copy link

Updated by Varsha Rao over 3 years ago

Assignee set to Varsha Rao

Actions

Copy link

Updated by Varsha Rao over 3 years ago

Status changed from New to Can't reproduce

The mount command does not fail with latest builds: http://pulpito.front.sepia.ceph.com/varsha-2020-09-24_10:49:55-rados-wip-varsha-testing-nfs-distro-basic-smithi/

Actions

Copy link

Updated by Neha Ojha over 3 years ago

Status changed from Can't reproduce to New

2020-10-02T09:14:36.059 INFO:teuthology.orchestra.run.smithi003:> sudo mount -t nfs -o port=2049 172.21.15.3:/cephfs /mnt
2020-10-02T09:14:36.125 INFO:teuthology.orchestra.run.smithi003.stderr:mount.nfs: mounting 172.21.15.3:/cephfs failed, reason given by server: No such file or directory
2020-10-02T09:14:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 32
2020-10-02T09:14:36.127 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_exports_on_mgr_restart'
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 368, in test_exports_on_mgr_restart
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:    r.wait()
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2020-10-02T09:14:37.234 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.3:/cephfs /mnt'

/a/teuthology-2020-10-02_07:01:01-rados-master-distro-basic-smithi/5488975

Actions

Copy link

Updated by Varsha Rao over 3 years ago

Status changed from New to Fix Under Review
Pull request ID set to 37571

I am still not able to reproduce the issue with latest master branch. Looking at failure logs, I suspect ganesha daemons were being refreshed when we tried to mount cephfs exports.

Actions

Copy link

Updated by Patrick Donnelly over 3 years ago

Status changed from Fix Under Review to Resolved
Target version set to v16.0.0

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » CephFS

Custom queries

Bug #47591

TestNFS: test_exports_on_mgr_restart: command failed with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt'

Updated by Patrick Donnelly over 3 years ago

Updated by Varsha Rao over 3 years ago

Updated by Varsha Rao over 3 years ago

Updated by Neha Ojha over 3 years ago

Updated by Varsha Rao over 3 years ago

Updated by Patrick Donnelly over 3 years ago