Project

General

Profile

Bug #47591

TestNFS: test_exports_on_mgr_restart: command failed with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt'

Added by Michael Fritch 4 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Component(FS):
Ganesha FSAL, mgr/nfs, qa-suite
Labels (FS):
NFS-cluster
Pull request ID:
Crash signature:

Description

a/mgfritch-2020-09-21_20:24:35-rados:cephadm-wip-mgfritch-testing-2020-09-21-1034-distro-basic-smithi/5457554/teuthology.log

2020-09-22T02:00:26.889 INFO:tasks.mgr.mgr_test_case:Restarted after module load (new active a/14671)
2020-09-22T02:00:26.890 DEBUG:tasks.ceph_test_case:wait_until_true: success in 10s
2020-09-22T02:00:26.890 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early orch set backend cephadm
2020-09-22T02:00:27.234 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early nfs export ls test
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout:[
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout:  "/cephfs" 
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout:]
2020-09-22T02:00:27.625 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early nfs cluster info test
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout:{
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout:    "test": [
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout:        {
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:            "hostname": "smithi077",
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:            "ip": [
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:                "172.21.15.77" 
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:            ],
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:            "port": 2049
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:        }
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout:    ]
2020-09-22T02:00:27.982 INFO:teuthology.orchestra.run.smithi077.stdout:}
2020-09-22T02:00:27.996 INFO:teuthology.orchestra.run.smithi077:> sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt
2020-09-22T02:00:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:mount.nfs: Protocol not supported
2020-09-22T02:00:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 32
2020-09-22T02:00:28.104 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_exports_on_mgr_restart'
2020-09-22T02:00:29.307 INFO:tasks.cephfs_test_runner:test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:======================================================================
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgfritch-testing-2020-09-21-1034/qa/tasks/cephfs/test_nfs.py", line 368, in test_exports_on_mgr_restart
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgfritch-testing-2020-09-21-1034/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner:    r.wait()
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi077 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt'
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:
2020-09-22T02:00:29.313 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------

History

#1 Updated by Patrick Donnelly 4 months ago

  • Description updated (diff)

#2 Updated by Varsha Rao 4 months ago

  • Assignee set to Varsha Rao

#3 Updated by Varsha Rao 4 months ago

  • Status changed from New to Can't reproduce

#4 Updated by Neha Ojha 4 months ago

  • Status changed from Can't reproduce to New
2020-10-02T09:14:36.059 INFO:teuthology.orchestra.run.smithi003:> sudo mount -t nfs -o port=2049 172.21.15.3:/cephfs /mnt
2020-10-02T09:14:36.125 INFO:teuthology.orchestra.run.smithi003.stderr:mount.nfs: mounting 172.21.15.3:/cephfs failed, reason given by server: No such file or directory
2020-10-02T09:14:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 32
2020-10-02T09:14:36.127 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_exports_on_mgr_restart'
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 368, in test_exports_on_mgr_restart
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:    r.wait()
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2020-10-02T09:14:37.234 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.3:/cephfs /mnt'

/a/teuthology-2020-10-02_07:01:01-rados-master-distro-basic-smithi/5488975

#5 Updated by Varsha Rao 4 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 37571

I am still not able to reproduce the issue with latest master branch. Looking at failure logs, I suspect ganesha daemons were being refreshed when we tried to mount cephfs exports.

#6 Updated by Patrick Donnelly 4 months ago

  • Status changed from Fix Under Review to Resolved
  • Target version set to v16.0.0

Also available in: Atom PDF