Bug #47591
TestNFS: test_exports_on_mgr_restart: command failed with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt'
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Component(FS):
Ganesha FSAL, mgr/nfs, qa-suite
Labels (FS):
NFS-cluster
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
a/mgfritch-2020-09-21_20:24:35-rados:cephadm-wip-mgfritch-testing-2020-09-21-1034-distro-basic-smithi/5457554/teuthology.log
2020-09-22T02:00:26.889 INFO:tasks.mgr.mgr_test_case:Restarted after module load (new active a/14671)
2020-09-22T02:00:26.890 DEBUG:tasks.ceph_test_case:wait_until_true: success in 10s
2020-09-22T02:00:26.890 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early orch set backend cephadm
2020-09-22T02:00:27.234 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early nfs export ls test
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout:[
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout: "/cephfs"
2020-09-22T02:00:27.606 INFO:teuthology.orchestra.run.smithi077.stdout:]
2020-09-22T02:00:27.625 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early nfs cluster info test
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout:{
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout: "test": [
2020-09-22T02:00:27.980 INFO:teuthology.orchestra.run.smithi077.stdout: {
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077",
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout: "ip": [
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout: "172.21.15.77"
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout: ],
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout: "port": 2049
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout: }
2020-09-22T02:00:27.981 INFO:teuthology.orchestra.run.smithi077.stdout: ]
2020-09-22T02:00:27.982 INFO:teuthology.orchestra.run.smithi077.stdout:}
2020-09-22T02:00:27.996 INFO:teuthology.orchestra.run.smithi077:> sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt
2020-09-22T02:00:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:mount.nfs: Protocol not supported
2020-09-22T02:00:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 32
2020-09-22T02:00:28.104 INFO:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_exports_on_mgr_restart'
2020-09-22T02:00:29.307 INFO:tasks.cephfs_test_runner:test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:======================================================================
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-09-22T02:00:29.308 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgfritch-testing-2020-09-21-1034/qa/tasks/cephfs/test_nfs.py", line 368, in test_exports_on_mgr_restart
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner: self._test_mnt(self.pseudo_path, port, ip)
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-mgfritch-testing-2020-09-21-1034/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner: f'{ip}:{pseudo_path}', '/mnt'])
2020-09-22T02:00:29.309 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner: return [remote.run(**kwargs) for remote in remotes]
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner: return [remote.run(**kwargs) for remote in remotes]
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-09-22T02:00:29.310 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner: r.wait()
2020-09-22T02:00:29.311 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner: self._raise_for_status()
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi077 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.77:/cephfs /mnt'
2020-09-22T02:00:29.312 INFO:tasks.cephfs_test_runner:
2020-09-22T02:00:29.313 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
History
#1 Updated by Patrick Donnelly about 3 years ago
- Description updated (diff)
#2 Updated by Varsha Rao about 3 years ago
- Assignee set to Varsha Rao
#3 Updated by Varsha Rao about 3 years ago
- Status changed from New to Can't reproduce
The mount command does not fail with latest builds: http://pulpito.front.sepia.ceph.com/varsha-2020-09-24_10:49:55-rados-wip-varsha-testing-nfs-distro-basic-smithi/
#4 Updated by Neha Ojha almost 3 years ago
- Status changed from Can't reproduce to New
2020-10-02T09:14:36.059 INFO:teuthology.orchestra.run.smithi003:> sudo mount -t nfs -o port=2049 172.21.15.3:/cephfs /mnt
2020-10-02T09:14:36.125 INFO:teuthology.orchestra.run.smithi003.stderr:mount.nfs: mounting 172.21.15.3:/cephfs failed, reason given by server: No such file or directory
2020-10-02T09:14:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 32
2020-10-02T09:14:36.127 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_exports_on_mgr_restart'
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:
2020-10-02T09:14:37.229 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-10-02T09:14:37.230 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 368, in test_exports_on_mgr_restart
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner: self._test_mnt(self.pseudo_path, port, ip)
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner: f'{ip}:{pseudo_path}', '/mnt'])
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
2020-10-02T09:14:37.231 INFO:tasks.cephfs_test_runner: return [remote.run(**kwargs) for remote in remotes]
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner: return [remote.run(**kwargs) for remote in remotes]
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-10-02T09:14:37.232 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner: r.wait()
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner: self._raise_for_status()
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-10-02T09:14:37.233 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label
2020-10-02T09:14:37.234 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.3:/cephfs /mnt'
/a/teuthology-2020-10-02_07:01:01-rados-master-distro-basic-smithi/5488975
#5 Updated by Varsha Rao almost 3 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 37571
I am still not able to reproduce the issue with latest master branch. Looking at failure logs, I suspect ganesha daemons were being refreshed when we tried to mount cephfs exports.
#6 Updated by Patrick Donnelly almost 3 years ago
- Status changed from Fix Under Review to Resolved
- Target version set to v16.0.0