Project

General

Profile

Bug #48502

ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)

Added by Neha Ojha 5 months ago. Updated 3 months ago.

Status:
Triaged
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
pacific,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
mgr/nfs
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2020-12-07T09:36:14.838 INFO:teuthology.orchestra.run.smithi171:> sudo mount -t nfs -o port=2049 172.21.15.171:/cephfs /mnt
2020-12-07T09:36:14.893 INFO:teuthology.orchestra.run.smithi171.stderr:mount.nfs: mounting 172.21.15.171:/cephfs failed, reason given by server: No such file or directory
2020-12-07T09:36:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 32
2020-12-07T09:36:14.896 INFO:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_exports_on_mgr_restart'
2020-12-07T09:36:15.352 INFO:tasks.cephfs_test_runner:test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2020-12-07T09:36:15.352 INFO:tasks.cephfs_test_runner:
2020-12-07T09:36:15.353 INFO:tasks.cephfs_test_runner:======================================================================
2020-12-07T09:36:15.353 INFO:tasks.cephfs_test_runner:ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)
2020-12-07T09:36:15.353 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-12-07T09:36:15.354 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-12-07T09:36:15.354 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 370, in test_exports_on_mgr_restart
2020-12-07T09:36:15.354 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2020-12-07T09:36:15.355 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2020-12-07T09:36:15.355 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2020-12-07T09:36:15.355 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
2020-12-07T09:36:15.355 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-12-07T09:36:15.356 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2020-12-07T09:36:15.356 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-12-07T09:36:15.356 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-12-07T09:36:15.356 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-12-07T09:36:15.357 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-12-07T09:36:15.357 INFO:tasks.cephfs_test_runner:    r.wait()
2020-12-07T09:36:15.357 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-12-07T09:36:15.358 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-12-07T09:36:15.358 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-12-07T09:36:15.358 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2020-12-07T09:36:15.358 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi171 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.171:/cephfs /mnt'

/a/teuthology-2020-12-06_07:01:02-rados-master-distro-basic-smithi/5684842

History

#1 Updated by Patrick Donnelly 5 months ago

  • Status changed from New to Triaged
  • Assignee set to Varsha Rao
  • Target version set to v16.0.0
  • Source set to Q/A
  • Backport set to octopus
  • Component(FS) mgr/nfs added

#2 Updated by Varsha Rao 5 months ago

In the latest test run, I cannot reproduce it. Sometimes ganesha takes time to be restarted completely. This can be the case here.

/varsha-2020-12-09_11:46:16-rados-wip-nfs-remove-ganesha-prefix-distro-basic-smithi/

#3 Updated by Patrick Donnelly 4 months ago

  • Target version changed from v16.0.0 to v17.0.0
  • Backport changed from octopus to pacific,octopus,nautilus

#4 Updated by Varsha Rao 4 months ago

  • Backport changed from pacific,octopus,nautilus to pacific,octopus

#5 Updated by Sebastian Wagner 3 months ago

2021-02-25T12:11:39.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph nfs cluster info test
2021-02-25T12:11:40.070 INFO:teuthology.orchestra.run.smithi006.stdout:{
2021-02-25T12:11:40.071 INFO:teuthology.orchestra.run.smithi006.stdout:    "test": [
2021-02-25T12:11:40.071 INFO:teuthology.orchestra.run.smithi006.stdout:        {
2021-02-25T12:11:40.071 INFO:teuthology.orchestra.run.smithi006.stdout:            "hostname": "smithi006",
2021-02-25T12:11:40.072 INFO:teuthology.orchestra.run.smithi006.stdout:            "ip": [
2021-02-25T12:11:40.072 INFO:teuthology.orchestra.run.smithi006.stdout:                "127.0.1.1",
2021-02-25T12:11:40.072 INFO:teuthology.orchestra.run.smithi006.stdout:                "172.21.15.6" 
2021-02-25T12:11:40.073 INFO:teuthology.orchestra.run.smithi006.stdout:            ],
2021-02-25T12:11:40.073 INFO:teuthology.orchestra.run.smithi006.stdout:            "port": 2049
2021-02-25T12:11:40.073 INFO:teuthology.orchestra.run.smithi006.stdout:        }
2021-02-25T12:11:40.074 INFO:teuthology.orchestra.run.smithi006.stdout:    ]
2021-02-25T12:11:40.074 INFO:teuthology.orchestra.run.smithi006.stdout:}
2021-02-25T12:11:40.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t nfs -o port=2049 127.0.1.1:/cephfs /mnt
2021-02-25T12:11:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:mount.nfs: mounting 127.0.1.1:/cephfs failed, reason given by server: No such file or directory
2021-02-25T12:11:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 32
2021-02-25T12:11:40.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_create_and_delete_export'
2021-02-25T12:11:40.295 INFO:journalctl@ceph.mon.a.smithi006.stdout:Feb 25 12:11:40 smithi006 conmon[31095]: audit 2021-02-25T12:11:38.679879+0000 mgr.a (mgr.14241) 178 : audit [DBG] from='client.14391 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "nfs-cephfs", "clusterid": "test", "binding": " 
2021-02-25T12:11:40.295 INFO:journalctl@ceph.mon.a.smithi006.stdout:Feb 25 12:11:40 smithi006 conmon[31095]: cluster 2021-02-25T12:11:39.227766+0000 mon.a (mon.0) 510 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in
2021-02-25T12:11:40.295 INFO:journalctl@ceph.mon.a.smithi006.stdout:Feb 25 12:11:40 smithi006 conmon[31095]: cluster 2021-02-25T12:11:39.272035+0000 mon.a (mon.0) 511 : cluster [DBG] mds.? [v2:172.21.15.6:6800/1006157824,v1:172.21.15.6:6801/1006157824] up:active
2021-02-25T12:11:40.296 INFO:journalctl@ceph.mon.a.smithi006.stdout:Feb 25 12:11:40 smithi006 conmon[31095]: cluster 2021-02-25T12:11:39.272126+0000 mon.a (mon.0) 512 : cluster [DBG] fsmap nfs-cephfs:1 {0=a.smithi006.zzcrpq=up:active}
2021-02-25T12:11:41.251 INFO:tasks.cephfs_test_runner:test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2021-02-25T12:11:41.252 INFO:tasks.cephfs_test_runner:
2021-02-25T12:11:41.253 INFO:tasks.cephfs_test_runner:======================================================================
2021-02-25T12:11:41.253 INFO:tasks.cephfs_test_runner:ERROR: test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS)
2021-02-25T12:11:41.254 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-02-25T12:11:41.255 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-02-25T12:11:41.255 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_7242a9001aafb312ce7338dd349899af52369f36/qa/tasks/cephfs/test_nfs.py", line 318, in test_create_and_delete_export
2021-02-25T12:11:41.256 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2021-02-25T12:11:41.256 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_7242a9001aafb312ce7338dd349899af52369f36/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2021-02-25T12:11:41.257 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2021-02-25T12:11:41.258 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in run
2021-02-25T12:11:41.258 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2021-02-25T12:11:41.259 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2021-02-25T12:11:41.259 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2021-02-25T12:11:41.260 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run
2021-02-25T12:11:41.261 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2021-02-25T12:11:41.261 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 455, in run
2021-02-25T12:11:41.262 INFO:tasks.cephfs_test_runner:    r.wait()
2021-02-25T12:11:41.262 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 161, in wait
2021-02-25T12:11:41.263 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2021-02-25T12:11:41.264 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 183, in _raise_for_status
2021-02-25T12:11:41.264 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2021-02-25T12:11:41.265 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 32: 'sudo mount -t nfs -o port=2049 127.0.1.1:/cephfs /mnt'
2021-02-25T12:11:41.265 INFO:tasks.cephfs_test_runner:

https://pulpito.ceph.com/swagner-2021-02-25_11:28:56-rados:cephadm-wip-swagner-testing-2021-02-24-1710-distro-basic-smithi/5913275/

Also available in: Atom PDF