Bug #48502: ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) - CephFS - Ceph

Bug #48502

ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)

Added by Neha Ojha almost 3 years ago. Updated 4 months ago.

Status:

Triaged

Priority:

Normal

Assignee:

Varsha Rao

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

pacific,octopus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

kceph, mgr/nfs

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Tags:

test-failure

Description

2020-12-07T09:36:14.838 INFO:teuthology.orchestra.run.smithi171:> sudo mount -t nfs -o port=2049 172.21.15.171:/cephfs /mnt
2020-12-07T09:36:14.893 INFO:teuthology.orchestra.run.smithi171.stderr:mount.nfs: mounting 172.21.15.171:/cephfs failed, reason given by server: No such file or directory
2020-12-07T09:36:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 32
2020-12-07T09:36:14.896 INFO:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_exports_on_mgr_restart'
2020-12-07T09:36:15.352 INFO:tasks.cephfs_test_runner:test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2020-12-07T09:36:15.352 INFO:tasks.cephfs_test_runner:
2020-12-07T09:36:15.353 INFO:tasks.cephfs_test_runner:======================================================================
2020-12-07T09:36:15.353 INFO:tasks.cephfs_test_runner:ERROR: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS)
2020-12-07T09:36:15.353 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-12-07T09:36:15.354 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-12-07T09:36:15.354 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 370, in test_exports_on_mgr_restart
2020-12-07T09:36:15.354 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2020-12-07T09:36:15.355 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2020-12-07T09:36:15.355 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2020-12-07T09:36:15.355 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
2020-12-07T09:36:15.355 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-12-07T09:36:15.356 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2020-12-07T09:36:15.356 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2020-12-07T09:36:15.356 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-12-07T09:36:15.356 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-12-07T09:36:15.357 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-12-07T09:36:15.357 INFO:tasks.cephfs_test_runner:    r.wait()
2020-12-07T09:36:15.357 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-12-07T09:36:15.358 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-12-07T09:36:15.358 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-12-07T09:36:15.358 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2020-12-07T09:36:15.358 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi171 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.171:/cephfs /mnt'

/a/teuthology-2020-12-06_07:01:02-rados-master-distro-basic-smithi/5684842

History

#1 Updated by Patrick Donnelly almost 3 years ago

Status changed from New to Triaged
Assignee set to Varsha Rao
Target version set to v16.0.0
Source set to Q/A
Backport set to octopus
Component(FS) mgr/nfs added

#2 Updated by Varsha Rao almost 3 years ago

In the latest test run, I cannot reproduce it. Sometimes ganesha takes time to be restarted completely. This can be the case here.

/varsha-2020-12-09_11:46:16-rados-wip-nfs-remove-ganesha-prefix-distro-basic-smithi/

#3 Updated by Patrick Donnelly almost 3 years ago

Target version changed from v16.0.0 to v17.0.0
Backport changed from octopus to pacific,octopus,nautilus

#4 Updated by Varsha Rao almost 3 years ago

Backport changed from pacific,octopus,nautilus to pacific,octopus

#5 Updated by Sebastian Wagner almost 3 years ago

2021-02-25T12:11:39.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph nfs cluster info test
2021-02-25T12:11:40.070 INFO:teuthology.orchestra.run.smithi006.stdout:{
2021-02-25T12:11:40.071 INFO:teuthology.orchestra.run.smithi006.stdout:    "test": [
2021-02-25T12:11:40.071 INFO:teuthology.orchestra.run.smithi006.stdout:        {
2021-02-25T12:11:40.071 INFO:teuthology.orchestra.run.smithi006.stdout:            "hostname": "smithi006",
2021-02-25T12:11:40.072 INFO:teuthology.orchestra.run.smithi006.stdout:            "ip": [
2021-02-25T12:11:40.072 INFO:teuthology.orchestra.run.smithi006.stdout:                "127.0.1.1",
2021-02-25T12:11:40.072 INFO:teuthology.orchestra.run.smithi006.stdout:                "172.21.15.6" 
2021-02-25T12:11:40.073 INFO:teuthology.orchestra.run.smithi006.stdout:            ],
2021-02-25T12:11:40.073 INFO:teuthology.orchestra.run.smithi006.stdout:            "port": 2049
2021-02-25T12:11:40.073 INFO:teuthology.orchestra.run.smithi006.stdout:        }
2021-02-25T12:11:40.074 INFO:teuthology.orchestra.run.smithi006.stdout:    ]
2021-02-25T12:11:40.074 INFO:teuthology.orchestra.run.smithi006.stdout:}
2021-02-25T12:11:40.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t nfs -o port=2049 127.0.1.1:/cephfs /mnt
2021-02-25T12:11:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:mount.nfs: mounting 127.0.1.1:/cephfs failed, reason given by server: No such file or directory
2021-02-25T12:11:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 32
2021-02-25T12:11:40.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_create_and_delete_export'
2021-02-25T12:11:40.295 INFO:journalctl@ceph.mon.a.smithi006.stdout:Feb 25 12:11:40 smithi006 conmon[31095]: audit 2021-02-25T12:11:38.679879+0000 mgr.a (mgr.14241) 178 : audit [DBG] from='client.14391 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "nfs-cephfs", "clusterid": "test", "binding": " 
2021-02-25T12:11:40.295 INFO:journalctl@ceph.mon.a.smithi006.stdout:Feb 25 12:11:40 smithi006 conmon[31095]: cluster 2021-02-25T12:11:39.227766+0000 mon.a (mon.0) 510 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in
2021-02-25T12:11:40.295 INFO:journalctl@ceph.mon.a.smithi006.stdout:Feb 25 12:11:40 smithi006 conmon[31095]: cluster 2021-02-25T12:11:39.272035+0000 mon.a (mon.0) 511 : cluster [DBG] mds.? [v2:172.21.15.6:6800/1006157824,v1:172.21.15.6:6801/1006157824] up:active
2021-02-25T12:11:40.296 INFO:journalctl@ceph.mon.a.smithi006.stdout:Feb 25 12:11:40 smithi006 conmon[31095]: cluster 2021-02-25T12:11:39.272126+0000 mon.a (mon.0) 512 : cluster [DBG] fsmap nfs-cephfs:1 {0=a.smithi006.zzcrpq=up:active}
2021-02-25T12:11:41.251 INFO:tasks.cephfs_test_runner:test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2021-02-25T12:11:41.252 INFO:tasks.cephfs_test_runner:
2021-02-25T12:11:41.253 INFO:tasks.cephfs_test_runner:======================================================================
2021-02-25T12:11:41.253 INFO:tasks.cephfs_test_runner:ERROR: test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS)
2021-02-25T12:11:41.254 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-02-25T12:11:41.255 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-02-25T12:11:41.255 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_7242a9001aafb312ce7338dd349899af52369f36/qa/tasks/cephfs/test_nfs.py", line 318, in test_create_and_delete_export
2021-02-25T12:11:41.256 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2021-02-25T12:11:41.256 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_7242a9001aafb312ce7338dd349899af52369f36/qa/tasks/cephfs/test_nfs.py", line 253, in _test_mnt
2021-02-25T12:11:41.257 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2021-02-25T12:11:41.258 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in run
2021-02-25T12:11:41.258 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2021-02-25T12:11:41.259 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2021-02-25T12:11:41.259 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2021-02-25T12:11:41.260 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run
2021-02-25T12:11:41.261 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2021-02-25T12:11:41.261 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 455, in run
2021-02-25T12:11:41.262 INFO:tasks.cephfs_test_runner:    r.wait()
2021-02-25T12:11:41.262 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 161, in wait
2021-02-25T12:11:41.263 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2021-02-25T12:11:41.264 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 183, in _raise_for_status
2021-02-25T12:11:41.264 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2021-02-25T12:11:41.265 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 32: 'sudo mount -t nfs -o port=2049 127.0.1.1:/cephfs /mnt'
2021-02-25T12:11:41.265 INFO:tasks.cephfs_test_runner:

https://pulpito.ceph.com/swagner-2021-02-25_11:28:56-rados:cephadm-wip-swagner-testing-2021-02-24-1710-distro-basic-smithi/5913275/

#6 Updated by Kefu Chai over 2 years ago

Component(FS) kceph added

2021-06-28T16:32:42.617 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t nfs -o port=2049 172.21.15.22:/cephfs /mnt
2021-06-28T16:32:42.682 INFO:teuthology.orchestra.run.smithi022.stderr:mount.nfs: mounting 172.21.15.22:/cephfs failed, reason given by server: No such file or directory
2021-06-28T16:32:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 32
2021-06-28T16:32:42.684 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_create_and_delete_export'
2021-06-28T16:32:42.947 INFO:journalctl@ceph.mon.a.smithi022.stdout:Jun 28 16:32:42 smithi022 ceph-mon[31875]: Deploying daemon mds.nfs-cephfs.smithi022.debamp on smithi022
2021-06-28T16:32:42.947 INFO:journalctl@ceph.mon.a.smithi022.stdout:Jun 28 16:32:42 smithi022 ceph-mon[31875]: from='client.14491 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "nfs-cephfs", "cluster_id": "test", "pseudo_path": "/cephfs", " 
target": ["mon-mgr", ""]}]: dispatch
2021-06-28T16:32:42.947 INFO:journalctl@ceph.mon.a.smithi022.stdout:Jun 28 16:32:42 smithi022 ceph-mon[31875]: pgmap v119: 97 pgs: 64 unknown, 33 active+clean; 577 KiB data, 21 MiB used, 268 GiB / 268 GiB avail; 170 B/s rd, 0 op/s
2021-06-28T16:32:42.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Jun 28 16:32:42 smithi022 ceph-mon[31875]: osdmap e41: 3 total, 3 up, 3 in
2021-06-28T16:32:42.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Jun 28 16:32:42 smithi022 ceph-mon[31875]: mds.? [v2:172.21.15.22:6828/822350855,v1:172.21.15.22:6829/822350855] up:active
2021-06-28T16:32:42.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Jun 28 16:32:42 smithi022 ceph-mon[31875]: fsmap nfs-cephfs:1 {0=a.smithi022.cbqbee=up:active} 1 up:standby
2021-06-28T16:32:42.948 INFO:journalctl@ceph.mon.a.smithi022.stdout:Jun 28 16:32:42 smithi022 ceph-mon[31875]: fsmap nfs-cephfs:1 {0=a.smithi022.cbqbee=up:active} 1 up:standby
2021-06-28T16:32:43.828 INFO:tasks.cephfs_test_runner:test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2021-06-28T16:32:43.828 INFO:tasks.cephfs_test_runner:
2021-06-28T16:32:43.829 INFO:tasks.cephfs_test_runner:======================================================================
2021-06-28T16:32:43.829 INFO:tasks.cephfs_test_runner:ERROR: test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS)
2021-06-28T16:32:43.829 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-06-28T16:32:43.830 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-06-28T16:32:43.830 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_6f6b1570d03f6ac6406bdef18bf76835445f20d8/qa/tasks/cephfs/test_nfs.py", line 321, in test_create_and_delete_export
2021-06-28T16:32:43.830 INFO:tasks.cephfs_test_runner:    self._test_mnt(self.pseudo_path, port, ip)
2021-06-28T16:32:43.831 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_6f6b1570d03f6ac6406bdef18bf76835445f20d8/qa/tasks/cephfs/test_nfs.py", line 259, in _test_mnt
2021-06-28T16:32:43.831 INFO:tasks.cephfs_test_runner:    f'{ip}:{pseudo_path}', '/mnt'])
2021-06-28T16:32:43.831 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in run
2021-06-28T16:32:43.832 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2021-06-28T16:32:43.832 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in <listcomp>
2021-06-28T16:32:43.832 INFO:tasks.cephfs_test_runner:    return [remote.run(**kwargs) for remote in remotes]
2021-06-28T16:32:43.832 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/remote.py", line 509, in run
2021-06-28T16:32:43.833 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2021-06-28T16:32:43.833 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 455, in run
2021-06-28T16:32:43.834 INFO:tasks.cephfs_test_runner:    r.wait()
2021-06-28T16:32:43.834 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 161, in wait
2021-06-28T16:32:43.834 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2021-06-28T16:32:43.835 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 183, in _raise_for_status
2021-06-28T16:32:43.835 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2021-06-28T16:32:43.835 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.22:/cephfs /mnt'

/a/kchai-2021-06-27_13:33:07-rados-wip-kefu-testing-2021-06-27-1907-distro-basic-smithi/6238357

#7 Updated by Sebastian Wagner about 2 years ago

http://pulpito.front.sepia.ceph.com/adking-2021-10-21_19:20:35-rados:cephadm-wip-adk-testing-2021-10-21-1228-distro-basic-smithi/6456286

#8 Updated by Patrick Donnelly over 1 year ago

Target version deleted (~~v17.0.0~~)

#9 Updated by Laura Flores 8 months ago

Tags set to test-failure

/a/lflores-2023-03-27_20:42:09-rados-wip-aclamk-bs-elastic-shared-blob-quincy-distro-default-smithi/7221650

#10 Updated by Nitzan Mordechai 4 months ago

/a/yuriw-2023-08-09_19:52:16-rados-wip-yuri5-testing-2023-08-08-0807-quincy-distro-default-smithi/7364400/

Also available in: Atom PDF

Project

General

Profile

Ceph » CephFS

Issues

Tags

Custom queries