Bug #48873
test_cluster_set_reset_user_config: AssertionError: NFS Ganesha cluster deployment failed
% Done:
0%
Source:
Q/A
Tags:
Backport:
pacific,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
mgr/nfs, qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2021-01-13T11:55:37.749 INFO:tasks.cephfs_test_runner:Starting test: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2021-01-13T11:55:37.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_nfs.TestNFS.test_cluster_set_reset_user_config' 2021-01-13T11:55:38.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl status nfs-server 2021-01-13T11:55:38.777 INFO:teuthology.orchestra.run.smithi043.stdout:* nfs-server.service - NFS server and services 2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout: Loaded: loaded (/lib/systemd/system/nfs-server.service; disabled; vendor preset: enabled) 2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout: Active: inactive (dead) 2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout: 2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:43:35 smithi006 systemd[1]: Starting NFS server and services... 2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:43:35 smithi006 systemd[1]: Started NFS server and services. 2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:55:10 smithi043 systemd[1]: Stopping NFS server and services... 2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:55:10 smithi043 systemd[1]: Stopped NFS server and services. 2021-01-13T11:55:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 3 2021-01-13T11:55:38.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph nfs cluster create cephfs test 2021-01-13T11:55:39.183 INFO:teuthology.orchestra.run.smithi043.stdout:NFS Cluster Created Successfully 2021-01-13T11:55:49.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test 2021-01-13T11:55:49.532 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported 2021-01-13T11:56:09.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test 2021-01-13T11:56:09.883 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported 2021-01-13T11:56:39.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test 2021-01-13T11:56:40.234 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported 2021-01-13T11:57:20.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test 2021-01-13T11:57:20.590 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported 2021-01-13T11:58:10.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test 2021-01-13T11:58:10.942 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported 2021-01-13T11:59:10.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test 2021-01-13T11:59:11.291 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported 2021-01-13T11:59:11.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_cluster_set_reset_user_config' 2021-01-13T11:59:12.048 INFO:tasks.cephfs_test_runner:test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) ... FAIL 2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner: 2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner:====================================================================== 2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner:FAIL: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2021-01-13T11:59:12.050 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-01-13T11:59:12.050 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 462, in test_cluster_set_reset_user_config 2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner: self._test_create_cluster() 2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 123, in _test_create_cluster 2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner: self._check_nfs_cluster_status('running', 'NFS Ganesha cluster deployment failed') 2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 88, in _check_nfs_cluster_status 2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner: self.fail(fail_msg) 2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner:AssertionError: NFS Ganesha cluster deployment failed
Interestingly, previous calls to _test_create_cluster succeeded.
History
#1 Updated by Patrick Donnelly about 3 years ago
- Assignee set to Varsha Rao
- Target version set to v16.0.0
- Backport set to octopus
- Component(FS) mgr/nfs, qa-suite added
- Labels (FS) qa, qa-failure added
#2 Updated by Patrick Donnelly about 3 years ago
- Target version changed from v16.0.0 to v17.0.0
- Backport changed from octopus to pacific,octopus,nautilus
#3 Updated by Varsha Rao about 3 years ago
- Status changed from New to Triaged
- Backport changed from pacific,octopus,nautilus to pacific,octopus
It looks like ganesha daemon did not come up. I cannot reproduce this issue in latest test run:
http://qa-proxy.ceph.com/teuthology/varsha-2021-01-15_16:04:50-rados-wip-nfs-export-update-distro-basic-smithi/5789180/teuthology.log
#4 Updated by Laura Flores about 2 years ago
/a/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728547
#5 Updated by Patrick Donnelly over 1 year ago
- Target version deleted (
v17.0.0)
#6 Updated by Milind Changire 5 months ago
- Assignee changed from Varsha Rao to Dhairya Parmar
2023-10-26T09:36:56.536 INFO:tasks.cephfs_test_runner:Test cluster is created using user config and reverts back to default ... ERROR 2023-10-26T09:36:56.537 INFO:tasks.cephfs_test_runner: 2023-10-26T09:36:56.537 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-26T09:36:56.538 INFO:tasks.cephfs_test_runner:ERROR: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2023-10-26T09:36:56.538 INFO:tasks.cephfs_test_runner:Test cluster is created using user config and reverts back to default 2023-10-26T09:36:56.538 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-26T09:36:56.538 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-26T09:36:56.539 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/cephfs/test_nfs.py", line 645, in test_cluster_set_reset_user_config 2023-10-26T09:36:56.539 INFO:tasks.cephfs_test_runner: self._test_mnt(pseudo_path, port, ip) 2023-10-26T09:36:56.539 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/cephfs/test_nfs.py", line 329, in _test_mnt 2023-10-26T09:36:56.539 INFO:tasks.cephfs_test_runner: self.ctx.cluster.run( 2023-10-26T09:36:56.540 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/cluster.py", line 85, in run 2023-10-26T09:36:56.540 INFO:tasks.cephfs_test_runner: procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] 2023-10-26T09:36:56.540 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/cluster.py", line 85, in <listcomp> 2023-10-26T09:36:56.540 INFO:tasks.cephfs_test_runner: procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] 2023-10-26T09:36:56.541 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run 2023-10-26T09:36:56.541 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-10-26T09:36:56.541 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run 2023-10-26T09:36:56.541 INFO:tasks.cephfs_test_runner: r.wait() 2023-10-26T09:36:56.542 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait 2023-10-26T09:36:56.542 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-10-26T09:36:56.542 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-10-26T09:36:56.542 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-10-26T09:36:56.543 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.118:/ceph /mnt'