Project

General

Profile

Bug #48873

test_cluster_set_reset_user_config: AssertionError: NFS Ganesha cluster deployment failed

Added by Sebastian Wagner about 3 years ago. Updated 5 months ago.

Status:
Triaged
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
pacific,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
mgr/nfs, qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

https://pulpito.ceph.com/swagner-2021-01-13_11:19:08-rados:cephadm-wip-swagner3-testing-2021-01-12-1316-distro-basic-smithi/5783002/

2021-01-13T11:55:37.749 INFO:tasks.cephfs_test_runner:Starting test: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2021-01-13T11:55:37.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_nfs.TestNFS.test_cluster_set_reset_user_config'
2021-01-13T11:55:38.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl status nfs-server
2021-01-13T11:55:38.777 INFO:teuthology.orchestra.run.smithi043.stdout:* nfs-server.service - NFS server and services
2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout:   Loaded: loaded (/lib/systemd/system/nfs-server.service; disabled; vendor preset: enabled)
2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout:   Active: inactive (dead)
2021-01-13T11:55:38.778 INFO:teuthology.orchestra.run.smithi043.stdout:
2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:43:35 smithi006 systemd[1]: Starting NFS server and services...
2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:43:35 smithi006 systemd[1]: Started NFS server and services.
2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:55:10 smithi043 systemd[1]: Stopping NFS server and services...
2021-01-13T11:55:38.779 INFO:teuthology.orchestra.run.smithi043.stdout:Jan 13 11:55:10 smithi043 systemd[1]: Stopped NFS server and services.
2021-01-13T11:55:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 3
2021-01-13T11:55:38.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph nfs cluster create cephfs test
2021-01-13T11:55:39.183 INFO:teuthology.orchestra.run.smithi043.stdout:NFS Cluster Created Successfully
2021-01-13T11:55:49.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:55:49.532 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:56:09.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:56:09.883 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:56:39.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:56:40.234 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:57:20.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:57:20.590 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:58:10.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:58:10.942 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:59:10.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2021-01-13T11:59:11.291 INFO:teuthology.orchestra.run.smithi043.stdout:No daemons reported
2021-01-13T11:59:11.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_cluster_set_reset_user_config'
2021-01-13T11:59:12.048 INFO:tasks.cephfs_test_runner:test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) ... FAIL
2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner:
2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner:======================================================================
2021-01-13T11:59:12.049 INFO:tasks.cephfs_test_runner:FAIL: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2021-01-13T11:59:12.050 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-01-13T11:59:12.050 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 462, in test_cluster_set_reset_user_config
2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner:    self._test_create_cluster()
2021-01-13T11:59:12.051 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 123, in _test_create_cluster
2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner:    self._check_nfs_cluster_status('running', 'NFS Ganesha cluster deployment failed')
2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2021-01-12-1316/qa/tasks/cephfs/test_nfs.py", line 88, in _check_nfs_cluster_status
2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner:    self.fail(fail_msg)
2021-01-13T11:59:12.052 INFO:tasks.cephfs_test_runner:AssertionError: NFS Ganesha cluster deployment failed

Interestingly, previous calls to _test_create_cluster succeeded.

History

#1 Updated by Patrick Donnelly about 3 years ago

  • Assignee set to Varsha Rao
  • Target version set to v16.0.0
  • Backport set to octopus
  • Component(FS) mgr/nfs, qa-suite added
  • Labels (FS) qa, qa-failure added

#2 Updated by Patrick Donnelly about 3 years ago

  • Target version changed from v16.0.0 to v17.0.0
  • Backport changed from octopus to pacific,octopus,nautilus

#3 Updated by Varsha Rao about 3 years ago

  • Status changed from New to Triaged
  • Backport changed from pacific,octopus,nautilus to pacific,octopus

It looks like ganesha daemon did not come up. I cannot reproduce this issue in latest test run:
http://qa-proxy.ceph.com/teuthology/varsha-2021-01-15_16:04:50-rados-wip-nfs-export-update-distro-basic-smithi/5789180/teuthology.log

#4 Updated by Laura Flores about 2 years ago

/a/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728547

#5 Updated by Patrick Donnelly over 1 year ago

  • Target version deleted (v17.0.0)

#6 Updated by Milind Changire 5 months ago

  • Assignee changed from Varsha Rao to Dhairya Parmar
2023-10-26T09:36:56.536 INFO:tasks.cephfs_test_runner:Test cluster is created using user config and reverts back to default ... ERROR
2023-10-26T09:36:56.537 INFO:tasks.cephfs_test_runner:
2023-10-26T09:36:56.537 INFO:tasks.cephfs_test_runner:======================================================================
2023-10-26T09:36:56.538 INFO:tasks.cephfs_test_runner:ERROR: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2023-10-26T09:36:56.538 INFO:tasks.cephfs_test_runner:Test cluster is created using user config and reverts back to default
2023-10-26T09:36:56.538 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2023-10-26T09:36:56.538 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2023-10-26T09:36:56.539 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/cephfs/test_nfs.py", line 645, in test_cluster_set_reset_user_config
2023-10-26T09:36:56.539 INFO:tasks.cephfs_test_runner:    self._test_mnt(pseudo_path, port, ip)
2023-10-26T09:36:56.539 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/cephfs/test_nfs.py", line 329, in _test_mnt
2023-10-26T09:36:56.539 INFO:tasks.cephfs_test_runner:    self.ctx.cluster.run(
2023-10-26T09:36:56.540 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/cluster.py", line 85, in run
2023-10-26T09:36:56.540 INFO:tasks.cephfs_test_runner:    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
2023-10-26T09:36:56.540 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/cluster.py", line 85, in <listcomp>
2023-10-26T09:36:56.540 INFO:tasks.cephfs_test_runner:    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
2023-10-26T09:36:56.541 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run
2023-10-26T09:36:56.541 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2023-10-26T09:36:56.541 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run
2023-10-26T09:36:56.541 INFO:tasks.cephfs_test_runner:    r.wait()
2023-10-26T09:36:56.542 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait
2023-10-26T09:36:56.542 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2023-10-26T09:36:56.542 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status
2023-10-26T09:36:56.542 INFO:tasks.cephfs_test_runner:    raise CommandFailedError(
2023-10-26T09:36:56.543 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 32: 'sudo mount -t nfs -o port=2049 172.21.15.118:/ceph /mnt'

https://pulpito.ceph.com/yuriw-2023-10-25_14:39:02-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7437170/

Also available in: Atom PDF