Ceph » Orchestrator

/a/yuriw-2022-02-16_15:53:49-rados-wip-yuri11-testing-2022-02-15-1643-distro-default-smithi/6688708

Second-to-last time the nfs cluster is created:

2022-02-16T17:41:33.980 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:41:33 smithi065 bash[13997]: audit 2022-02-16T17:41:32.101094+0000 mgr.a (mgr.14306) 412 : audit [DBG] from='client.14651 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "test", "target": ["mon-mgr", ""]}]: dispatch
2022-02-16T17:41:33.981 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:41:33 smithi065 bash[13997]: cephadm 2022-02-16T17:41:32.107964+0000 mgr.a (mgr.14306) 413 : cephadm [INF] Saving service nfs.test spec with placement count:1
2022-02-16T17:41:33.981 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:41:33 smithi065 bash[13997]: cephadm 2022-02-16T17:41:32.167065+0000 mgr.a (mgr.14306) 414 : cephadm [INF] Creating key for client.nfs.test.0.0.smithi065.dwycpc
2022-02-16T17:41:33.981 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:41:33 smithi065 bash[13997]: cephadm 2022-02-16T17:41:32.175620+0000 mgr.a (mgr.14306) 415 : cephadm [INF] Ensuring nfs.test.0 is in the ganesha grace table
2022-02-16T17:41:33.982 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:41:33 smithi065 bash[13997]: cephadm 2022-02-16T17:41:32.315647+0000 mgr.a (mgr.14306) 416 : cephadm [INF] Rados config object exists: conf-nfs.test
2022-02-16T17:41:33.982 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:41:33 smithi065 bash[13997]: cephadm 2022-02-16T17:41:32.315885+0000 mgr.a (mgr.14306) 417 : cephadm [INF] Creating key for client.nfs.test.0.0.smithi065.dwycpc-rgw

Last time the nfs cluster was called to be created. No actions like "Saving service..." or "Creating key for..." follow this call, indicating that the nfs cluster was not successfully created.

2022-02-16T17:42:29.731 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:42:29 smithi065 bash[13997]: audit 2022-02-16T17:42:28.179314+0000 mgr.a (mgr.14306) 466 : audit [DBG] from='client.14687 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "test", "target": ["mon-mgr", ""]}]: dispatch
2022-02-16T17:42:29.732 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:42:29 smithi065 bash[13997]: audit 2022-02-16T17:42:28.363818+0000 mon.a (mon.0) 1214 : audit [INF] from='mgr.14306 172.21.15.65:0/628062573' entity='mgr.a'
2022-02-16T17:42:29.732 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:42:29 smithi065 bash[13997]: audit 2022-02-16T17:42:28.369532+0000 mon.a (mon.0) 1215 : audit [INF] from='mgr.14306 172.21.15.65:0/628062573' entity='mgr.a'
2022-02-16T17:42:29.732 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:42:29 smithi065 bash[13997]: audit 2022-02-16T17:42:29.358055+0000 mon.a (mon.0) 1216 : audit [DBG] from='mgr.14306 172.21.15.65:0/628062573' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
2022-02-16T17:42:29.732 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:42:29 smithi065 bash[13997]: audit 2022-02-16T17:42:29.360212+0000 mon.a (mon.0) 1217 : audit [DBG] from='mgr.14306 172.21.15.65:0/628062573' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2022-02-16T17:42:30.730 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:42:30 smithi065 bash[13997]: cluster 2022-02-16T17:42:29.272416+0000 mgr.a (mgr.14306) 467 : cluster [DBG] pgmap v278: 97 pgs: 97 active+clean; 594 KiB data, 35 MiB used, 268 GiB / 268 GiB avail; 85 B/s wr, 0 op/s
2022-02-16T17:42:30.730 INFO:journalctl@ceph.mon.a.smithi065.stdout:Feb 16 17:42:30 smithi065 bash[13997]: audit 2022-02-16T17:42:29.361964+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.14306 172.21.15.65:0/628062573' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch

...

2022-02-16T17:45:59.809 DEBUG:teuthology.orchestra.run.smithi065:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test
2022-02-16T17:46:00.108 INFO:teuthology.orchestra.run.smithi065.stdout:No daemons reported
2022-02-16T17:46:00.124 DEBUG:teuthology.orchestra.run.smithi065:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_nfs.TestNFS.test_create_multiple_exports'
2022-02-16T17:46:00.791 INFO:tasks.cephfs_test_runner:test_create_multiple_exports (tasks.cephfs.test_nfs.TestNFS) ... FAIL
2022-02-16T17:46:00.792 INFO:tasks.cephfs_test_runner:
2022-02-16T17:46:00.792 INFO:tasks.cephfs_test_runner:======================================================================
2022-02-16T17:46:00.793 INFO:tasks.cephfs_test_runner:FAIL: test_create_multiple_exports (tasks.cephfs.test_nfs.TestNFS)
2022-02-16T17:46:00.793 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-02-16T17:46:00.793 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-02-16T17:46:00.794 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_c11e21b2a403e128a89552f2aa1019a3a9f8a012/qa/tasks/cephfs/test_nfs.py", line 367, in test_create_multiple_exports
2022-02-16T17:46:00.794 INFO:tasks.cephfs_test_runner:    self._create_default_export()
2022-02-16T17:46:00.794 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_c11e21b2a403e128a89552f2aa1019a3a9f8a012/qa/tasks/cephfs/test_nfs.py", line 186, in _create_default_export
2022-02-16T17:46:00.795 INFO:tasks.cephfs_test_runner:    self._test_create_cluster()
2022-02-16T17:46:00.795 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_c11e21b2a403e128a89552f2aa1019a3a9f8a012/qa/tasks/cephfs/test_nfs.py", line 125, in _test_create_cluster
2022-02-16T17:46:00.795 INFO:tasks.cephfs_test_runner:    self._check_nfs_cluster_status('running', 'NFS Ganesha cluster deployment failed')
2022-02-16T17:46:00.796 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_c11e21b2a403e128a89552f2aa1019a3a9f8a012/qa/tasks/cephfs/test_nfs.py", line 89, in _check_nfs_cluster_status
2022-02-16T17:46:00.796 INFO:tasks.cephfs_test_runner:    self.fail(fail_msg)
2022-02-16T17:46:00.796 INFO:tasks.cephfs_test_runner:AssertionError: NFS Ganesha cluster deployment failed
2022-02-16T17:46:00.796 INFO:tasks.cephfs_test_runner:
2022-02-16T17:46:00.797 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-02-16T17:46:00.797 INFO:tasks.cephfs_test_runner:Ran 12 tests in 735.039s

Another example:
/a/matan-2022-02-13_13:18:27-orch-master-distro-basic-smithi/6680199