Ceph » Orchestrator

/a/yuriw-2022-06-09_22:06:32-rados-wip-yuri3-testing-2022-06-09-1314-distro-default-smithi/6871372
Description: rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_nfs}

This Traceback appeared first. Looks like possibly a cephadm symptom caused by a cephfs bug:

2022-06-09T22:59:15.108 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: debug 2022-06-09T22:59:15.102+0000 7f29c3b38700 -1 log_channel(cephadm) log [ERR] : Failed to apply mds.a spec MDSSpec.from_json(yaml.safe_load('''service_type: mds
2022-06-09T22:59:15.109 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: service_id: a
2022-06-09T22:59:15.110 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: service_name: mds.a
2022-06-09T22:59:15.110 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: placement:
2022-06-09T22:59:15.112 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   count: 2
2022-06-09T22:59:15.112 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: ''')): cephadm exited with an error code: 1, stderr: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`
2022-06-09T22:59:15.112 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: Traceback (most recent call last):
2022-06-09T22:59:15.113 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 507, in _apply_all_services
2022-06-09T22:59:15.113 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     if self._apply_service(spec):
2022-06-09T22:59:15.113 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 829, in _apply_service
2022-06-09T22:59:15.113 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     self._remove_daemon(d.name(), d.hostname)
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1223, in _remove_daemon
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     host, name, 'rm-daemon', args))
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     return self.event_loop.get_result(coro)
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result
2022-06-09T22:59:15.115 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
2022-06-09T22:59:15.115 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
2022-06-09T22:59:15.115 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     return self.__get_result()
2022-06-09T22:59:15.116 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2022-06-09T22:59:15.279 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     raise self._exception
2022-06-09T22:59:15.280 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1366, in _run_cephadm
2022-06-09T22:59:15.280 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     f'cephadm exited with an error code: {code}, stderr: {err}')
2022-06-09T22:59:15.280 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`

Later in the teuthology log, `test_update_export` failed, which looks similar to Tracker #55808.

2022-06-09T23:11:48.292 INFO:tasks.cephfs_test_runner:test_update_export (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2022-06-09T23:11:48.294 INFO:tasks.cephfs_test_runner:
2022-06-09T23:11:48.294 INFO:tasks.cephfs_test_runner:======================================================================
2022-06-09T23:11:48.295 INFO:tasks.cephfs_test_runner:ERROR: test_update_export (tasks.cephfs.test_nfs.TestNFS)
2022-06-09T23:11:48.295 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-06-09T23:11:48.296 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-06-09T23:11:48.297 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_db6e20bf340a0dc7427e79a4a30764ff6f613604/qa/tasks/cephfs/test_nfs.py", line 642, in test_update_export
2022-06-09T23:11:48.297 INFO:tasks.cephfs_test_runner:    if not self._check_nfs_cluster_event('restart'):
2022-06-09T23:11:48.298 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_db6e20bf340a0dc7427e79a4a30764ff6f613604/qa/tasks/cephfs/test_nfs.py", line 91, in _check_nfs_cluster_event
2022-06-09T23:11:48.299 INFO:tasks.cephfs_test_runner:    for event in daemons_details[0]['events']:
2022-06-09T23:11:48.300 INFO:tasks.cephfs_test_runner:KeyError: 'events'
2022-06-09T23:11:48.300 INFO:tasks.cephfs_test_runner:
2022-06-09T23:11:48.302 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------