Project

General

Profile

Bug #56000

task/test_nfs: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`

Added by Laura Flores 6 months ago. Updated 4 days ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
quincy, pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2022-06-09_22:06:32-rados-wip-yuri3-testing-2022-06-09-1314-distro-default-smithi/6871372
Description: rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_nfs}

This Traceback appeared first. Looks like possibly a cephadm symptom caused by a cephfs bug:

2022-06-09T22:59:15.108 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: debug 2022-06-09T22:59:15.102+0000 7f29c3b38700 -1 log_channel(cephadm) log [ERR] : Failed to apply mds.a spec MDSSpec.from_json(yaml.safe_load('''service_type: mds
2022-06-09T22:59:15.109 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: service_id: a
2022-06-09T22:59:15.110 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: service_name: mds.a
2022-06-09T22:59:15.110 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: placement:
2022-06-09T22:59:15.112 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   count: 2
2022-06-09T22:59:15.112 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: ''')): cephadm exited with an error code: 1, stderr: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`
2022-06-09T22:59:15.112 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: Traceback (most recent call last):
2022-06-09T22:59:15.113 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 507, in _apply_all_services
2022-06-09T22:59:15.113 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     if self._apply_service(spec):
2022-06-09T22:59:15.113 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 829, in _apply_service
2022-06-09T22:59:15.113 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     self._remove_daemon(d.name(), d.hostname)
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1223, in _remove_daemon
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     host, name, 'rm-daemon', args))
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     return self.event_loop.get_result(coro)
2022-06-09T22:59:15.114 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result
2022-06-09T22:59:15.115 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
2022-06-09T22:59:15.115 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
2022-06-09T22:59:15.115 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     return self.__get_result()
2022-06-09T22:59:15.116 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2022-06-09T22:59:15.279 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     raise self._exception
2022-06-09T22:59:15.280 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1366, in _run_cephadm
2022-06-09T22:59:15.280 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]:     f'cephadm exited with an error code: {code}, stderr: {err}')
2022-06-09T22:59:15.280 INFO:journalctl@ceph.mgr.a.smithi060.stdout:Jun 09 22:59:15 smithi060 bash[39018]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`

Later in the teuthology log, `test_update_export` failed, which looks similar to Tracker #55808.

2022-06-09T23:11:48.292 INFO:tasks.cephfs_test_runner:test_update_export (tasks.cephfs.test_nfs.TestNFS) ... ERROR
2022-06-09T23:11:48.294 INFO:tasks.cephfs_test_runner:
2022-06-09T23:11:48.294 INFO:tasks.cephfs_test_runner:======================================================================
2022-06-09T23:11:48.295 INFO:tasks.cephfs_test_runner:ERROR: test_update_export (tasks.cephfs.test_nfs.TestNFS)
2022-06-09T23:11:48.295 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-06-09T23:11:48.296 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-06-09T23:11:48.297 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_db6e20bf340a0dc7427e79a4a30764ff6f613604/qa/tasks/cephfs/test_nfs.py", line 642, in test_update_export
2022-06-09T23:11:48.297 INFO:tasks.cephfs_test_runner:    if not self._check_nfs_cluster_event('restart'):
2022-06-09T23:11:48.298 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_db6e20bf340a0dc7427e79a4a30764ff6f613604/qa/tasks/cephfs/test_nfs.py", line 91, in _check_nfs_cluster_event
2022-06-09T23:11:48.299 INFO:tasks.cephfs_test_runner:    for event in daemons_details[0]['events']:
2022-06-09T23:11:48.300 INFO:tasks.cephfs_test_runner:KeyError: 'events'
2022-06-09T23:11:48.300 INFO:tasks.cephfs_test_runner:
2022-06-09T23:11:48.302 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------


Related issues

Duplicates Orchestrator - Bug #55808: task/test_nfs: KeyError: 'events' Resolved

History

#1 Updated by Laura Flores 6 months ago

  • Related to Bug #55808: task/test_nfs: KeyError: 'events' added

#2 Updated by Ramana Raja 6 months ago

  • Assignee set to Ramana Raja

#3 Updated by Venky Shankar 6 months ago

  • Target version set to v18.0.0
  • Backport set to quincy, pacific
  • Labels (FS) qa-failure added

#4 Updated by Laura Flores 6 months ago

/a/yuriw-2022-06-11_02:24:12-rados-quincy-release-distro-default-smithi/6873817

#5 Updated by Ramana Raja 6 months ago

  • Project changed from CephFS to Orchestrator
  • Status changed from New to Duplicate
  • Assignee deleted (Ramana Raja)

The teuthology run, /a/yuriw-2022-06-09_22:06:32-rados-wip-yuri3-testing-2022-06-09-1314-distro-default-smithi/6871372 , linked in this bug failed due to the error raised by "test_update_export".

This "test_update_export" failure is the same issue as https://tracker.ceph.com/issues/55808

#6 Updated by Laura Flores 6 months ago

/a/yuriw-2022-06-11_02:24:12-rados-quincy-release-distro-default-smithi/6873817

#7 Updated by Laura Flores 4 months ago

  • Related to deleted (Bug #55808: task/test_nfs: KeyError: 'events')

#8 Updated by Laura Flores 4 months ago

  • Duplicates Bug #55808: task/test_nfs: KeyError: 'events' added

#9 Updated by Laura Flores 6 days ago

/a/yuriw-2022-11-28_21:21:22-rados-wip-yuri5-testing-2022-11-18-1554-quincy-distro-default-smithi/7095434

#10 Updated by Laura Flores 4 days ago

/a/yuriw-2022-11-28_21:09:37-rados-wip-yuri4-testing-2022-11-10-1051-distro-default-smithi/7094866

Also available in: Atom PDF