Bug #55808
task/test_nfs: KeyError: 'events'
0%
Description
/a/yuriw-2022-05-27_21:59:17-rados-wip-yuri-testing-2022-05-27-0934-distro-default-smithi/6851013
2022-05-27T22:51:21.158 INFO:tasks.cephfs_test_runner:======================================================================
2022-05-27T22:51:21.158 INFO:tasks.cephfs_test_runner:ERROR: test_update_export (tasks.cephfs.test_nfs.TestNFS)
2022-05-27T22:51:21.158 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-05-27T22:51:21.159 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-05-27T22:51:21.159 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_a847be1e97ec9c953b2352ff329516e826d4d048/qa/tasks/cephfs/test_nfs.py", line 642, in test_update_export
2022-05-27T22:51:21.159 INFO:tasks.cephfs_test_runner: if not self._check_nfs_cluster_event('restart'):
2022-05-27T22:51:21.159 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_a847be1e97ec9c953b2352ff329516e826d4d048/qa/tasks/cephfs/test_nfs.py", line 91, in _check_nfs_cluster_event
2022-05-27T22:51:21.159 INFO:tasks.cephfs_test_runner: for event in daemons_details[0]['events']:
2022-05-27T22:51:21.159 INFO:tasks.cephfs_test_runner:KeyError: 'events'
2022-05-27T22:51:21.159 INFO:tasks.cephfs_test_runner:
2022-05-27T22:51:21.159 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
Related issues
History
#1 Updated by Laura Flores almost 2 years ago
- Project changed from Orchestrator to CephFS
- Component(FS) mgr/nfs added
#2 Updated by Laura Flores almost 2 years ago
/a/yuriw-2022-06-03_14:09:08-rados-wip-yuri7-testing-2022-06-02-1633-distro-default-smithi/6862503
#3 Updated by Ramana Raja almost 2 years ago
- Assignee set to Ramana Raja
#4 Updated by Laura Flores almost 2 years ago
- Related to Bug #56000: task/test_nfs: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls` added
#5 Updated by Ramana Raja almost 2 years ago
In the logs,
2022-06-03T15:24:59.294 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph orch ps --service_name=nfs.test --format=json
'events' key is missing in the output of `ceph orch ps --service_name=nfs.test --format=json` command.
2022-06-03T15:24:59.670 INFO:teuthology.orchestra.run.smithi036.stdout:[{"container_id": "320b4a67e293", "container_image_digests": ["quay.ceph.io/ceph-ci/ceph@sha256:76c5b9629cc5219226e93c4f515f3c20ae34e55b1ec935b2d8316b0939ee88a2"], "container_image_id": "cde988ec5559676e7b593f63c6eed00fd55715b7ec061d379984cfb9176b68d7", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:76c5b9629cc5219226e93c4f515f3c20ae34e55b1ec935b2d8316b0939ee88a2", "cpu_percentage": "0.20%", "created": "2022-06-03T15:23:03.782604Z", "daemon_id": "test.0.0.smithi036.xvvoix", "daemon_name": "nfs.test.0.0.smithi036.xvvoix", "daemon_type": "nfs", "hostname": "smithi036", "is_active": false, "last_refresh": "2022-06-03T15:24:51.711974Z", "memory_usage": 44449136, "ports": [2049], "rank": 0, "rank_generation": 0, "service_name": "nfs.test", "started": "2022-06-03T15:23:03.886913Z", "status": 1, "status_desc": "running", "version": "3.5"}] 2022-06-03T15:24:59.672 INFO:tasks.cephfs.test_nfs:daemons details [{'container_id': '320b4a67e293', 'container_image_digests': ['quay.ceph.io/ceph-ci/ceph@sha256:76c5b9629cc5219226e93c4f515f3c20ae34e55b1ec935b2d8316b0939ee88a2'], 'container_image_id': 'cde988ec5559676e7b593f63c6eed00fd55715b7ec061d379984cfb9176b68d7', 'container_image_name': 'quay.ceph.io/ceph-ci/ceph@sha256:76c5b9629cc5219226e93c4f515f3c20ae34e55b1ec935b2d8316b0939ee88a2', 'cpu_percentage': '0.20%', 'created': '2022-06-03T15:23:03.782604Z', 'daemon_id': 'test.0.0.smithi036.xvvoix', 'daemon_name': 'nfs.test.0.0.smithi036.xvvoix', 'daemon_type': 'nfs', 'hostname': 'smithi036', 'is_active': False, 'last_refresh': '2022-06-03T15:24:51.711974Z', 'memory_usage': 44449136, 'ports': [2049], 'rank': 0, 'rank_generation': 0, 'service_name': 'nfs.test', 'started': '2022-06-03T15:23:03.886913Z', 'status': 1, 'status_desc': 'running', 'version': '3.5'}]
I'm not sure why "events" key is not in the output of `ceph orch ps --format=json` command. I see it locally with ceph main branch using vstart cluster.
#6 Updated by Ramana Raja almost 2 years ago
"test_nfs" consistently passes in the same run when agent is off. See,
https://pulpito.ceph.com/yuriw-2022-05-27_21:59:17-rados-wip-yuri-testing-2022-05-27-0934-distro-default-smithi/6851203/
https://pulpito.ceph.com/yuriw-2022-05-27_21:59:17-rados-wip-yuri-testing-2022-05-27-0934-distro-default-smithi/6851319/
"test_nfs" fails when the agent is on as seen in this tracker ticker description and https://tracker.ceph.com/issues/56000
https://pulpito.ceph.com/yuriw-2022-06-09_22:06:32-rados-wip-yuri3-testing-2022-06-09-1314-distro-default-smithi/6871372/
It's possibly an issue with cephadm agent.
#7 Updated by Laura Flores almost 2 years ago
@Ramana we can re-file this as an Orchestrator bug if you believe it's a problem with cephadm.
#8 Updated by Ramana Raja almost 2 years ago
- Project changed from CephFS to Orchestrator
This most likely is a cephadm bug, where `ceph orch ps ceph --service_name=nfs.<nfs cluster name> --format=json` does not output "events" when adm agent is on. Changing this to an orchestrator bug.
#9 Updated by Ramana Raja almost 2 years ago
- Assignee deleted (
Ramana Raja)
#10 Updated by Laura Flores almost 2 years ago
- Backport set to quincy
/a/yuriw-2022-06-16_16:41:04-rados-wip-yuri6-testing-2022-06-16-0651-quincy-distro-default-smithi/6882321$
#11 Updated by Adam King over 1 year ago
- Status changed from New to Pending Backport
- Pull request ID set to 46897
#12 Updated by Backport Bot over 1 year ago
- Copied to Backport #56584: quincy: task/test_nfs: KeyError: 'events' added
#13 Updated by Laura Flores over 1 year ago
- Related to deleted (Bug #56000: task/test_nfs: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`)
#14 Updated by Laura Flores over 1 year ago
- Duplicated by Bug #56000: task/test_nfs: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls` added
#15 Updated by Backport Bot over 1 year ago
- Tags set to backport_processed
#16 Updated by Adam King over 1 year ago
- Status changed from Pending Backport to Resolved
#17 Updated by Laura Flores over 1 year ago
- Duplicated by deleted (Bug #56000: task/test_nfs: ERROR: Daemon not found: mds.a.smithi060.ujwxef. See `cephadm ls`)