Project

General

Profile

Actions

Bug #57482

open

cephadm/smoke-roleless: nfs-ingress test times out

Added by Laura Flores over 1 year ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.front.sepia.ceph.com/amathuri-2022-09-08_14:19:23-rados-wip-yuri5-testing-2022-09-06-1334-pacific-distro-default-smithi/7020907/

Description: rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}

Fails due to "reached maximum tries (300) after waiting for 300 seconds"

Events leading up to the Traceback look like this:

2022-09-08T21:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout:
2022-09-08T21:08:10.072 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2022-09-08T20:30:12.783776Z", "last_refresh": "2022-09-08T21:07:15.446221Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2022-09-08T20:30:02.589006Z", "last_refresh": "2022-09-08T21:06:07.919047Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2022-09-08T20:30:07.064243Z", "last_refresh": "2022-09-08T21:07:15.446431Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2022-09-08T20:39:17.949473Z service:ingress.nfs.foo [INFO] \"service was created\"", "2022-09-08T20:40:47.133522Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi179.zxqaiu on smithi179: Unable to identify interface for 10.0.31.8/16 on smithi179\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.8/16"}, "status": {"created": "2022-09-08T20:39:17.944601Z", "last_refresh": "2022-09-08T21:06:07.921675Z", "ports": [2049, 9002], "running": 3, "size": 4, "virtual_ip": "10.0.31.8/16"}}, {"events": ["2022-09-08T20:39:11.603344Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2022-09-08T20:39:11.594721Z", "last_refresh": "2022-09-08T21:06:07.921193Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2022-09-08T20:30:00.364547Z", "last_refresh": "2022-09-08T21:06:07.919461Z", "running": 2, "size": 2}}, {"events": ["2022-09-08T20:31:46.780582Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi179:172.21.15.179=smithi179"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2022-09-08T20:31:46.775377Z", "last_refresh": "2022-09-08T21:06:07.919726Z", "running": 2, "size": 2}}, {"events": ["2022-09-08T20:39:17.944267Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2022-09-08T20:39:17.937224Z", "last_refresh": "2022-09-08T21:06:07.921435Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2022-09-08T20:30:09.286316Z", "last_refresh": "2022-09-08T21:06:07.919973Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2022-09-08T20:32:36.819653Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2022-09-08T20:32:36.814585Z", "last_refresh": "2022-09-08T21:06:07.920214Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2022-09-08T20:30:04.819581Z", "last_refresh": "2022-09-08T21:07:15.446660Z", "ports": [9095], "running": 1, "size": 1}}]
2022-09-08T21:08:10.826 INFO:tasks.cephadm:ingress.nfs.foo has 3/4

Actions #1

Updated by Adam King over 1 year ago

Failed while placing keepalived.nfs.foo.smithi179.zxqaiu on smithi179: Unable to identify interface for 10.0.31.8/16 on smithi179

looks like the important bit. Will have to investigate how that VIP was picked (it might be hardcoded, haven't checked yet) and if the relevant interface actually isn't present on the nodes or if keepalived is failing to see something.

Actions

Also available in: Atom PDF