Project

General

Profile

Actions

Bug #59530

open

mgr-nfs-upgrade: mds.foofs has 0/2

Added by Laura Flores 12 months ago. Updated 11 months ago.

Status:
Triaged
Priority:
Normal
Assignee:
Category:
Correctness/Safety
Target version:
% Done:

0%

Source:
Tags:
Backport:
reef,quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2023-04-06_15:37:58-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7234302

2023-04-06T17:18:29.812 INFO:teuthology.orchestra.run.smithi032.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-04-06T16:58:44.270875Z", "last_refresh": "2023-04-06T17:01:12.273256Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-04-06T16:58:31.723002Z", "last_refresh": "2023-04-06T17:00:55.018699Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-04-06T16:58:42.138873Z", "last_refresh": "2023-04-06T17:01:12.273379Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-04-06T17:03:14.635859Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-04-06T17:03:14.627014Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-04-06T16:58:30.565287Z", "last_refresh": "2023-04-06T17:00:55.018846Z", "running": 2, "size": 2}}, {"events": ["2023-04-06T16:59:30.458608Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi032:172.21.15.32=smithi032", "smithi093:172.21.15.93=smithi093"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-04-06T16:59:30.456677Z", "last_refresh": "2023-04-06T17:00:55.018914Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-04-06T16:58:43.161934Z", "last_refresh": "2023-04-06T17:00:55.018976Z", "ports": [9100], "running": 2, "size": 2}}, {"placement": {}, "service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "6933c2a0b7ddc222e77dc8d9dc471a0c639a1c0bded5077be53ad3a9557b4355", "container_image_name": "docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-04-06T16:58:40.476534Z", "last_refresh": "2023-04-06T17:01:12.273496Z", "ports": [9095], "running": 1, "size": 1}}]
2023-04-06T17:18:29.894 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Apr 06 17:18:29 smithi093 ceph-1d7dc652-d49c-11ed-9aff-001a4aab830c-mon.smithi093[133644]: cluster 2023-04-06T17:18:28.336011+0000 mgr.smithi032.faijfv (mgr.14164) 957 : cluster [DBG] pgmap v630: 65 pgs: 65 active+clean; 0 B data, 44 MiB used, 715 GiB / 715 GiB avail
2023-04-06T17:18:30.273 INFO:tasks.cephadm:mds.foofs has 0/2
2023-04-06T17:18:30.273 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/run_tasks.py", line 84, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5c985a09c75a9cc73e9774aadc248a51b2319e16/qa/tasks/cephadm.py", line 1114, in wait_for_service
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (300) after waiting for 300 seconds
2023-04-06T17:18:30.336 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e0d66fbf392b42e9ae4e4860d1d06a1d
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/run_tasks.py", line 84, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5c985a09c75a9cc73e9774aadc248a51b2319e16/qa/tasks/cephadm.py", line 1114, in wait_for_service
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (300) after waiting for 300 seconds


Related issues 1 (1 open0 closed)

Related to Orchestrator - Bug #59529: cluster upgrade stuck with OSDs and MDSs not upgraded.TriagedAdam King

Actions
Actions #1

Updated by Venky Shankar 12 months ago

  • Category set to Correctness/Safety
  • Status changed from New to Triaged
  • Assignee set to Venky Shankar
  • Target version set to v19.0.0
  • Backport set to reef,quincy,pacific
Actions #2

Updated by Laura Flores 12 months ago

/a/yuriw-2023-04-26_20:20:05-rados-pacific-release-distro-default-smithi/7255292

2023-04-27T01:19:59.802 INFO:teuthology.orchestra.run.smithi163.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-04-27T01:01:17.196057Z", "last_refresh": "2023-04-27T01:03:25.035388Z", "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-04-27T01:01:06.004774Z", "last_refresh": "2023-04-27T01:03:22.660324Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-04-27T01:01:15.266400Z", "last_refresh": "2023-04-27T01:03:25.035507Z", "running": 1, "size": 1}}, {"events": ["2023-04-27T01:05:35.898440Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-04-27T01:05:35.892172Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-04-27T01:01:04.768961Z", "last_refresh": "2023-04-27T01:03:22.660463Z", "running": 2, "size": 2}}, {"events": ["2023-04-27T01:01:55.290792Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi163:172.21.15.163=smithi163", "smithi188:172.21.15.188=smithi188"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-04-27T01:01:55.288191Z", "last_refresh": "2023-04-27T01:03:22.660533Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-04-27T01:01:16.232830Z", "last_refresh": "2023-04-27T01:03:22.660607Z", "running": 2, "size": 2}}, {"placement": {}, "service_id": "unmanaged", "service_name": "osd.unmanaged", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29", "container_image_name": "docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-04-27T01:01:14.042606Z", "last_refresh": "2023-04-27T01:03:25.035659Z", "running": 1, "size": 1}}]
2023-04-27T01:20:00.143 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Apr 27 01:20:00 smithi163 ceph-dab30822-e496-11ed-9b00-001a4aab830c-mon.smithi163[128476]: debug 2023-04-27T01:19:59.999+0000 7f2128cf7700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds
2023-04-27T01:20:00.173 INFO:tasks.cephadm:mds.foofs has 0/2
Actions #3

Updated by Laura Flores 12 months ago

  • Related to Bug #59529: cluster upgrade stuck with OSDs and MDSs not upgraded. added
Actions #4

Updated by Laura Flores 12 months ago

/a/yuriw-2023-04-25_18:56:08-rados-wip-yuri5-testing-2023-04-25-0837-pacific-distro-default-smithi/7252409

Actions #6

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-17_19:39:18-rados-wip-yuri5-testing-2023-05-09-1324-pacific-distro-default-smithi/7276765

Actions

Also available in: Atom PDF