Project

General

Profile

Actions
Copy link

Bug #57255

open

rados/cephadm/mds_upgrade_sequence, pacific : cephadm [ERR] Upgrade: Paused due to UPGRADE_NO_STANDBY_MGR: Upgrade: Need standby mgr daemon

Added by Prashant D over 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

mgrmap from mon log:

2022-08-19T21:38:01.213+0000 7f749d751700  0 log_channel(cluster) log [DBG] : mgrmap e1: no daemons active
2022-08-19T21:38:04.885+0000 7f5bd8c0c700  0 log_channel(cluster) log [DBG] : mgrmap e1: no daemons active
2022-08-19T21:38:11.412+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e2: smithi124.pdzpkv(active, starting, since 0.00261735s)
2022-08-19T21:38:12.415+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e3: smithi124.pdzpkv(active, since 1.00527s)
2022-08-19T21:38:14.417+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e4: smithi124.pdzpkv(active, since 3s)
2022-08-19T21:38:19.420+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e5: smithi124.pdzpkv(active, starting, since 0.0531152s)
2022-08-19T21:38:20.430+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e6: smithi124.pdzpkv(active, since 1.0637s)
2022-08-19T21:38:21.433+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e7: smithi124.pdzpkv(active, since 2s)
2022-08-19T21:38:38.154+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e8: smithi124.pdzpkv(active, since 18s)
2022-08-19T21:38:43.104+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e9: smithi124.pdzpkv(active, starting, since 0.0530632s)
2022-08-19T21:38:44.106+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e10: smithi124.pdzpkv(active, since 1.05519s)
2022-08-19T21:38:45.976+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e11: smithi124.pdzpkv(active, since 2s)
2022-08-19T21:38:50.884+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e12: smithi124.pdzpkv(active, since 7s)
2022-08-19T21:38:55.945+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e13: smithi124.pdzpkv(active, starting, since 0.053293s)
2022-08-19T21:38:56.952+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e14: smithi124.pdzpkv(active, since 1.06024s)
2022-08-19T21:38:58.852+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e15: smithi124.pdzpkv(active, since 2s)
2022-08-19T21:39:02.758+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e16: smithi124.pdzpkv(active, since 6s)
2022-08-19T21:40:00.612+0000 7f5bc2229700  0 log_channel(cluster) log [DBG] : mgrmap e16: smithi124.pdzpkv(active, since 64s)
2022-08-19T21:40:00.669+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e17: smithi124.pdzpkv(active, since 64s), standbys: smithi138.iazeci
2022-08-19T21:45:14.976+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e18: smithi124.pdzpkv(active, since 6m)
2022-08-19T21:45:23.103+0000 7f5bc0a26700  0 log_channel(cluster) log [DBG] : mgrmap e19: smithi124.pdzpkv(active, since 6m), standbys: smithi138.iazeci

mon log :

2022-08-19T21:45:22.600+0000 7f5bc2229700 20 mon.smithi124@0(leader).mgrstat health checks:
{
    "UPGRADE_NO_STANDBY_MGR": {
        "severity": "HEALTH_WARN",
        "summary": {
            "message": "Upgrade: Need standby mgr daemon",
            "count": 1
        },
        "detail": [
            {
                "message": "The upgrade process needs to upgrade the mgr, but it needs at least one standby to proceed." 
            }
        ]
    }
}

2022-08-19T21:45:22.600+0000 7f5bc2229700 20 mon.smithi124@0(leader).mgrstat progress events:
{
    "dc77e5f1-4e84-448e-9ca0-a8322772fa2b": {
        "message": "Upgrade to 16.2.10-668-g67571cd8 (33s)\n      [............................] ",
        "progress": 0,
        "add_to_ceph_s": true
    }
}
...
2022-08-19T21:45:22.987+0000 7f5bc2229700 10 mon.smithi124@0(leader).log v297  logging 2022-08-19T21:45:22.592032+0000 mgr.smithi124.pdzpkv (mgr.14162) 288 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi124.pdzpkv)
2022-08-19T21:45:22.987+0000 7f5bc2229700 10 mon.smithi124@0(leader).log v297  logging 2022-08-19T21:45:22.593440+0000 mgr.smithi124.pdzpkv (mgr.14162) 289 : cephadm [ERR] Upgrade: Paused due to UPGRADE_NO_STANDBY_MGR: Upgrade: Need standby mgr daemon

https://pulpito.ceph.com/yuriw-2022-08-19_20:57:42-rados-wip-yuri6-testing-2022-08-19-0940-pacific-distro-default-smithi/6981184

Related issues 1 (0 open — 1 closed)

Related to Orchestrator - Bug #54419: `ceph orch upgrade start` seems to never reach completionDuplicateAdam King

Actions
Actions
Copy link

Also available in: Atom PDF