Project

General

Profile

Actions
Copy link

Bug #50114

closed

cephadm: upgrade loop on target_digests list mismatch

Added by Sage Weil about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Sage Weil
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
40577
Crash signature (v1):
Crash signature (v2):

Description

I tried to update Ceph 15.2.10 to 16.2.0 via ceph orch. In the
beginning everything seems to work fine and the new MGR and MONs where
deployed. But now I enden up in a pulling loop and I am unable to fix
the issue by my self.

#ceph -W cephadm --watch-debu

2021-04-02T10:36:20.704960+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Need to upgrade myself (mgr.mon-a-02.tvcrfq)
2021-04-02T10:36:21.837596+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-01
2021-04-02T10:36:24.591487+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-01 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42f ecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T10:36:37.054786+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Need to upgrade myself (mgr.mon-a-02.tvcrfq)
2021-04-02T10:36:38.419014+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-01
2021-04-02T10:36:41.172835+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-01 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000','docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42f
ecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting

After I stoppend the update I got the following health error:

Module 'cephadm' has failed: 'NoneType' object has no attribute
'target_digests'

Actions
Copy link
 #1

Updated by Sage Weil about 3 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Sage Weil about 3 years ago

it is alternating between two digest variants, one with docker.io/ prefix, and one without:

2021-04-02T21:00:20.346+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking crash daemons
2021-04-02T21:00:20.983+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: Pulling docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on smb-ctdb-a-01
2021-04-02T21:00:20.983+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: Pulling docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on smb-ctdb-a-01
2021-04-02T21:00:22.888+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: image docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on smb-ctdb-a-01 got new digests ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000'] (not ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T21:00:22.888+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: image docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on smb-ctdb-a-01 got new digests ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000'] (not ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T21:00:23.613+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mgr daemons
2021-04-02T21:00:23.613+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade myself (mgr.mon-a-01.wcwbur)
2021-04-02T21:00:23.613+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade myself (mgr.mon-a-01.wcwbur)
2021-04-02T21:00:24.754+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-02
2021-04-02T21:00:24.754+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-02
2021-04-02T21:00:27.628+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-02 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T21:00:27.628+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-02 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T21:00:28.343+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mgr daemons
2021-04-02T21:00:28.348+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: All mgr daemons are up to date.
2021-04-02T21:00:28.348+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mon daemons
2021-04-02T21:00:28.352+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: All mon daemons are up to date.
2021-04-02T21:00:28.352+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking crash daemons
2021-04-02T21:00:28.967+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: Pulling docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on smb-ctdb-a-01
2021-04-02T21:00:28.967+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: Pulling docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on smb-ctdb-a-01
2021-04-02T21:00:30.728+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: image docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on smb-ctdb-a-01 got new digests ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000'] (not ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T21:00:30.728+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: image docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on smb-ctdb-a-01 got new digests ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000'] (not ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T21:00:31.449+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mgr daemons
2021-04-02T21:00:31.449+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade myself (mgr.mon-a-01.wcwbur)
2021-04-02T21:00:31.449+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade myself (mgr.mon-a-01.wcwbur)
2021-04-02T21:00:32.576+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-02
2021-04-02T21:00:32.576+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-02
2021-04-02T21:00:35.375+0000 7fb64da11700  0 [cephadm INFO cephadm.upgrade] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-02 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T21:00:35.375+0000 7fb64da11700  0 log_channel(cephadm) log [INF] : Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-02 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T21:00:36.409+0000 7fb64da11700  0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mgr daemons

Actions
Copy link
 #3

Updated by Sage Weil about 3 years ago

  • Subject changed from cephadm: upgarde loop on target_digests list mismatch to cephadm: upgrade loop on target_digests list mismatch
  • Status changed from New to In Progress
  • Assignee set to Sage Weil
Actions
Copy link
 #4

Updated by Sage Weil about 3 years ago

  • Status changed from In Progress to Fix Under Review
  • Backport set to pacific
  • Pull request ID set to 40577
Actions
Copy link
 #5

Updated by Sebastian Wagner about 3 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #6

Updated by Sage Weil about 3 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF