Bug #50114
closed
cephadm: upgrade loop on target_digests list mismatch
Added by Sage Weil about 3 years ago.
Updated about 3 years ago.
Description
I tried to update Ceph 15.2.10 to 16.2.0 via ceph orch. In the
beginning everything seems to work fine and the new MGR and MONs where
deployed. But now I enden up in a pulling loop and I am unable to fix
the issue by my self.
#ceph -W cephadm --watch-debu
2021-04-02T10:36:20.704960+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Need to upgrade myself (mgr.mon-a-02.tvcrfq)
2021-04-02T10:36:21.837596+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-01
2021-04-02T10:36:24.591487+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-01 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42f ecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T10:36:37.054786+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Need to upgrade myself (mgr.mon-a-02.tvcrfq)
2021-04-02T10:36:38.419014+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-01
2021-04-02T10:36:41.172835+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-01 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000','docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42f
ecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
After I stoppend the update I got the following health error:
Module 'cephadm' has failed: 'NoneType' object has no attribute
'target_digests'
- Description updated (diff)
it is alternating between two digest variants, one with docker.io/ prefix, and one without:
2021-04-02T21:00:20.346+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking crash daemons
2021-04-02T21:00:20.983+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: Pulling docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on smb-ctdb-a-01
2021-04-02T21:00:20.983+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: Pulling docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on smb-ctdb-a-01
2021-04-02T21:00:22.888+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: image docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on smb-ctdb-a-01 got new digests ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000'] (not ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T21:00:22.888+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: image docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on smb-ctdb-a-01 got new digests ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000'] (not ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T21:00:23.613+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mgr daemons
2021-04-02T21:00:23.613+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade myself (mgr.mon-a-01.wcwbur)
2021-04-02T21:00:23.613+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade myself (mgr.mon-a-01.wcwbur)
2021-04-02T21:00:24.754+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-02
2021-04-02T21:00:24.754+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-02
2021-04-02T21:00:27.628+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-02 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T21:00:27.628+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-02 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T21:00:28.343+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mgr daemons
2021-04-02T21:00:28.348+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: All mgr daemons are up to date.
2021-04-02T21:00:28.348+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mon daemons
2021-04-02T21:00:28.352+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: All mon daemons are up to date.
2021-04-02T21:00:28.352+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking crash daemons
2021-04-02T21:00:28.967+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: Pulling docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on smb-ctdb-a-01
2021-04-02T21:00:28.967+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: Pulling docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on smb-ctdb-a-01
2021-04-02T21:00:30.728+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: image docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on smb-ctdb-a-01 got new digests ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000'] (not ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T21:00:30.728+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: image docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on smb-ctdb-a-01 got new digests ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000'] (not ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T21:00:31.449+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mgr daemons
2021-04-02T21:00:31.449+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: Need to upgrade myself (mgr.mon-a-01.wcwbur)
2021-04-02T21:00:31.449+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: Need to upgrade myself (mgr.mon-a-01.wcwbur)
2021-04-02T21:00:32.576+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-02
2021-04-02T21:00:32.576+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 on mon-a-02
2021-04-02T21:00:35.375+0000 7fb64da11700 0 [cephadm INFO cephadm.upgrade] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-02 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T21:00:35.375+0000 7fb64da11700 0 log_channel(cephadm) log [INF] : Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 pull on mon-a-02 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000']), restarting
2021-04-02T21:00:36.409+0000 7fb64da11700 0 [cephadm DEBUG cephadm.upgrade] Upgrade: Checking mgr daemons
- Subject changed from cephadm: upgarde loop on target_digests list mismatch to cephadm: upgrade loop on target_digests list mismatch
- Status changed from New to In Progress
- Assignee set to Sage Weil
- Status changed from In Progress to Fix Under Review
- Backport set to pacific
- Pull request ID set to 40577
- Status changed from Fix Under Review to Pending Backport
- Status changed from Pending Backport to Resolved
Also available in: Atom
PDF