Bug #23172: mds: fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades - CephFS - Ceph

Actions

Copy link

Bug #23172

closed

mds: fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades

Added by Dan van der Ster about 6 years ago. Updated about 6 years ago.

Status:

Resolved

Priority:

High

Assignee:

Patrick Donnelly

Category:

Target version:

Ceph - v13.0.0

% Done:

Source:

Community (user)

Tags:

Backport:

luminous

Regression:

Yes

Severity:

4 - irritation

Reviewed:

Affected Versions:

Ceph - v12.2.3

ceph-qa-suite:

Component(FS):

MDS

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

commit ddba907279719631903e3a20543056d81d176a1b changed the id of MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 from 8 to 9. (this was to fix http://tracker.ceph.com/issues/21985)

When we update and restart any MDS (jewel or luminous before v12.2.3) to 12.2.3, then all MDS's not yet running the new version aware of feature bit 9 will suicide:


2018-02-28 10:25:22.761413 7f03da1b9700  0 mds.cephdwightmds0
handle_mds_map mdsmap compatset compat={},rocompat={},incompat={1=base
v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir
inode in separate object,5=mds uses versioned encoding,6=dirfrag is
stored in omap,8=no anchor table,9=file layout v2} not writeable with
daemon features compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,7=mds uses inline data,8=file layout v2}, killing myself
2018-02-28 10:25:22.761429 7f03da1b9700  1 mds.cephdwightmds0 suicide.
wanted state up:active
2018-02-28 10:25:23.763226 7f03da1b9700  1 mds.0.18147 shutdown:
shutting down rank 0

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Patrick Donnelly about 6 years ago

Subject changed from Fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades to mds: fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades
Priority changed from Normal to High

I don't see an easy way to resolve this unfortunately other than to provide a release note on a safe way to upgrade. Copy paste from the mailing list discussion:

I think it will be necessary to reduce the actives to 1 (max_mds -> 1;
deactivate other ranks), shutdown standbys, upgrade the single active,
then upgrade/start the standbys.

Any other ideas?

Actions

Copy link

Updated by Nathan Cutler about 6 years ago

Related to Bug #21985: mds: definition of MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 is wrong added

Actions

Copy link

Updated by Jan Fajerski about 6 years ago

Would the following work:
It seems like the pre-12.2.4 MDS's suicide because the feature flag 8 changed values (file layout v2 -> no anchor table). Would be possible to change not the definition of file layout v2 but of no anchor table? This way feature 8 would not change but the prior hidden(?) feature would be added as 9?

Actions

Copy link

Updated by Jan Fajerski about 6 years ago

Jan Fajerski wrote:

Would the following work:
It seems like the pre-12.2.4 MDS's suicide because the feature flag 8 changed values (file layout v2 -> no anchor table). Would be possible to change not the definition of file layout v2 but of no anchor table? This way feature 8 would not change but the prior hidden(?) feature would be added as 9?

Hm no. Just realised they are all marked INCOMPAT. Sorry for the noise.

Actions

Copy link