Project

General

Profile

Bug #23172

mds: fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades

Added by Dan van der Ster 11 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
High
Category:
-
Target version:
Start date:
02/28/2018
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
luminous
Regression:
Yes
Severity:
4 - irritation
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:

Description

commit ddba907279719631903e3a20543056d81d176a1b changed the id of MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 from 8 to 9. (this was to fix http://tracker.ceph.com/issues/21985)

When we update and restart any MDS (jewel or luminous before v12.2.3) to 12.2.3, then all MDS's not yet running the new version aware of feature bit 9 will suicide:


2018-02-28 10:25:22.761413 7f03da1b9700  0 mds.cephdwightmds0
handle_mds_map mdsmap compatset compat={},rocompat={},incompat={1=base
v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir
inode in separate object,5=mds uses versioned encoding,6=dirfrag is
stored in omap,8=no anchor table,9=file layout v2} not writeable with
daemon features compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,7=mds uses inline data,8=file layout v2}, killing myself
2018-02-28 10:25:22.761429 7f03da1b9700  1 mds.cephdwightmds0 suicide.
wanted state up:active
2018-02-28 10:25:23.763226 7f03da1b9700  1 mds.0.18147 shutdown:
shutting down rank 0

Related issues

Related to fs - Bug #21985: mds: definition of MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 is wrong Resolved 10/31/2017
Related to fs - Documentation #23568: doc: outline the steps for upgrading an MDS cluster Resolved 04/05/2018
Copied to fs - Backport #23414: luminous: mds: fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades Resolved

History

#1 Updated by Patrick Donnelly 11 months ago

  • Subject changed from Fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades to mds: fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades
  • Priority changed from Normal to High

I don't see an easy way to resolve this unfortunately other than to provide a release note on a safe way to upgrade. Copy paste from the mailing list discussion:

I think it will be necessary to reduce the actives to 1 (max_mds -> 1;
deactivate other ranks), shutdown standbys, upgrade the single active,
then upgrade/start the standbys.

Any other ideas?

#2 Updated by Nathan Cutler 10 months ago

  • Related to Bug #21985: mds: definition of MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 is wrong added

#3 Updated by Jan Fajerski 10 months ago

Would the following work:
It seems like the pre-12.2.4 MDS's suicide because the feature flag 8 changed values (file layout v2 -> no anchor table). Would be possible to change not the definition of file layout v2 but of no anchor table? This way feature 8 would not change but the prior hidden(?) feature would be added as 9?

#4 Updated by Jan Fajerski 10 months ago

Jan Fajerski wrote:

Would the following work:
It seems like the pre-12.2.4 MDS's suicide because the feature flag 8 changed values (file layout v2 -> no anchor table). Would be possible to change not the definition of file layout v2 but of no anchor table? This way feature 8 would not change but the prior hidden(?) feature would be added as 9?

Hm no. Just realised they are all marked INCOMPAT. Sorry for the noise.

#5 Updated by Patrick Donnelly 10 months ago

  • Status changed from New to Need Review
  • Assignee set to Patrick Donnelly
  • Target version set to v13.0.0
  • Backport set to luminous
  • Severity changed from 1 - critical to 4 - irritation
  • Affected Versions deleted (v12.2.4)

#6 Updated by Patrick Donnelly 10 months ago

  • Status changed from Need Review to Pending Backport

#7 Updated by Nathan Cutler 10 months ago

  • Copied to Backport #23414: luminous: mds: fixed MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition breaks luminous upgrades added

#8 Updated by Nathan Cutler 10 months ago

  • Status changed from Pending Backport to Resolved

#9 Updated by Patrick Donnelly 10 months ago

Also available in: Atom PDF