Project

General

Profile

Bug #16113

unfound objects after setting sortbitwise on a cluster after jewel upgrade

Added by Samuel Just over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
06/01/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Bug seen on sepia long running cluster, and some reports on list

2016-06-01 14:33:07.665212 7f305bf01700 10 osd.123 pg_epoch: 724336 pg[1.2e( v 724309'6620010 (724089'6616950,724309'6620010] local-les=724336 n=23575 ec=1 les/c/f 724299/724024/652051 724318/724326/724007) [123,100,20,12] r=0 lpr=724326 pi=724007-724325/7 crt=724308'6620007 lcod 724308'6620007 mlcod 0'0 inactive] search_for_missing 1:7442c937:::100177fc6c5.00000000:head 724309'6620009 also missing on osd.123 (last_backfill MAX but with wrong sort order)
2016-06-01 14:33:07.665322 7f3063710700 15 osd.123 724336 project_pg_history 4.fd0 from 724327 to 724336, start ec=322215 les/c/f 724281/724281/652051 724318/724327/720451
2016-06-01 14:33:07.665321 7f305bf01700 10 osd.123 pg_epoch: 724336 pg[1.2e( v 724309'6620010 (724089'6616950,724309'6620010] local-les=724336 n=23575 ec=1 les/c/f 724299/724024/652051 724318/724326/724007) [123,100,20,12] r=0 lpr=724326 pi=724007-724325/7 crt=724308'6620007 lcod 724308'6620007 mlcod 0'0 inactive] search_for_missing 1:77a52b15:::100177fc398.00000000:head 724309'6620010 also missing on osd.123 (last_backfill MAX but with wrong sort order)


Related issues

Copied to Ceph - Backport #16282: jewel: unfound objects after setting sortbitwise on a cluster after jewel upgrade Resolved

History

#1 Updated by Samuel Just over 3 years ago

  • Backport set to jewel

#2 Updated by Samuel Just over 3 years ago

  • Status changed from New to Testing

#3 Updated by Samuel Just over 3 years ago

This is what a max last_backfill value should be:

"last_backfill": {
"oid": "",
"key": "",
"snapid": 0,
"hash": 0,
"max": 1,
"pool": -9223372036854775808,
"namespace": ""
},

1.2e on the LRC had a value of:

"last_backfill": {
"oid": "",
"key": "",
"snapid": 0,
"hash": 0,
"max": 1,
"pool": -1,
"namespace": ""
},

I think the issue is that hammer creates max() objects with the old min pool value instead of the new one. The above branch should fix it, but I'm surprised this hasn't come up in testing. I'll look into that tomorrow.

#5 Updated by Nathan Cutler over 3 years ago

  • Status changed from Testing to Pending Backport

#6 Updated by Nathan Cutler over 3 years ago

  • Copied to Backport #16282: jewel: unfound objects after setting sortbitwise on a cluster after jewel upgrade added

#7 Updated by Loic Dachary about 3 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF