Bug #16113: unfound objects after setting sortbitwise on a cluster after jewel upgrade - Ceph - Ceph

Actions

Copy link

Bug #16113

closed

unfound objects after setting sortbitwise on a cluster after jewel upgrade

Added by Samuel Just almost 8 years ago. Updated almost 8 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Samuel Just

Category:

Target version:

% Done:

Source:

other

Tags:

Backport:

jewel

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Bug seen on sepia long running cluster, and some reports on list

2016-06-01 14:33:07.665212 7f305bf01700 10 osd.123 pg_epoch: 724336 pg[1.2e( v 724309'6620010 (724089'6616950,724309'6620010] local-les=724336 n=23575 ec=1 les/c/f 724299/724024/652051 724318/724326/724007) [123,100,20,12] r=0 lpr=724326 pi=724007-724325/7 crt=724308'6620007 lcod 724308'6620007 mlcod 0'0 inactive] search_for_missing 1:7442c937:::100177fc6c5.00000000:head 724309'6620009 also missing on osd.123 (last_backfill MAX but with wrong sort order)
2016-06-01 14:33:07.665322 7f3063710700 15 osd.123 724336 project_pg_history 4.fd0 from 724327 to 724336, start ec=322215 les/c/f 724281/724281/652051 724318/724327/720451
2016-06-01 14:33:07.665321 7f305bf01700 10 osd.123 pg_epoch: 724336 pg[1.2e( v 724309'6620010 (724089'6616950,724309'6620010] local-les=724336 n=23575 ec=1 les/c/f 724299/724024/652051 724318/724326/724007) [123,100,20,12] r=0 lpr=724326 pi=724007-724325/7 crt=724308'6620007 lcod 724308'6620007 mlcod 0'0 inactive] search_for_missing 1:77a52b15:::100177fc398.00000000:head 724309'6620010 also missing on osd.123 (last_backfill MAX but with wrong sort order)

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Samuel Just almost 8 years ago

Backport set to jewel

Actions

Copy link

Updated by Samuel Just almost 8 years ago

Status changed from New to 7

https://github.com/ceph/ceph/tree/wip-16113

Actions

Copy link

Updated by Samuel Just almost 8 years ago

This is what a max last_backfill value should be:

"last_backfill": {
        "oid": "",
        "key": "",
        "snapid": 0,
        "hash": 0,
        "max": 1,
        "pool": -9223372036854775808,
        "namespace": "" 
    },

1.2e on the LRC had a value of:

"last_backfill": {
        "oid": "",
        "key": "",
        "snapid": 0,
        "hash": 0,
        "max": 1,
        "pool": -1,
        "namespace": "" 
    },

I think the issue is that hammer creates max() objects with the old min pool value instead of the new one. The above branch should fix it, but I'm surprised this hasn't come up in testing. I'll look into that tomorrow.

Actions

Copy link