Ceph

At filestore debug level 10 this message "get_hash_info: Mismatch of total_chunk_size 466944" should have been generated in a recovery read attempt at osd.187 and/or osd.167.

The primary should have had a message like this for an attempt to read osd.187: "_failed_push: Read error 143:xxxxxxxx:::default.63296332.1__shadow_304299676.2~3Mnd94mMNaUb0tK0fEB5bZn8YrB-A-F.1_2:head r=0 errors={187(8)=-5}

I truncated shard 1 on on osd 1 as a test:

$ find dev -name 'bar*' -ls
   409702  19420 -rw-r--r--   1 dzafman  dzafman  19881984 Aug 29 13:45 dev/osd3/current/2.0s0_head/bar__head_EFE6384B__2_ffffffffffffffff_0
   409703  19420 -rw-r--r--   1 dzafman  dzafman  19881984 Aug 29 13:45 dev/osd2/current/2.0s2_head/bar__head_EFE6384B__2_ffffffffffffffff_2
   409704   5468 -rw-r--r--   1 dzafman  dzafman   5595136 Aug 29 13:48 dev/osd1/current/2.0s1_head/bar__head_EFE6384B__2_ffffffffffffffff_1
$ bin/ceph-objectstore-tool --data-path dev/osd1 '["2.0s1",{"oid":"bar","key":"","snapid":-2,"hash":4024842315,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' get-attr hinfo_key | bin/ceph-dencoder import - type ECUtil::HashInfo decode dump_json
{
    "total_chunk_size": 19881984,
    "cumulative_shard_hashes": {
        "hash": {
            "shard": 0,
            "hash": 3279216067
        },
        "hash": {
            "shard": 1,
            "hash": 1616075201
        },
        "hash": {
            "shard": 2,
            "hash": 3966965908
        }
    }
}

[root@ceph017 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-187/ '["143.371s8",{"oid":"default.63296332.1__shadow_304299676.2~3Mnd94mMNaUb0tK0fEB5bZn8YrB-A-F.1_2","key":"","snapid":-2,"hash":764501873,"max":0,"pool":143,"namespace":"","shard_id":8,"max":0}]' get-attr hinfo_key | ceph-dencoder import - type ECUtil::HashInfo decode dump_json {
"total_chunk_size": 58080,
"cumulative_shard_hashes": {
"hash": {
"shard": 0,
"hash": 1528934879
},
"hash": {
"shard": 1,
"hash": 4161392522
},
"hash": {
"shard": 2,
"hash": 3354715535
},
"hash": {
"shard": 3,
"hash": 1867130716
},
"hash": {
"shard": 4,
"hash": 2427655492
},
"hash": {
"shard": 5,
"hash": 4243843471
},
"hash": {
"shard": 6,
"hash": 3460445679
},
"hash": {
"shard": 7,
"hash": 3397239655
},
"hash": {
"shard": 8,
"hash": 680106050
},
"hash": {
"shard": 9,
"hash": 1264594567
},
"hash": {
"shard": 10,
"hash": 2253242928
},
"hash": {
"shard": 11,
"hash": 1663251712
}
}
}

And this from a good chunk:

[root@ceph014 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-132/ '["143.371s0",{"oid":"default.63296332.1__shadow_304299676.2~3Mnd94mMNaUb0tK0fEB5bZn8YrB-A-F.1_2","key":"","snapid":-2,"hash":764501873,"max":0,"pool":143,"namespace":"","shard_id":0,"max":0}]' get-attr hinfo_key | ceph-dencoder import - type ECUtil::HashInfo decode dump_json {
"total_chunk_size": 466080,
"cumulative_shard_hashes": {
"hash": {
"shard": 0,
"hash": 2716396157
},
"hash": {
"shard": 1,
"hash": 3823712376
},
"hash": {
"shard": 2,
"hash": 643392995
},
"hash": {
"shard": 3,
"hash": 3096269466
},
"hash": {
"shard": 4,
"hash": 3697605281
},
"hash": {
"shard": 5,
"hash": 2035529382
},
"hash": {
"shard": 6,
"hash": 102109390
},
"hash": {
"shard": 7,
"hash": 2587959089
},
"hash": {
"shard": 8,
"hash": 1273445979
},
"hash": {
"shard": 9,
"hash": 2924648671
},
"hash": {
"shard": 10,
"hash": 1959366179
},
"hash": {
"shard": 11,
"hash": 917397527
}
}
}

One example had mod time Aug 5 for the smaller shards and Aug 7 for the larger.

mustafam: ...upgraded to Kraken 11.2.0 about that time, then 11.2.1, then 12.1.4 RC all while the cluster is almost dead (we hit another bug about osdmap)

Not sure what "about that time" was referring to.

David Zafman wrote:

One example had mod time Aug 5 for the smaller shards and Aug 7 for the larger.

mustafam: ...upgraded to Kraken 11.2.0 about that time, then 11.2.1, then 12.1.4 RC all while the cluster is almost dead (we hit another bug about osdmap)

Not sure what "about that time" was referring to.

OK, I meant around Aug 7, but now I think I found the upgrade timing in history, we upgraded on Aug 8, nothing was on Aug 5 and Aug 7.