Bug #39097: _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23 - bluestore - Ceph

Actions

#1

Updated by Sage Weil about 5 years ago

2019-04-03 16:32:14.584 7fb9406c2700  7 osd.1 pg_epoch: 1064 pg[9.17( v 1064'2543 (0'0,1064'2543] local-lis/les=1060/1061 n=1915 ec=1011/1000 lis/c 1060/1033 les/c/f 1061/1034/0 1059/1060/1055) [1,8,4]/[1,8] async=[4] r=0 lpr=1060 pi=[1033,1060)/1 rops=1 crt=1064'2543 lcod 1064'2542 mlcod 1021'1464 active+recoverin
g+undersized+remapped mbc={255={(2+1)=137,(2+2)=708}}] build_push_op 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head v 1021'1470 size 65536 recovery_info: ObjectRecoveryInfo(9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head@1021'1470, size: 65536, copy_subset: [0~65536], clone_subset: {}, sn
apset: 0=[]:{})
2019-04-03 16:32:14.588 7fb9406c2700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23, device location [0x1ead0000~10000], logical extent 0x0~10000, object #9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head#
2019-04-03 16:32:14.591 7fb9406c2700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23, device location [0x1ead0000~10000], logical extent 0x0~10000, object #9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head#
2019-04-03 16:32:14.593 7fb9406c2700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23, device location [0x1ead0000~10000], logical extent 0x0~10000, object #9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head#
2019-04-03 16:32:14.593 7fb9406c2700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23, device location [0x1ead0000~10000], logical extent 0x0~10000, object #9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head#
2019-04-03 16:32:14.593 7fb9406c2700 10 osd.1 pg_epoch: 1064 pg[9.17( v 1064'2543 (0'0,1064'2543] local-lis/les=1060/1061 n=1915 ec=1011/1000 lis/c 1060/1033 les/c/f 1061/1034/0 1059/1060/1055) [1,8,4]/[1,8] async=[4] r=0 lpr=1060 pi=[1033,1060)/1 rops=1 crt=1064'2543 lcod 1064'2542 mlcod 1021'1464 active+recovering+undersized+remapped mbc={255={(2+1)=137,(2+2)=708}}] start_pushes clean up peer 4
2019-04-03 16:32:14.593 7fb9406c2700  0 osd.1 pg_epoch: 1064 pg[9.17( v 1064'2543 (0'0,1064'2543] local-lis/les=1060/1061 n=1915 ec=1011/1000 lis/c 1060/1033 les/c/f 1061/1034/0 1059/1060/1055) [1,8,4]/[1,8] async=[4] r=0 lpr=1060 pi=[1033,1060)/1 rops=1 crt=1064'2543 lcod 1064'2542 mlcod 1021'1464 active+recovering+undersized+remapped mbc={255={(2+1)=137,(2+2)=708}}] prep_object_replica_pushes Error -5 on oid 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head

Actions

Copy link

#2

Updated by Sage Weil about 5 years ago

Status changed from 12 to Need More Info

no bluestore logs. object was written, and a minute later read with a crc error.

written here:

remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.116 7fb9406c2700 10 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1469 lc 0'0 (0'0,1021'1469] local-lis/les=0/0 n=855 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1469 active m=826 mbc={}] do_repop 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head v 1021'1470 (transaction) 182
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.117 7fb9406c2700 10 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1469 lc 0'0 (0'0,1021'1469] local-lis/les=0/0 n=856 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1469 active m=826 mbc={}] append_log log((0'0,1021'1469], crt=1021'1469) [1021'1470 (0'0) modify   9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head by client.9824.0:26625 2019-04-03 16:31:11.118200 0]
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.117 7fb9406c2700 10 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1470 lc 0'0 (0'0,1021'1470] local-lis/les=0/0 n=856 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1469 active m=826 mbc={}] add_log_entry 1021'1470 (0'0) modify   9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head by client.9824.0:26625 2019-04-03 16:31:11.118200 0
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.117 7fb9406c2700 20 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1470 lc 0'0 (0'0,1021'1470] local-lis/les=0/0 n=856 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1470 active m=826 mbc={}] rollforward: entry=1021'1470 (0'0) modify   9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head by client.9824.0:26625 2019-04-03 16:31:11.118200 0
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.117 7fb9406c2700 10 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1470 lc 0'0 (0'0,1021'1470] local-lis/les=0/0 n=856 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1470 active m=826 mbc={}] repop_commit on op osd_repop(client.9824.0:26625 9.17 e1021/1020 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head v 1021'1470) v2, sending commit to osd.2

Actions

Copy link

#3

Updated by Sage Weil about 5 years ago

Project changed from RADOS to bluestore
Subject changed from [ERR] 9.17 missing primary copy of 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head, will try copies on 2,8,11 to _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23