Actions
Bug #39097
closed_verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23
Status:
Won't Fix
Priority:
High
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2019-04-03 16:32:14.608 7fb9406c2700 10 osd.1 pg_epoch: 1064 pg[9.17( v 1064'2543 (0'0,1064'2543] local-lis/les=1060/1061 n=1915 ec=1011/1000 lis/c 1060/1033 les/c/f 1061/1034/0 1059/1060/1055) [1,8,4]/[1,8] async=[4] r=0 lpr=1060 pi=[1033,1060)/1 crt=1064'2543 lcod 1064'2542 mlcod 1021'1464 active+recovering+undersized+remapped m=1 mbc={255={(1+2)=1,(2+1)=137,(2+2)=707}}] 9.17 unexpectedly missing 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head v1021'1470, there should be a copy on shard 8
2019-04-03 16:32:14.609 7fb9406c2700 -1 log_channel(cluster) log [ERR] : 9.17 missing primary copy of 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head, will try copies on 2,8,11
2019-04-03 16:32:14.609 7fb9406c2700 10 osd.1 pg_epoch: 1064 pg[9.17( v 1064'2543 (0'0,1064'2543] local-lis/les=1060/1061 n=1915 ec=1011/1000 lis/c 1060/1033 les/c/f 1061/1034/0 1059/1060/1055) [1,8,4]/[1,8] async=[4] r=0 lpr=1060 pi=[1033,1060)/1 crt=1064'2543 lcod 1064'2542 mlcod 1021'1464 active+recovering+undersized+remapped m=1 mbc={255={(1+2)=1,(2+1)=137,(2+2)=707}}] recover_replicas: recover_object_replicas(9:ece43a4f:::benchmark_data_smithi018_32371_object26656:head)
/a/sage-2019-04-03_02:18:56-rados-wip-sage2-testing-2019-04-02-1625-distro-basic-smithi/3803320
Updated by Sage Weil about 5 years ago
2019-04-03 16:32:14.584 7fb9406c2700 7 osd.1 pg_epoch: 1064 pg[9.17( v 1064'2543 (0'0,1064'2543] local-lis/les=1060/1061 n=1915 ec=1011/1000 lis/c 1060/1033 les/c/f 1061/1034/0 1059/1060/1055) [1,8,4]/[1,8] async=[4] r=0 lpr=1060 pi=[1033,1060)/1 rops=1 crt=1064'2543 lcod 1064'2542 mlcod 1021'1464 active+recoverin
g+undersized+remapped mbc={255={(2+1)=137,(2+2)=708}}] build_push_op 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head v 1021'1470 size 65536 recovery_info: ObjectRecoveryInfo(9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head@1021'1470, size: 65536, copy_subset: [0~65536], clone_subset: {}, sn
apset: 0=[]:{})
2019-04-03 16:32:14.588 7fb9406c2700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23, device location [0x1ead0000~10000], logical extent 0x0~10000, object #9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head#
2019-04-03 16:32:14.591 7fb9406c2700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23, device location [0x1ead0000~10000], logical extent 0x0~10000, object #9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head#
2019-04-03 16:32:14.593 7fb9406c2700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23, device location [0x1ead0000~10000], logical extent 0x0~10000, object #9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head#
2019-04-03 16:32:14.593 7fb9406c2700 -1 bluestore(/var/lib/ceph/osd/ceph-1) _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23, device location [0x1ead0000~10000], logical extent 0x0~10000, object #9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head#
2019-04-03 16:32:14.593 7fb9406c2700 10 osd.1 pg_epoch: 1064 pg[9.17( v 1064'2543 (0'0,1064'2543] local-lis/les=1060/1061 n=1915 ec=1011/1000 lis/c 1060/1033 les/c/f 1061/1034/0 1059/1060/1055) [1,8,4]/[1,8] async=[4] r=0 lpr=1060 pi=[1033,1060)/1 rops=1 crt=1064'2543 lcod 1064'2542 mlcod 1021'1464 active+recovering+undersized+remapped mbc={255={(2+1)=137,(2+2)=708}}] start_pushes clean up peer 4
2019-04-03 16:32:14.593 7fb9406c2700 0 osd.1 pg_epoch: 1064 pg[9.17( v 1064'2543 (0'0,1064'2543] local-lis/les=1060/1061 n=1915 ec=1011/1000 lis/c 1060/1033 les/c/f 1061/1034/0 1059/1060/1055) [1,8,4]/[1,8] async=[4] r=0 lpr=1060 pi=[1033,1060)/1 rops=1 crt=1064'2543 lcod 1064'2542 mlcod 1021'1464 active+recovering+undersized+remapped mbc={255={(2+1)=137,(2+2)=708}}] prep_object_replica_pushes Error -5 on oid 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head
Updated by Sage Weil about 5 years ago
- Status changed from 12 to Need More Info
no bluestore logs. object was written, and a minute later read with a crc error.
written here:
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.116 7fb9406c2700 10 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1469 lc 0'0 (0'0,1021'1469] local-lis/les=0/0 n=855 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1469 active m=826 mbc={}] do_repop 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head v 1021'1470 (transaction) 182
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.117 7fb9406c2700 10 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1469 lc 0'0 (0'0,1021'1469] local-lis/les=0/0 n=856 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1469 active m=826 mbc={}] append_log log((0'0,1021'1469], crt=1021'1469) [1021'1470 (0'0) modify 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head by client.9824.0:26625 2019-04-03 16:31:11.118200 0]
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.117 7fb9406c2700 10 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1470 lc 0'0 (0'0,1021'1470] local-lis/les=0/0 n=856 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1469 active m=826 mbc={}] add_log_entry 1021'1470 (0'0) modify 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head by client.9824.0:26625 2019-04-03 16:31:11.118200 0
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.117 7fb9406c2700 20 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1470 lc 0'0 (0'0,1021'1470] local-lis/les=0/0 n=856 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1470 active m=826 mbc={}] rollforward: entry=1021'1470 (0'0) modify 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head by client.9824.0:26625 2019-04-03 16:31:11.118200 0
remote/smithi073/log/ceph-osd.1.log:2019-04-03 16:31:11.117 7fb9406c2700 10 osd.1 pg_epoch: 1021 pg[9.17( v 1021'1470 lc 0'0 (0'0,1021'1470] local-lis/les=0/0 n=856 ec=1011/1000 lis/c 0/1011 les/c/f 0/1012/0 1019/1020/1013) [2,11,1]/[2,11] r=-1 lpr=1020 pi=[1011,1020)/2 luod=0'0 lua=1018'1457 crt=1021'1470 active m=826 mbc={}] repop_commit on op osd_repop(client.9824.0:26625 9.17 e1021/1020 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head v 1021'1470) v2, sending commit to osd.2
Updated by Sage Weil about 5 years ago
- Project changed from RADOS to bluestore
- Subject changed from [ERR] 9.17 missing primary copy of 9:e9fb9d5c:::benchmark_data_smithi018_32371_object26624:head, will try copies on 2,8,11 to _verify_csum bad crc32c/0x10000 checksum at blob offset 0x0, got 0x478682d5, expected 0x28f49e23
Updated by yang wang almost 5 years ago
hi， sage， i got the same err, http://tracker.ceph.com/issues/40459 in ceph 12.2.5 and centos7.4, any idea to solve this?
Updated by Neha Ojha over 3 years ago
- Status changed from Need More Info to Won't Fix
This just seems to be a checksum error, which is expected in some cases.
Actions