Actions
Bug #40522
closedon_local_recover doesn't touch?
Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2019-06-22T11:45:01.779+0000 7f87a234e700 10 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=4}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] on_local_recover: 2:b022980f:::smithi04715146-47:head
2019-06-22T11:45:01.779+0000 7f87a234e700 10 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=4}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] got missing 2:b022980f:::smithi04715146-47:head v 1351'676
2019-06-22T11:45:01.779+0000 7f87a234e700 10 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=4}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] last_complete now 1350'674 log.complete_to 1351'675
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats actingset 5,7 upset 5,7 acting_recovery_backfill 5,7
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats shard 5 primary objects 4 missing 3
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats shard 1 objects 4 missing 0
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats shard 7 objects 4 missing 0
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats object_location_counts {1,5,7=1,1,7=3}
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats ml 3 upset size 2 up 1
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats ml 1 upset size 2 up 2
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats missing based degraded 0
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats missing based misplaced 3
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats degraded 0
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] update_calc_stats misplaced 3
2019-06-22T11:45:01.779+0000 7f87a234e700 20 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] prepare_stats_for_publish reporting purged_snaps [322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]
2019-06-22T11:45:01.779+0000 7f87a234e700 15 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] publish_stats_to_osd 1355:4108
2019-06-22T11:45:01.779+0000 7f87a234e700 10 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] release_backoffs [2:b022980f:::smithi04715146-47:head,2:b022980f:::smithi04715146-47:head)
2019-06-22T11:45:01.779+0000 7f87a234e700 10 osd.5 pg_epoch: 1355 pg[2.d( v 1351'678 lc 1350'674 (997'542,1351'678] local-lis/les=1354/1355 n=4 ec=917/21 lis/c 1354/1349 les/c/f 1355/1350/0 1353/1354/1354) [5,7] r=0 lpr=1354 pi=[1166,1354)/2 crt=1351'678 mlcod 0'0 active+recovering rops=1 m=1 mbc={255={(1+1)=3,(2+1)=1}} ps=[322~2,327~2,32b~1,32d~1,32f~2,332~1,334~1,336~2]] readable_with_acting: locs:1,5,7
...
-3556> 2019-06-22T11:45:01.779+0000 7f87a234e700 10 bluestore(/var/lib/ceph/osd/ceph-5) queue_transactions ch 0xd359200 2.d_head
-3552> 2019-06-22T11:45:01.779+0000 7f87a234e700 20 bluestore(/var/lib/ceph/osd/ceph-5) _txc_create osr 0xd36eb40 = 0x10e6d000 seq 12
-3544> 2019-06-22T11:45:01.779+0000 7f87a234e700 10 bluestore(/var/lib/ceph/osd/ceph-5) _txc_add_transaction op 28 got ENOENT on #2:b022980f:::smithi04715146-47:head#
-3541> 2019-06-22T11:45:01.779+0000 7f87a234e700 10 bluestore(/var/lib/ceph/osd/ceph-5) _txc_add_transaction op 12 got ENOENT on #2:b022980f:::smithi04715146-47:head#
-3538> 2019-06-22T11:45:01.779+0000 7f87a234e700 10 bluestore(/var/lib/ceph/osd/ceph-5) _txc_add_transaction op 15 got ENOENT on #2:b022980f:::smithi04715146-47:head#
-3528> 2019-06-22T11:45:01.779+0000 7f87a234e700 -1 bluestore(/var/lib/ceph/osd/ceph-5) _txc_add_transaction error (2) No such file or directory not handled on operation 15 (op 2, counting from 0)
-3527> 2019-06-22T11:45:01.779+0000 7f87a234e700 -1 bluestore(/var/lib/ceph/osd/ceph-5) unexpected error code
-3462> 2019-06-22T11:45:01.779+0000 7f87a234e700 0 _dump_transaction transaction dump:
{
"ops": [
{
"op_num": 0,
"op_name": "rmattrs",
"collection": "2.d_head",
"oid": "#2:b022980f:::smithi04715146-47:head#"
},
{
"op_num": 1,
"op_name": "truncate",
"collection": "2.d_head",
"oid": "#2:b022980f:::smithi04715146-47:head#",
"offset": 2672643
},
{
"op_num": 2,
"op_name": "setattrs",
"collection": "2.d_head",
"oid": "#2:b022980f:::smithi04715146-47:head#",
"attr_lens": {
"_": 272,
"__header": 58,
"snapset": 131
}
},
{
"op_num": 3,
"op_name": "omap_rmkeys",
"collection": "2.d_head",
"oid": "#2:b0000000::::head#",
"attrs": [
"missing/0000000000000002.D044910F.head.smithi04715146-47.."
]
},
{
"op_num": 4,
"op_name": "omap_setkeys",
"collection": "2.d_head",
"oid": "#2:b0000000::::head#",
"attr_lens": {
"_info": 992
}
}
]
}
/a/sage-2019-06-21_16:36:51-rados:thrash-wip-sage2-testing-2019-06-20-2154-distro-basic-smithi/4057772
Actions