Project

General

Profile

Actions
Copy link

Bug #36331

closed

FAILED ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 (zeros)

Added by Neha Ojha over 5 years ago. Updated almost 5 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2018-10-05T23:15:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:available_objects: 1003 in_flight_objects: 0 total objects: 1003 in_flight 0
2018-10-05T23:16:32.345 INFO:teuthology.orchestra.run.smithi203.stdout:==> rm -r bluestore.test_temp_dir
2018-10-05T23:16:32.434 INFO:teuthology.orchestra.run.smithi203.stdout:[  FAILED  ] ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2, where GetParam() = "bluestore" (2601632 ms)

Seen in mimic: /a/yuriw-2018-10-05_14:14:11-rados-wip-yuri3-testing-2018-10-04-2213-mimic-distro-basic-smithi/3104939/

Actions
Copy link
 #1

Updated by Sage Weil over 5 years ago

  • Status changed from New to 12
2018-10-05T23:06:21.008 INFO:teuthology.orchestra.run.smithi203.stdout:--- buffer mismatch between offset 0x2decc and 0x370cb, total 0x65ecc
2018-10-05T23:06:21.014 INFO:teuthology.orchestra.run.smithi203.stdout:--- expected:
2018-10-05T23:06:21.014 INFO:teuthology.orchestra.run.smithi203.stdout:00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
2018-10-05T23:06:21.014 INFO:teuthology.orchestra.run.smithi203.stdout:*
2018-10-05T23:06:21.014 INFO:teuthology.orchestra.run.smithi203.stdout:0002dcc0  00 00 00 00 00 00 00 00  00 00 00 00 38 37 34 34  |............8744|
2018-10-05T23:06:21.014 INFO:teuthology.orchestra.run.smithi203.stdout:0002dcd0  39 37 32 39 33 36 32 33  39 33 39 39 37 32 35 37  |9729362393997257|
...
2018-10-05T23:06:21.293 INFO:teuthology.orchestra.run.smithi203.stdout:000370b0  34 34 37 38 33 39 37 37  33 32 35 30 38 31 39 30  |4478397732508190|
2018-10-05T23:06:21.293 INFO:teuthology.orchestra.run.smithi203.stdout:000370c0  32 31 35 35 36 32 33 34  34 34 39 00 00 00 00 00  |21556234449.....|
2018-10-05T23:06:21.293 INFO:teuthology.orchestra.run.smithi203.stdout:000370d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
2018-10-05T23:06:21.293 INFO:teuthology.orchestra.run.smithi203.stdout:*
2018-10-05T23:06:21.293 INFO:teuthology.orchestra.run.smithi203.stdout:00065ec0  00 00 00 00 00 00 00 00  00 00 00 00              |............|
2018-10-05T23:06:21.293 INFO:teuthology.orchestra.run.smithi203.stdout:00065ecc
2018-10-05T23:06:21.293 INFO:teuthology.orchestra.run.smithi203.stdout:--- actual:
2018-10-05T23:06:21.293 INFO:teuthology.orchestra.run.smithi203.stdout:00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:*
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dcc0  00 00 00 00 00 00 00 00  00 00 00 00 38 37 34 34  |............8744|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dcd0  39 37 32 39 33 36 32 33  39 33 39 39 37 32 35 37  |9729362393997257|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dce0  37 35 37 30 30 32 33 36  38 38 33 38 38 37 32 37  |7570023688388727|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dcf0  37 36 38 30 32 30 35 34  33 37 35 32 39 32 39 36  |7680205437529296|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd00  37 38 36 37 30 31 35 31  39 39 39 39 38 34 38 37  |7867015199998487|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd10  32 36 37 35 36 33 31 39  30 36 33 31 38 34 37 37  |2675631906318477|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd20  32 35 36 35 36 32 36 37  33 37 37 33 31 37 31 34  |2565626737731714|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd30  34 30 31 30 33 32 32 35  30 35 38 38 30 35 37 34  |4010322505880574|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd40  32 33 39 30 35 35 30 30  33 39 34 36 36 37 30 30  |2390550039466700|
2018-10-05T23:06:21.294 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd50  37 31 33 31 35 35 38 35  32 37 35 34 34 32 39 39  |7131558527544299|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd60  36 38 39 33 36 31 36 31  30 30 37 37 37 38 37 36  |6893616100777876|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd70  31 32 39 39 39 38 36 32  37 32 38 33 36 39 32 34  |1299986272836924|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd80  38 34 30 36 35 36 37 36  38 34 35 35 34 32 33 36  |8406567684554236|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002dd90  37 33 37 36 31 33 30 30  37 39 33 34 38 36 30 38  |7376130079348608|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002dda0  32 30 34 37 38 33 33 36  38 38 33 32 33 37 30 30  |2047833688323700|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002ddb0  30 37 38 33 33 39 33 30  38 38 36 36 36 37 37 38  |0783393088666778|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002ddc0  39 31 36 38 35 31 34 35  30 38 39 35 35 32 37 37  |9168514508955277|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002ddd0  31 35 30 34 34 33 35 34  33 31 33 30 38 30 38 30  |1504435431308080|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002dde0  33 34 38 30 36 34 35 38  34 37 33 31 39 32 38 30  |3480645847319280|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002ddf0  39 30 37 36 33 32 30 39  33 35 39 34 35 37 34 31  |9076320935945741|
2018-10-05T23:06:21.295 INFO:teuthology.orchestra.run.smithi203.stdout:0002de00  32 34 31 30 30 39 38 35  36 33 38 37 37 37 39 36  |2410098563877796|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de10  37 36 34 33 38 35 32 34  32 33 30 38 30 36 31 34  |7643852423080614|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de20  30 32 34 32 31 34 39 39  39 38 38 38 37 38 35 36  |0242149998887856|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de30  36 39 39 35 36 33 31 39  38 31 39 39 37 30 35 39  |6995631981997059|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de40  34 32 31 38 36 31 39 38  31 38 36 38 38 33 34 34  |4218619818688344|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de50  33 36 39 31 31 32 32 30  33 33 39 32 35 34 31 30  |3691122033925410|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de60  38 35 38 35 38 39 35 39  37 31 39 35 37 33 30 32  |8585895971957302|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de70  39 31 33 33 34 38 33 39  31 34 32 37 30 35 37 39  |9133483914270579|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de80  30 37 36 30 36 31 31 36  34 30 33 31 34 33 33 35  |0760611640314335|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002de90  37 39 38 33 37 31 32 38  37 36 37 38 32 36 37 34  |7983712876782674|
2018-10-05T23:06:21.296 INFO:teuthology.orchestra.run.smithi203.stdout:0002dea0  33 33 35 32 36 36 30 30  37 33 34 33 39 37 30 38  |3352660073439708|
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:0002deb0  38 31 31 35 34 35 36 32  32 35 30 34 32 37 30 37  |8115456225042707|
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:0002dec0  32 37 31 38 34 31 38 33  37 34 38 38 00 00 00 00  |271841837488....|
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:0002ded0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:*
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:00065ec0  00 00 00 00 00 00 00 00  00 00 00 00              |............|
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:00065ecc
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:/build/ceph-13.2.2-65-g56049d5/src/test/objectstore/store_test.cc:3976: Failure
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:Value of: bl_eq(bl, result)
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:  Actual: false
2018-10-05T23:06:21.297 INFO:teuthology.orchestra.run.smithi203.stdout:Expected: true

the diff seems to show that a bunch of blocks got zeroed.
Actions
Copy link
 #2

Updated by Sage Weil over 5 years ago

  • Subject changed from FAILED ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2, where GetParam() = "bluestore" to FAILED ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 (zeros)
Actions
Copy link
 #3

Updated by Sage Weil about 5 years ago

  • Status changed from 12 to Need More Info

this was an ubuntu 18.04 kernel. maybe this was the pread vs swap zeroed pages kerenl bug?

i think we need another occurence to infer a pattern..

Actions
Copy link
 #4

Updated by Josh Durgin almost 5 years ago

  • Status changed from Need More Info to Can't reproduce

Looks like this hasn't happened in 6 months

Actions
Copy link

Also available in: Atom PDF