Project

General

Profile

Actions
Copy link

Bug #20738

closed

memstore: return bad data during ObjectStore/StoreTest.Synthetic/0

Added by Sage Weil over 6 years ago. Updated almost 3 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-07-21T03:59:56.610 INFO:teuthology.orchestra.run.smithi191.stderr:    -1> 2017-07-21 03:59:51.671473 7fc355134700 -1 --- buffer lengths mismatch expected 0x0 != actual 0x6c28
2017-07-21T03:59:56.610 INFO:teuthology.orchestra.run.smithi191.stderr:     0> 2017-07-21 03:59:51.672458 7fc355134700 -1 /build/ceph-12.1.1-293-gba7ecfb/src/test/objectstore/store_test.cc: In function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' thread 7fc355134700 time 2017-07-21 03:59:5
1.671497
2017-07-21T03:59:56.610 INFO:teuthology.orchestra.run.smithi191.stderr:/build/ceph-12.1.1-293-gba7ecfb/src/test/objectstore/store_test.cc: 3412: FAILED assert(bl_eq(state->contents[noid].data, r2))
2017-07-21T03:59:56.610 INFO:teuthology.orchestra.run.smithi191.stderr:
2017-07-21T03:59:56.611 INFO:teuthology.orchestra.run.smithi191.stderr: ceph version 12.1.1-293-gba7ecfb (ba7ecfbdf7222cffea62f05052ee010e4790c309) luminous (rc)
2017-07-21T03:59:56.611 INFO:teuthology.orchestra.run.smithi191.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc35c0bdda2]
2017-07-21T03:59:56.611 INFO:teuthology.orchestra.run.smithi191.stderr: 2: (SyntheticWorkloadState::C_SyntheticOnClone::finish(int)+0x39a) [0x55a836feef9a]
2017-07-21T03:59:56.611 INFO:teuthology.orchestra.run.smithi191.stderr: 3: (Context::complete(int)+0x9) [0x55a836fca179]

/a/sage-2017-07-21_03:24:52-rados-wip-sage-testing-distro-basic-smithi/1426476
rados/objectstore/objectstore.yaml

Related issues 1 (1 open — 0 closed)

Related to RADOS - Bug #54438: test/objectstore/store_test.cc: FAILED ceph_assert(bl_eq(state->contents[noid].data, r2)) in function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' New

Actions
Actions
Copy link
 #1

Updated by Dan Mick over 6 years ago

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/3000/consoleFull#-18536214386733401c-e9d0-4737-9832-6594c5da0afa is similar, but a different assert location:

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/objectstore/store_test.cc: 3358: FAILED assert(bl_eq(state->contents[hoid].data, r2))

Actions
Copy link
 #2

Updated by Sage Weil over 6 years ago

  • Subject changed from memstore: return bad data during ObjectStore/StoreTest.Synthetic/0 to memstore,filestore: return bad data during ObjectStore/StoreTest.Synthetic/0

/a/sage-2017-07-26_14:40:34-rados-wip-sage-testing-distro-basic-smithi/1447369

2017-07-26T16:09:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:    -3> 2017-07-26 16:07:52.055670 7f4caf512e00  1 journal _open store_test_temp_journal fd 117: 419430400 bytes, block size 4096 bytes, directio = 1, aio = 0
2017-07-26T16:09:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:    -2> 2017-07-26 16:07:52.055892 7f4caf512e00  1 filestore(filestore.test_temp_dir) upgrade(1341)
2017-07-26T16:09:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:    -1> 2017-07-26 16:08:00.213237 7f4c9f638700 -1 --- buffer lengths mismatch expected 0x59f7a != actual 0x0
2017-07-26T16:09:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:     0> 2017-07-26 16:08:03.453161 7f4c9f638700 -1 /build/ceph-12.1.1-609-g392d888/src/test/objectstore/store_test.cc: In function 'virtual void SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)' thread 7f4c9f638700 time 2017-07-26 16:0
8:00.213262
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr:/build/ceph-12.1.1-609-g392d888/src/test/objectstore/store_test.cc: 3358: FAILED assert(bl_eq(state->contents[hoid].data, r2))
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr:
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: ceph version 12.1.1-609-g392d888 (392d888dffbe31593a9d9f3d4dbf4f83284fdc58) luminous (rc)
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f4ca5e18c92]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 2: (SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)+0x372) [0x561e51081392]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 3: (Context::complete(int)+0x9) [0x561e5105b869]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 4: (Finisher::finisher_thread_entry()+0x460) [0x7f4ca5e16ea0]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 5: (()+0x76ba) [0x7f4caf0f76ba]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 6: (clone()+0x6d) [0x7f4ca4a953dd]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

filestore, though!
Actions
Copy link
 #3

Updated by Sage Weil over 6 years ago

  • Priority changed from High to Urgent
Actions
Copy link
 #4

Updated by Sage Weil over 6 years ago

  • Subject changed from memstore,filestore: return bad data during ObjectStore/StoreTest.Synthetic/0 to memstore: return bad data during ObjectStore/StoreTest.Synthetic/0
  • Status changed from 12 to Need More Info
  • Priority changed from Urgent to High
Actions
Copy link
 #5

Updated by Sage Weil over 6 years ago

still failing semi-regularly on make check, too.

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/objectstore/store_test.cc: In function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' thread ffffa9bd7b30 time 2017-08-11 13:16:12.341072
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/objectstore/store_test.cc: 3412: FAILED assert(bl_eq(state->contents[noid].data, r2))
Actions
Copy link
 #7

Updated by David Zafman over 6 years ago

Happened again even with memstore_page_set set to false.

http://pulpito.ceph.com/dzafman-2017-10-17_21:29:56-rados-wip-zafman-testing-distro-basic-smithi/1744814/

2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr:/build/ceph-13.0.0-2040-g954ef40/src/test/objectstore/store_test.cc: In function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' thread 7f5f44b42700 time 2017-10-18 04:51:38.379149
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr:/build/ceph-13.0.0-2040-g954ef40/src/test/objectstore/store_test.cc: 3494: FAILED assert(bl_eq(state->contents[noid].data, r2))
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: ceph version 13.0.0-2040-g954ef40 (954ef4068a7261b4e4fdf8f5203d2deb19bcf030) mimic (dev)
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f5f4bebc1d2]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 2: (SyntheticWorkloadState::C_SyntheticOnClone::finish(int)+0x39a) [0x56537a1cd94a]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 3: (Context::complete(int)+0x9) [0x56537a1a7d99]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 4: (Finisher::finisher_thread_entry()+0x460) [0x7f5f4beba3e0]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 5: (()+0x76ba) [0x7f5f551e56ba]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 6: (clone()+0x6d) [0x7f5f4ad8a3dd]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr:*** Caught signal (Aborted) **
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: in thread 7f5f44b42700 thread_name:fn_appl_fstore
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: ceph version 13.0.0-2040-g954ef40 (954ef4068a7261b4e4fdf8f5203d2deb19bcf030) mimic (dev)
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 1: (()+0x5981b4) [0x56537a4581b4]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 2: (()+0x11390) [0x7f5f551ef390]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 3: (gsignal()+0x38) [0x7f5f4acb8428]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 4: (abort()+0x16a) [0x7f5f4acba02a]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7f5f4bebc35e]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 6: (SyntheticWorkloadState::C_SyntheticOnClone::finish(int)+0x39a) [0x56537a1cd94a]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 7: (Context::complete(int)+0x9) [0x56537a1a7d99]
2017-10-18T04:51:38.578 INFO:teuthology.orchestra.run.smithi101.stderr: 8: (Finisher::finisher_thread_entry()+0x460) [0x7f5f4beba3e0]
Actions
Copy link
 #8

Updated by Sage Weil almost 3 years ago

  • Status changed from Need More Info to Can't reproduce
Actions
Copy link
 #9

Updated by Radoslaw Zarzynski about 2 years ago

  • Related to Bug #54438: test/objectstore/store_test.cc: FAILED ceph_assert(bl_eq(state->contents[noid].data, r2)) in function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' added
Actions
Copy link

Also available in: Atom PDF