Bug #20738: memstore: return bad data during ObjectStore/StoreTest.Synthetic/0 - Ceph - Ceph

Custom queries

Backports: mimic
Backports: missing release
Backports: nautilus
Bluestore
Bug queue
Bug queue - no subprojects
Bug triage
Ceph backlog
Crash queue
Crash triage
Feature Requests
Feedback
My issues
Need Review
Pending backports
Priority queue
Product Backlog Scrub
Project Triage
Test Failures

Actions

Copy link

Bug #20738

closed

memstore: return bad data during ObjectStore/StoreTest.Synthetic/0

Added by Sage Weil almost 7 years ago. Updated almost 3 years ago.

Status:

Can't reproduce

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2017-07-21T03:59:56.610 INFO:teuthology.orchestra.run.smithi191.stderr:    -1> 2017-07-21 03:59:51.671473 7fc355134700 -1 --- buffer lengths mismatch expected 0x0 != actual 0x6c28
2017-07-21T03:59:56.610 INFO:teuthology.orchestra.run.smithi191.stderr:     0> 2017-07-21 03:59:51.672458 7fc355134700 -1 /build/ceph-12.1.1-293-gba7ecfb/src/test/objectstore/store_test.cc: In function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' thread 7fc355134700 time 2017-07-21 03:59:5
1.671497
2017-07-21T03:59:56.610 INFO:teuthology.orchestra.run.smithi191.stderr:/build/ceph-12.1.1-293-gba7ecfb/src/test/objectstore/store_test.cc: 3412: FAILED assert(bl_eq(state->contents[noid].data, r2))
2017-07-21T03:59:56.610 INFO:teuthology.orchestra.run.smithi191.stderr:
2017-07-21T03:59:56.611 INFO:teuthology.orchestra.run.smithi191.stderr: ceph version 12.1.1-293-gba7ecfb (ba7ecfbdf7222cffea62f05052ee010e4790c309) luminous (rc)
2017-07-21T03:59:56.611 INFO:teuthology.orchestra.run.smithi191.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc35c0bdda2]
2017-07-21T03:59:56.611 INFO:teuthology.orchestra.run.smithi191.stderr: 2: (SyntheticWorkloadState::C_SyntheticOnClone::finish(int)+0x39a) [0x55a836feef9a]
2017-07-21T03:59:56.611 INFO:teuthology.orchestra.run.smithi191.stderr: 3: (Context::complete(int)+0x9) [0x55a836fca179]

/a/sage-2017-07-21_03:24:52-rados-wip-sage-testing-distro-basic-smithi/1426476
rados/objectstore/objectstore.yaml

Related issues 1 (1 open — 0 closed)

Related to RADOS - Bug #54438: test/objectstore/store_test.cc: FAILED ceph_assert(bl_eq(state->contents[noid].data, r2)) in function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)'

New

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Dan Mick almost 7 years ago

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/3000/consoleFull#-18536214386733401c-e9d0-4737-9832-6594c5da0afa is similar, but a different assert location:

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/objectstore/store_test.cc: 3358: FAILED assert(bl_eq(state->contents[hoid].data, r2))

Actions

Copy link

Updated by Sage Weil almost 7 years ago

Subject changed from memstore: return bad data during ObjectStore/StoreTest.Synthetic/0 to memstore,filestore: return bad data during ObjectStore/StoreTest.Synthetic/0

/a/sage-2017-07-26_14:40:34-rados-wip-sage-testing-distro-basic-smithi/1447369

2017-07-26T16:09:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:    -3> 2017-07-26 16:07:52.055670 7f4caf512e00  1 journal _open store_test_temp_journal fd 117: 419430400 bytes, block size 4096 bytes, directio = 1, aio = 0
2017-07-26T16:09:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:    -2> 2017-07-26 16:07:52.055892 7f4caf512e00  1 filestore(filestore.test_temp_dir) upgrade(1341)
2017-07-26T16:09:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:    -1> 2017-07-26 16:08:00.213237 7f4c9f638700 -1 --- buffer lengths mismatch expected 0x59f7a != actual 0x0
2017-07-26T16:09:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:     0> 2017-07-26 16:08:03.453161 7f4c9f638700 -1 /build/ceph-12.1.1-609-g392d888/src/test/objectstore/store_test.cc: In function 'virtual void SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)' thread 7f4c9f638700 time 2017-07-26 16:0
8:00.213262
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr:/build/ceph-12.1.1-609-g392d888/src/test/objectstore/store_test.cc: 3358: FAILED assert(bl_eq(state->contents[hoid].data, r2))
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr:
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: ceph version 12.1.1-609-g392d888 (392d888dffbe31593a9d9f3d4dbf4f83284fdc58) luminous (rc)
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f4ca5e18c92]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 2: (SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)+0x372) [0x561e51081392]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 3: (Context::complete(int)+0x9) [0x561e5105b869]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 4: (Finisher::finisher_thread_entry()+0x460) [0x7f4ca5e16ea0]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 5: (()+0x76ba) [0x7f4caf0f76ba]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: 6: (clone()+0x6d) [0x7f4ca4a953dd]
2017-07-26T16:09:19.558 INFO:teuthology.orchestra.run.smithi115.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

filestore, though!

Actions

Copy link

Updated by Sage Weil almost 7 years ago

Priority changed from High to Urgent

Actions

Copy link

Updated by Sage Weil over 6 years ago

Subject changed from memstore,filestore: return bad data during ObjectStore/StoreTest.Synthetic/0 to memstore: return bad data during ObjectStore/StoreTest.Synthetic/0
Status changed from 12 to Need More Info
Priority changed from Urgent to High

https://github.com/ceph/ceph/pull/16923

Actions

Copy link

Updated by Sage Weil over 6 years ago

still failing semi-regularly on make check, too.

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/objectstore/store_test.cc: In function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' thread ffffa9bd7b30 time 2017-08-11 13:16:12.341072
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/objectstore/store_test.cc: 3412: FAILED assert(bl_eq(state->contents[noid].data, r2))

Actions

Copy link

Updated by Sage Weil over 6 years ago

https://github.com/ceph/ceph/pull/16995

Actions

Copy link

Updated by David Zafman over 6 years ago

Happened again even with memstore_page_set set to false.

http://pulpito.ceph.com/dzafman-2017-10-17_21:29:56-rados-wip-zafman-testing-distro-basic-smithi/1744814/

2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr:/build/ceph-13.0.0-2040-g954ef40/src/test/objectstore/store_test.cc: In function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' thread 7f5f44b42700 time 2017-10-18 04:51:38.379149
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr:/build/ceph-13.0.0-2040-g954ef40/src/test/objectstore/store_test.cc: 3494: FAILED assert(bl_eq(state->contents[noid].data, r2))
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: ceph version 13.0.0-2040-g954ef40 (954ef4068a7261b4e4fdf8f5203d2deb19bcf030) mimic (dev)
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f5f4bebc1d2]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 2: (SyntheticWorkloadState::C_SyntheticOnClone::finish(int)+0x39a) [0x56537a1cd94a]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 3: (Context::complete(int)+0x9) [0x56537a1a7d99]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 4: (Finisher::finisher_thread_entry()+0x460) [0x7f5f4beba3e0]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 5: (()+0x76ba) [0x7f5f551e56ba]
2017-10-18T04:51:38.576 INFO:teuthology.orchestra.run.smithi101.stderr: 6: (clone()+0x6d) [0x7f5f4ad8a3dd]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr:*** Caught signal (Aborted) **
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: in thread 7f5f44b42700 thread_name:fn_appl_fstore
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: ceph version 13.0.0-2040-g954ef40 (954ef4068a7261b4e4fdf8f5203d2deb19bcf030) mimic (dev)
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 1: (()+0x5981b4) [0x56537a4581b4]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 2: (()+0x11390) [0x7f5f551ef390]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 3: (gsignal()+0x38) [0x7f5f4acb8428]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 4: (abort()+0x16a) [0x7f5f4acba02a]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7f5f4bebc35e]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 6: (SyntheticWorkloadState::C_SyntheticOnClone::finish(int)+0x39a) [0x56537a1cd94a]
2017-10-18T04:51:38.577 INFO:teuthology.orchestra.run.smithi101.stderr: 7: (Context::complete(int)+0x9) [0x56537a1a7d99]
2017-10-18T04:51:38.578 INFO:teuthology.orchestra.run.smithi101.stderr: 8: (Finisher::finisher_thread_entry()+0x460) [0x7f5f4beba3e0]

Actions

Copy link

Updated by Sage Weil almost 3 years ago

Status changed from Need More Info to Can't reproduce

Actions

Copy link

Updated by Radoslaw Zarzynski about 2 years ago

Related to Bug #54438: test/objectstore/store_test.cc: FAILED ceph_assert(bl_eq(state->contents[noid].data, r2)) in function 'virtual void SyntheticWorkloadState::C_SyntheticOnClone::finish(int)' added

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph

Custom queries

Bug #20738

memstore: return bad data during ObjectStore/StoreTest.Synthetic/0

Updated by Dan Mick almost 7 years ago

Updated by Sage Weil almost 7 years ago

Updated by Sage Weil almost 7 years ago

Updated by Sage Weil over 6 years ago

Updated by Sage Weil over 6 years ago

Updated by Sage Weil over 6 years ago

Updated by David Zafman over 6 years ago

Updated by Sage Weil almost 3 years ago

Updated by Radoslaw Zarzynski about 2 years ago