Bug #24319: ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 failed fsck with stray objects - bluestore - Ceph

Actions

Copy link

Bug #24319

closed

ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 failed fsck with stray objects

Added by Sage Weil almost 6 years ago. Updated almost 6 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Target version:

% Done:

Source:

Tags:

Backport:

mimic,luminous

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2018-05-26T06:22:45.947 INFO:teuthology.orchestra.run.smithi081.stdout:[ RUN      ] ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2
2018-05-26T06:22:47.059 INFO:teuthology.orchestra.run.smithi081.stdout:---------------------- 1 / 16 ----------------------
...
2018-05-26T06:27:45.223 INFO:teuthology.orchestra.run.smithi081.stdout:---------------------- 3 / 16 ----------------------
2018-05-26T06:27:45.223 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_min_alloc_size = 4096
2018-05-26T06:27:45.223 INFO:teuthology.orchestra.run.smithi081.stdout:  max_write = 65536
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  max_size = 1048576
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  alignment = 512
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_max_blob_size = 262144
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_compression_mode = force
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_csum_type = none
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_default_buffered_read = false
2018-05-26T06:27:45.225 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_default_buffered_write = true
2018-05-26T06:27:45.225 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_sync_submit_transaction = true
2018-05-26T06:27:45.247 INFO:teuthology.orchestra.run.smithi081.stderr:seeding object 0
2018-05-26T06:27:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:seeding object 500
2018-05-26T06:27:45.596 INFO:teuthology.orchestra.run.smithi081.stderr:Op 0
2018-05-26T06:27:45.596 INFO:teuthology.orchestra.run.smithi081.stderr:available_objects: 1000 in_flight_objects: 0 total objects: 1000 in_flight 0
2018-05-26T06:27:48.008 INFO:teuthology.orchestra.run.smithi081.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-12-ga1ff11fb/rpm/el7/BUILD/ceph-14.0.0-12-ga1ff11fb/src/test/objectstore/store_test.cc: In function 'void SyntheticWorkloadState::fsck(bool)' thread 7fee194b8a00 time 2018-05-26 06:27:47.996668
2018-05-26T06:27:48.008 INFO:teuthology.orchestra.run.smithi081.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-12-ga1ff11fb/rpm/el7/BUILD/ceph-14.0.0-12-ga1ff11fb/src/test/objectstore/store_test.cc: 4135: FAILED assert(r == 0 || r == -95)

log has

2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:fe000000:::OBJ_638:head# not owned by any collection
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:fe000000:::OBJ_894:head# not owned by any collection
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:ff000000:::OBJ_254:head# not owned by any collection
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:ff000000:::OBJ_510:head# not owned by any collection
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:ff000000:::OBJ_766:64dd7e71# not owned by any collection
...
2018-05-26 06:27:47.629 7fee194b8a00  1 bluestore(bluestore.test_temp_dir) _fsck checking shared_blobs
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: found stray shared blob data for sbid 0x9e4
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: found stray shared blob data for sbid 0x9e5
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: found stray shared blob data for sbid 0x9e6
...
2018-05-26 06:27:47.630 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: actual store_statfs(0x22236d000/0x2625a0000, stored 0x244200/0x231000, compress 0x158c6d/0x152000/0x199800) != expected store_statfs(0x22236d000/0x2625a0000, stored 0x0/0x0, compress 0x0/0x0/0x0)
...
2018-05-26 06:27:47.694 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: leaked extent 0xbd7e000~ef000
2018-05-26 06:27:47.694 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: leaked extent 0xbe77000~142000
...
2018-05-26 06:27:47.995 7fee194b8a00  1 bluestore(bluestore.test_temp_dir) _fsck <<<FINISH>>> with 1017 errors, 0 repaired, 1017 remaining in 0.505983 seconds

/a/sage-2018-05-26_01:26:54-rados-wip-sage2-testing-2018-05-25-1756-distro-basic-smithi/2588806

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Sage Weil almost 6 years ago

/a/sage-2018-05-27_15:10:03-rados-wip-sage2-testing-2018-05-26-1602-distro-basic-smithi/2597987

same!

Actions

Copy link

Updated by Sage Weil almost 6 years ago

/a/sage-2018-05-28_00:58:32-rados-wip-sage2-testing-2018-05-27-1728-distro-basic-smithi/2598499

Actions

Copy link

Updated by Sage Weil almost 6 years ago

Priority changed from Urgent to Immediate
Backport set to mimic,luminous

this is a regression caused by the fix for #24211, which has been backported to mimic and luminous but not yet released. make sure we backport this fix before a release happens!

Actions

Copy link