Project

General

Profile

Bug #24319

ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 failed fsck with stray objects

Added by Sage Weil about 1 year ago. Updated 12 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Target version:
-
Start date:
05/26/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
mimic,luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

2018-05-26T06:22:45.947 INFO:teuthology.orchestra.run.smithi081.stdout:[ RUN      ] ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2
2018-05-26T06:22:47.059 INFO:teuthology.orchestra.run.smithi081.stdout:---------------------- 1 / 16 ----------------------
...
2018-05-26T06:27:45.223 INFO:teuthology.orchestra.run.smithi081.stdout:---------------------- 3 / 16 ----------------------
2018-05-26T06:27:45.223 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_min_alloc_size = 4096
2018-05-26T06:27:45.223 INFO:teuthology.orchestra.run.smithi081.stdout:  max_write = 65536
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  max_size = 1048576
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  alignment = 512
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_max_blob_size = 262144
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_compression_mode = force
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_csum_type = none
2018-05-26T06:27:45.224 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_default_buffered_read = false
2018-05-26T06:27:45.225 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_default_buffered_write = true
2018-05-26T06:27:45.225 INFO:teuthology.orchestra.run.smithi081.stdout:  bluestore_sync_submit_transaction = true
2018-05-26T06:27:45.247 INFO:teuthology.orchestra.run.smithi081.stderr:seeding object 0
2018-05-26T06:27:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:seeding object 500
2018-05-26T06:27:45.596 INFO:teuthology.orchestra.run.smithi081.stderr:Op 0
2018-05-26T06:27:45.596 INFO:teuthology.orchestra.run.smithi081.stderr:available_objects: 1000 in_flight_objects: 0 total objects: 1000 in_flight 0
2018-05-26T06:27:48.008 INFO:teuthology.orchestra.run.smithi081.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-12-ga1ff11fb/rpm/el7/BUILD/ceph-14.0.0-12-ga1ff11fb/src/test/objectstore/store_test.cc: In function 'void SyntheticWorkloadState::fsck(bool)' thread 7fee194b8a00 time 2018-05-26 06:27:47.996668
2018-05-26T06:27:48.008 INFO:teuthology.orchestra.run.smithi081.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-12-ga1ff11fb/rpm/el7/BUILD/ceph-14.0.0-12-ga1ff11fb/src/test/objectstore/store_test.cc: 4135: FAILED assert(r == 0 || r == -95)

log has
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:fe000000:::OBJ_638:head# not owned by any collection
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:fe000000:::OBJ_894:head# not owned by any collection
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:ff000000:::OBJ_254:head# not owned by any collection
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:ff000000:::OBJ_510:head# not owned by any collection
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: stray object #555:ff000000:::OBJ_766:64dd7e71# not owned by any collection
...
2018-05-26 06:27:47.629 7fee194b8a00  1 bluestore(bluestore.test_temp_dir) _fsck checking shared_blobs
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: found stray shared blob data for sbid 0x9e4
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: found stray shared blob data for sbid 0x9e5
2018-05-26 06:27:47.629 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: found stray shared blob data for sbid 0x9e6
...
2018-05-26 06:27:47.630 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: actual store_statfs(0x22236d000/0x2625a0000, stored 0x244200/0x231000, compress 0x158c6d/0x152000/0x199800) != expected store_statfs(0x22236d000/0x2625a0000, stored 0x0/0x0, compress 0x0/0x0/0x0)
...
2018-05-26 06:27:47.694 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: leaked extent 0xbd7e000~ef000
2018-05-26 06:27:47.694 7fee194b8a00 -1 bluestore(bluestore.test_temp_dir) fsck error: leaked extent 0xbe77000~142000
...
2018-05-26 06:27:47.995 7fee194b8a00  1 bluestore(bluestore.test_temp_dir) _fsck <<<FINISH>>> with 1017 errors, 0 repaired, 1017 remaining in 0.505983 seconds

/a/sage-2018-05-26_01:26:54-rados-wip-sage2-testing-2018-05-25-1756-distro-basic-smithi/2588806

Related issues

Related to bluestore - Bug #24550: collection sequencers are not reused; delete and create collection reordered Resolved 06/18/2018
Copied to bluestore - Backport #24502: mimic: ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 failed fsck with stray objects Resolved
Copied to bluestore - Backport #24503: luminous: ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 failed fsck with stray objects Resolved

History

#1 Updated by Sage Weil about 1 year ago

/a/sage-2018-05-27_15:10:03-rados-wip-sage2-testing-2018-05-26-1602-distro-basic-smithi/2597987

same!

#2 Updated by Sage Weil about 1 year ago

/a/sage-2018-05-28_00:58:32-rados-wip-sage2-testing-2018-05-27-1728-distro-basic-smithi/2598499

#3 Updated by Sage Weil about 1 year ago

  • Priority changed from Urgent to Immediate
  • Backport set to mimic,luminous

this is a regression caused by the fix for #24211, which has been backported to mimic and luminous but not yet released. make sure we backport this fix before a release happens!

#4 Updated by Sage Weil about 1 year ago

  • Status changed from Verified to Pending Backport

#5 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #24502: mimic: ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 failed fsck with stray objects added

#6 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #24503: luminous: ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 failed fsck with stray objects added

#7 Updated by Sage Weil about 1 year ago

  • Status changed from Pending Backport to In Progress

https://github.com/ceph/ceph/pull/22499 was an attempted fix, but it isn't complete (test still fails, but a bit differently now)

#8 Updated by Sage Weil 12 months ago

  • Status changed from In Progress to Pending Backport
  • Priority changed from Immediate to Urgent

#9 Updated by Sage Weil 12 months ago

  • Related to Bug #24550: collection sequencers are not reused; delete and create collection reordered added

#10 Updated by Nathan Cutler 12 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF