Bug #21063: rados/objectstore/objectstore.yaml fails in "ceph_test_objectstore --gtest_filter=-*/2" (jewel v10.2.10 integration testing) - Ceph - Ceph

Actions

Copy link

Bug #21063

closed

rados/objectstore/objectstore.yaml fails in "ceph_test_objectstore --gtest_filter=-*/2" (jewel v10.2.10 integration testing)

Added by Nathan Cutler over 6 years ago. Updated almost 3 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Test description: rados/objectstore/objectstore.yaml

Log excerpts:

2017-08-22T11:35:57.006 INFO:teuthology.orchestra.run.smithi143.stdout:test/objectstore/store_test.cc:310: Failure
2017-08-22T11:35:57.008 INFO:teuthology.orchestra.run.smithi143.stdout:Expected: (m[SKIP_STEP]) >= (3u), actual: 0 vs 3
2017-08-22T11:35:57.020 INFO:teuthology.orchestra.run.smithi143.stdout:==> rm -r store_test_temp_dir
2017-08-22T11:35:57.034 INFO:teuthology.orchestra.run.smithi143.stdout:[  FAILED  ] ObjectStore/StoreTest.FiemapHoles/3, where GetParam() = "kstore" (426 ms)

2017-08-22T12:16:51.796 INFO:teuthology.orchestra.run.smithi143.stdout:[  FAILED  ] 1 test, listed below:
2017-08-22T12:16:51.798 INFO:teuthology.orchestra.run.smithi143.stdout:[  FAILED  ] ObjectStore/StoreTest.FiemapHoles/3, where GetParam() = "kstore" 
2017-08-22T12:16:51.801 INFO:teuthology.orchestra.run.smithi143.stdout:
2017-08-22T12:16:51.807 INFO:teuthology.orchestra.run.smithi143.stdout: 1 FAILED TEST

Test failures:

Prime suspect:

The failure is being triggered by https://github.com/ceph/ceph/pull/15189 since the test fails with "test/objectstore/store_test.cc:310: Failure" and that line is added by 15189.

List of all rados PRs in the integration branch:

https://github.com/ceph/ceph/pull/15322 (osd snaptrimmer)
https://github.com/ceph/ceph/pull/15236 (MON only)
https://github.com/ceph/ceph/pull/15189 (the prime suspect)
https://github.com/ceph/ceph/pull/15083 (MON only)
https://github.com/ceph/ceph/pull/15065 (PrimaryLogPG)
https://github.com/ceph/ceph/pull/15050 (this one also touches objstore code)
https://github.com/ceph/ceph/pull/14943 (osd: leaked MOSDMap)

Actions

Copy link

Updated by Nathan Cutler over 6 years ago

Description updated (diff)

Actions

Copy link

Updated by Nathan Cutler over 6 years ago

When I removed PR#15189 from the integration branch and re-ran the test, I get a segfault:

2017-08-23T09:31:16.118 INFO:teuthology.orchestra.run.smithi017.stderr:*** Caught signal (Segmentation fault) **
2017-08-23T09:31:16.140 INFO:teuthology.orchestra.run.smithi017.stderr: in thread 7f676c2d04c0 thread_name:ceph_test_objec
2017-08-23T09:31:16.140 INFO:teuthology.orchestra.run.smithi017.stderr:2017-08-23 09:31:16.115868 7f6765273700  4 rocksdb: [default] [JOB 47] Generated table #108: 33 keys, 2165957 bytes
2017-08-23T09:31:16.140 INFO:teuthology.orchestra.run.smithi017.stderr:2017-08-23 09:31:16.115935 7f6765273700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503480676115891, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 108, "file_size": 2165957, "table_properties": {"data_size": 2164140, "index_size": 1414, "filter_size": 0, "raw_key_size": 858, "raw_average_key_size": 26, "raw_value_size": 2162688, "raw_average_value_size": 65536, "num_data_blocks": 33, "num_entries": 33, "filter_policy_name": "", "kDeletedKeys": "0"}}
2017-08-23T09:31:16.194 INFO:teuthology.orchestra.run.smithi017.stderr: ceph version 10.2.9-78-gbdfc04f (bdfc04f416a87e5c1c0f6010b28ab7be0e3ded2e)
2017-08-23T09:31:16.195 INFO:teuthology.orchestra.run.smithi017.stderr: 1: (()+0x5882fa) [0x7f676c8752fa]
2017-08-23T09:31:16.195 INFO:teuthology.orchestra.run.smithi017.stderr: 2: (()+0xf370) [0x7f676a678370]
2017-08-23T09:31:16.196 INFO:teuthology.orchestra.run.smithi017.stderr: 3: (KStore::_do_read_stripe(boost::intrusive_ptr<KStore::Onode>, unsigned long, ceph::buffer::list*)+0x6c) [0x7f676c6625fc]
2017-08-23T09:31:16.196 INFO:teuthology.orchestra.run.smithi017.stderr: 4: (KStore::_do_write(KStore::TransContext*, boost::intrusive_ptr<KStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x314) [0x7f676c6639b4]
2017-08-23T09:31:16.196 INFO:teuthology.orchestra.run.smithi017.stderr: 5: (KStore::_write(KStore::TransContext*, std::shared_ptr<KStore::Collection>&, boost::intrusive_ptr<KStore::Onode>&, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0xf2) [0x7f676c664b62]
2017-08-23T09:31:16.196 INFO:teuthology.orchestra.run.smithi017.stderr: 6: (KStore::_txc_add_transaction(KStore::TransContext*, ObjectStore::Transaction*)+0x16ff) [0x7f676c6773bf]
2017-08-23T09:31:16.196 INFO:teuthology.orchestra.run.smithi017.stderr: 7: (KStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, std::shared_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3a5) [0x7f676c6789c5]
2017-08-23T09:31:16.197 INFO:teuthology.orchestra.run.smithi017.stderr: 8: (ObjectStore::apply_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*)+0x1b6) [0x7f676c579bc6]
2017-08-23T09:31:16.197 INFO:teuthology.orchestra.run.smithi017.stderr: 9: (int apply_transaction<boost::scoped_ptr<ObjectStore> >(boost::scoped_ptr<ObjectStore>&, ObjectStore::Sequencer*, ObjectStore::Transaction&&)+0x7d) [0x7f676c565cad]
2017-08-23T09:31:16.197 INFO:teuthology.orchestra.run.smithi017.stderr: 10: (StoreTest_ManyBigWrite_Test::TestBody()+0x5d6) [0x7f676c526236]
2017-08-23T09:31:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: 11: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x43) [0x7f676c86d403]
2017-08-23T09:31:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: 12: (testing::Test::Run()+0xb7) [0x7f676c85ebc7]
2017-08-23T09:31:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: 13: (testing::TestInfo::Run()+0x9e) [0x7f676c85ec6e]
2017-08-23T09:31:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: 14: (testing::TestCase::Run()+0xa5) [0x7f676c85ed75]
2017-08-23T09:31:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: 15: (testing::internal::UnitTestImpl::RunAllTests()+0x268) [0x7f676c85f048]
2017-08-23T09:31:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: 16: (testing::UnitTest::Run()+0x68) [0x7f676c85f308]
2017-08-23T09:31:16.198 INFO:teuthology.orchestra.run.smithi017.stderr: 17: (main()+0x26c) [0x7f676c4d9a0c]
2017-08-23T09:31:16.199 INFO:teuthology.orchestra.run.smithi017.stderr: 18: (__libc_start_main()+0xf5) [0x7f6769482b35]
2017-08-23T09:31:16.199 INFO:teuthology.orchestra.run.smithi017.stderr: 19: (()+0x2335df) [0x7f676c5205df]

Test URL: http://pulpito.front.sepia.ceph.com/smithfarm-2017-08-23_08:55:27-rados-wip-jewel-backports-distro-basic-smithi/1554282/

NOTE: Sage says KStore is not supported, so this failure/segfault is not a blocker.

Actions

Copy link