Project

General

Profile

Actions
Copy link

Bug #42189

closed

SyntheticWorkloadState::scan abort

Added by Sage Weil over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Igor Fedotov
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
30753
Crash signature (v1):
Crash signature (v2):

Description

2019-10-06T20:09:47.328 INFO:teuthology.orchestra.run.smithi069.stdout:[ RUN      ] ObjectStore/StoreTest.Synthetic/2
2019-10-06T20:09:48.989 INFO:teuthology.orchestra.run.smithi069.stderr:seeding object 0
2019-10-06T20:09:49.193 INFO:teuthology.orchestra.run.smithi069.stderr:seeding object 500
2019-10-06T20:09:49.369 INFO:teuthology.orchestra.run.smithi069.stderr:Op 0
2019-10-06T20:09:49.369 INFO:teuthology.orchestra.run.smithi069.stderr:available_objects: 1000 in_flight_objects: 0 total objects: 1000 in_flight 0
2019-10-06T20:09:50.162 INFO:teuthology.orchestra.run.smithi069.stdout:/build/ceph-15.0.0-5845-gbbc7bb5/src/test/objectstore/store_test.cc:4451: Failure
2019-10-06T20:09:50.162 INFO:teuthology.orchestra.run.smithi069.stdout:Expected: (available_objects.count(*i)) > ((unsigned)0), actual: 0 vs 0
2019-10-06T20:09:51.647 INFO:teuthology.orchestra.run.smithi069.stderr:+ #555:04000000:::OBJ_31aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head#
2019-10-06T20:09:51.654 INFO:teuthology.orchestra.run.smithi069.stderr:/build/ceph-15.0.0-5845-gbbc7bb5/src/test/objectstore/store_test.cc: In function 'void SyntheticWorkloadState::scan()' thread 7f192699ddc0 time 2019-10-06T20:09:51.653618+0000
2019-10-06T20:09:51.654 INFO:teuthology.orchestra.run.smithi069.stderr:/build/ceph-15.0.0-5845-gbbc7bb5/src/test/objectstore/store_test.cc: 4435: ceph_abort_msg("abort() called")
2019-10-06T20:09:51.654 INFO:teuthology.orchestra.run.smithi069.stderr: ceph version 15.0.0-5845-gbbc7bb5 (bbc7bb5a222d2ac0b058c5f2d4319b70390111de) octopus (dev)
2019-10-06T20:09:51.655 INFO:teuthology.orchestra.run.smithi069.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xdf) [0x7f191c892d28]
2019-10-06T20:09:51.655 INFO:teuthology.orchestra.run.smithi069.stderr: 2: (SyntheticWorkloadState::scan()+0xba7) [0x564dc44cd767]
2019-10-06T20:09:51.655 INFO:teuthology.orchestra.run.smithi069.stderr: 3: (StoreTest::doSyntheticTest(int, unsigned long, unsigned long, unsigned long)+0x858) [0x564dc4483318]
2019-10-06T20:09:51.655 INFO:teuthology.orchestra.run.smithi069.stderr: 4: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x87) [0x564dc472f6a7]
2019-10-06T20:09:51.655 INFO:teuthology.orchestra.run.smithi069.stderr: 5: (testing::Test::Run()+0xc3) [0x564dc4726763]
2019-10-06T20:09:51.655 INFO:teuthology.orchestra.run.smithi069.stderr: 6: (testing::TestInfo::Run()+0x135) [0x564dc47268c5]
2019-10-06T20:09:51.656 INFO:teuthology.orchestra.run.smithi069.stderr: 7: (testing::TestSuite::Run()+0xb5) [0x564dc47269a5]
2019-10-06T20:09:51.656 INFO:teuthology.orchestra.run.smithi069.stderr: 8: (testing::internal::UnitTestImpl::RunAllTests()+0x453) [0x564dc4726e93]
2019-10-06T20:09:51.656 INFO:teuthology.orchestra.run.smithi069.stderr: 9: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x87) [0x564dc472fbb7]
2019-10-06T20:09:51.656 INFO:teuthology.orchestra.run.smithi069.stderr: 10: (testing::UnitTest::Run()+0x9c) [0x564dc47270ac]
2019-10-06T20:09:51.656 INFO:teuthology.orchestra.run.smithi069.stderr: 11: (main()+0x6fc) [0x564dc4417e8c]
2019-10-06T20:09:51.656 INFO:teuthology.orchestra.run.smithi069.stderr: 12: (__libc_start_main()+0xe7) [0x7f191b43bb97]
2019-10-06T20:09:51.656 INFO:teuthology.orchestra.run.smithi069.stderr: 13: (_start()+0x2a) [0x564dc441bf2a]

/a/sage-2019-10-06_19:16:50-rados-master-distro-basic-smithi/4364704

Related issues 1 (0 open — 1 closed)

Related to bluestore - Bug #42209: STATE_KV_SUBMITTED is set too early.ResolvedIgor Fedotov

Actions
Actions
Copy link
 #1

Updated by Igor Fedotov over 4 years ago

  • Assignee set to Igor Fedotov

I observed similar and some related issues, e.g. SimpleListTest test case. The reason is somehow bound to collection_list call. Looks like it might attempt to retrieve data from DB before previous writes land there. E.g. for SimpleListTest the call occasionally returns empty list immediately after 200 new objects have been created.
So I suspect collection flush mechanics is broken... Investigating deeper..

Actions
Copy link
 #2

Updated by Igor Fedotov over 4 years ago

  • Status changed from 12 to Fix Under Review
  • Pull request ID set to 30753
Actions
Copy link
 #3

Updated by Igor Fedotov over 4 years ago

This is a regression (caused by https://github.com/ceph/ceph/commit/a2fa546d02cfe2a910413acdec5ef11dbfacb359) )which made the existing issue more visible.
In fact we had had it originally as well but setting the STATE_KV_SUBMITTED inside _txc_state_proc was affected only. "regression" patch unified wrong behavior for every state updating point.
So we probably need some reevant fixes for earlier releases...

Actions
Copy link
 #4

Updated by Igor Fedotov over 4 years ago

Relevant ticket for Nautilus: https://tracker.ceph.com/issues/42209

Actions
Copy link
 #5

Updated by Nathan Cutler over 4 years ago

  • Related to Bug #42209: STATE_KV_SUBMITTED is set too early. added
Actions
Copy link
 #6

Updated by Sage Weil over 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF