Project

General

Profile

Actions
Copy link

Bug #20188

open

filestore: os/filestore/FileStore.h: 357: FAILED assert(q.empty()) from ceph_test_objectstore ObjectStore/StoreTest.MultiSmallWriteSameBlock/1

Added by Sage Weil almost 7 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Tests
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
FileStore
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-06-05T16:00:38.490 INFO:teuthology.orchestra.run.smithi021.stderr:   -10> 2017-06-05 16:00:38.317307 7f834af34b80  0 filestore(filestore.test_temp_dir) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -9> 2017-06-05 16:00:38.317386 7f834af34b80  2 journal open store_test_temp_journal fsid 5707d813-38cf-4423-95f9-05ec8314f403 fs_op_seq 1
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -8> 2017-06-05 16:00:38.317402 7f834af34b80 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -7> 2017-06-05 16:00:38.317407 7f834af34b80  1 journal _open store_test_temp_journal fd 11: 419430400 bytes, block size 4096 bytes, directio = 1, aio = 0
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -6> 2017-06-05 16:00:38.317590 7f834af34b80  2 journal open advancing committed_seq 0 to fs op_seq 1
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -5> 2017-06-05 16:00:38.317634 7f834af34b80  2 journal No further valid entries found, journal is most likely valid
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -4> 2017-06-05 16:00:38.317644 7f834af34b80  2 journal No further valid entries found, journal is most likely valid
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -3> 2017-06-05 16:00:38.317648 7f834af34b80  3 journal journal_replay: end of journal, done.
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -2> 2017-06-05 16:00:38.317665 7f834af34b80  1 journal _open store_test_temp_journal fd 11: 419430400 bytes, block size 4096 bytes, directio = 1, aio = 0
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:    -1> 2017-06-05 16:00:38.317877 7f834af34b80  1 filestore(filestore.test_temp_dir) upgrade
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:     0> 2017-06-05 16:00:38.416141 7f834af34b80 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.2-2248-g8c17cc5/rpm/el7/BUILD/ceph-12.
0.2-2248-g8c17cc5/src/os/filestore/FileStore.h: In function 'virtual FileStore::OpSequencer::~OpSequencer()' thread 7f834af34b80 time 2017-06-05 16:00:38.411756
2017-06-05T16:00:38.491 INFO:teuthology.orchestra.run.smithi021.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.2-2248-g8c17cc5/rpm/el7/BUILD/ceph-12.0.2-2248-g8c17cc5/src/os/filestore/FileStore.h: 357
: FAILED assert(q.empty())
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr:
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: ceph version  12.0.2-2248-g8c17cc5 (8c17cc5e289cdd593b537d3b627451e9483d742b) luminous (dev)
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f83423e89f0]
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 2: (()+0x158c52) [0x7f834b0adc52]
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 3: (FileStore::OpSequencer::~OpSequencer()+0x109) [0x7f834b22f699]
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 4: (RefCountedObject::put() const+0xfa) [0x7f834b1b849a]
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 5: (ObjectStore::Sequencer::~Sequencer()+0x35) [0x7f834b1b88a5]
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 6: (StoreTest_MultiSmallWriteSameBlock_Test::TestBody()+0x2fe) [0x7f834b17812e]
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 7: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x43) [0x7f834b413a43]
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 8: (testing::Test::Run()+0xb7) [0x7f834b406657]
2017-06-05T16:00:38.492 INFO:teuthology.orchestra.run.smithi021.stderr: 9: (testing::TestInfo::Run()+0x9e) [0x7f834b4066fe]
2017-06-05T16:00:38.493 INFO:teuthology.orchestra.run.smithi021.stderr: 10: (testing::TestCase::Run()+0xa5) [0x7f834b406805]
2017-06-05T16:00:38.493 INFO:teuthology.orchestra.run.smithi021.stderr: 11: (testing::internal::UnitTestImpl::RunAllTests()+0x278) [0x7f834b406ae8]
2017-06-05T16:00:38.493 INFO:teuthology.orchestra.run.smithi021.stderr: 12: (testing::UnitTest::Run()+0x68) [0x7f834b406db8]
2017-06-05T16:00:38.493 INFO:teuthology.orchestra.run.smithi021.stderr: 13: (main()+0x2de) [0x7f834b0afcce]
2017-06-05T16:00:38.493 INFO:teuthology.orchestra.run.smithi021.stderr: 14: (__libc_start_main()+0xf5) [0x7f833e9adb35]
2017-06-05T16:00:38.493 INFO:teuthology.orchestra.run.smithi021.stderr: 15: (()+0x1ec041) [0x7f834b141041]
2017-06-05T16:00:38.493 INFO:teuthology.orchestra.run.smithi021.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

/a/sage-2017-06-05_14:47:27-rados-wip-sage-testing-distro-basic-smithi/1260464
Actions
Copy link
 #1

Updated by Greg Farnum almost 7 years ago

  • Project changed from Ceph to RADOS
  • Category set to Tests
  • Component(RADOS) FileStore added

This testing branch didn't include any of the filestore improvements we've been getting, did it?

Actions
Copy link
 #2

Updated by Sage Weil over 6 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link
 #3

Updated by Kefu Chai over 6 years ago

  • Status changed from Can't reproduce to New

/a//kchai-2017-08-25_08:38:31-rados-wip-kefu-testing-distro-basic-smithi/1561884

2017-08-25T09:23:52.025 INFO:teuthology.orchestra.run.smithi197.stdout:[ RUN      ] ObjectStore/StoreTest.MultiSmallWriteSameBlock/1
2017-08-25T09:23:52.118 INFO:teuthology.orchestra.run.smithi197.stderr:Creating collection meta
2017-08-25T09:23:52.125 INFO:teuthology.orchestra.run.smithi197.stderr:Cleaning
2017-08-25T09:23:52.372 INFO:teuthology.orchestra.run.smithi197.stderr:Creating collection meta
2017-08-25T09:23:52.504 INFO:teuthology.orchestra.run.smithi197.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/cento
s7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.2-1115-g2b36d79/rpm/el7/BUILD/ceph-12.1.2-1115-g2b36d79/src/test/objectstore/store_test.cc:1855: Failure
2017-08-25T09:23:52.519 INFO:teuthology.orchestra.run.smithi197.stdout:Expected: (r) >= (0), actual: -2 vs 0
2017-08-25T09:23:52.603 INFO:teuthology.orchestra.run.smithi197.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/cento
s7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.2-1115-g2b36d79/rpm/el7/BUILD/ceph-12.1.2-1115-g2b36d79/src/os/filestore/FileStore.h: In function 'virtual FileStore::OpSequencer::~O
pSequencer()' thread 7fd2c9addd00 time 2017-08-25 09:23:52.507943
2017-08-25T09:23:52.606 INFO:teuthology.orchestra.run.smithi197.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/cento
s7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.2-1115-g2b36d79/rpm/el7/BUILD/ceph-12.1.2-1115-g2b36d79/src/os/filestore/FileStore.h: 358: FAILED assert(q.empty())
2017-08-25T09:23:52.661 INFO:teuthology.orchestra.run.smithi197.stderr: ceph version 12.1.2-1115-g2b36d79 (2b36d79ebc48283b920c68afac051329a6556085) mimic (dev)
2017-08-25T09:23:52.677 INFO:teuthology.orchestra.run.smithi197.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7fd2c0eec670]
2017-08-25T09:23:52.720 INFO:teuthology.orchestra.run.smithi197.stderr: 2: (()+0x174d3c) [0x7fd2c9c72d3c]
2017-08-25T09:23:52.759 INFO:teuthology.orchestra.run.smithi197.stderr: 3: (FileStore::OpSequencer::~OpSequencer()+0x100) [0x7fd2c9dfa9a0]
2017-08-25T09:23:52.776 INFO:teuthology.orchestra.run.smithi197.stderr: 4: (RefCountedObject::put() const+0xfc) [0x7fd2c9d80c1c]
2017-08-25T09:23:52.836 INFO:teuthology.orchestra.run.smithi197.stderr: 5: (ObjectStore::Sequencer::~Sequencer()+0x35) [0x7fd2c9d81025]
2017-08-25T09:23:52.840 INFO:teuthology.orchestra.run.smithi197.stderr: 6: (StoreTest_MultiSmallWriteSameBlock_Test::TestBody()+0x2f5) [0x7fd2c9d3f225]
2017-08-25T09:23:52.842 INFO:teuthology.orchestra.run.smithi197.stderr: 7: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (t
esting::Test::*)(), char const*)+0x43) [0x7fd2c9fe1dd3]
Actions
Copy link
 #4

Updated by Sage Weil over 6 years ago

  • Priority changed from High to Normal
Actions
Copy link
 #5

Updated by Kefu Chai over 6 years ago

/a/kchai-2017-12-01_08:15:14-rados-wip-kefu-testing-2017-12-01-1256-distro-basic-mira/1912013

Actions
Copy link
 #6

Updated by David Zafman over 4 years ago

  • Priority changed from Normal to High

/a/yuriw-2019-11-02_14:48:27-rados-wip-yuri-testing-2019-11-01-1917-luminous-distro-basic-smithi/4467375

Actions
Copy link
 #7

Updated by David Zafman over 4 years ago

  • Priority changed from High to Normal
Actions
Copy link

Also available in: Atom PDF