https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2016-11-04T17:01:13Z
Ceph
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=80783
2016-11-04T17:01:13Z
Kefu Chai
tchaikov@gmail.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/80783/diff?detail_id=77837">diff</a>)</li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=80784
2016-11-04T17:10:28Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><pre>
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
#1 0x00007ffff383640a in __GI_abort () at abort.c:89
#2 0x00007ffff382de47 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55555675da78 "bl_eq(state->contents[hoid].data, r2)",
file=file@entry=0x55555675c490 "/var/ceph/ceph/src/test/objectstore/store_test.cc", line=line@entry=3330,
function=function@entry=0x55555675f5e0 <SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)::__PRETTY_FUNCTION__> "virtual void SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)") at assert.c:92
#3 0x00007ffff382def2 in __GI___assert_fail (assertion=0x55555675da78 "bl_eq(state->contents[hoid].data, r2)",
file=0x55555675c490 "/var/ceph/ceph/src/test/objectstore/store_test.cc", line=3330,
function=0x55555675f5e0 <SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)::__PRETTY_FUNCTION__> "virtual void SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)") at assert.c:101
#4 0x0000555556036e19 in SyntheticWorkloadState::C_SyntheticOnReadable::finish (this=0x55555fce9170, r=290422) at /var/ceph/ceph/src/test/objectstore/store_test.cc:3330
#5 0x000055555602c64b in Context::complete (this=0x55555fce9170, r=0) at /var/ceph/ceph/src/include/Context.h:64
#6 0x000055555630e146 in Finisher::finisher_thread_entry (this=0x55555f023de0) at /var/ceph/ceph/src/common/Finisher.cc:68
#7 0x00005555560718d0 in Finisher::FinisherThread::entry (this=0x55555f023f38) at /var/ceph/ceph/src/common/Finisher.h:66
#8 0x00005555563d2dbb in Thread::entry_wrapper (this=0x55555f023f38) at /var/ceph/ceph/src/common/Thread.cc:89
#9 0x00005555563d2cf0 in Thread::_entry_func (arg=0x55555f023f38) at /var/ceph/ceph/src/common/Thread.cc:69
#10 0x00007ffff7bc4464 in start_thread (arg=0x7fffd3fff700) at pthread_create.c:333
#11 0x00007ffff38ea9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
</pre>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=80848
2016-11-08T12:08:27Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><p>git bisect shows that 933a1da6d7517b8215c0cc720e47374adedf381e is the offending commit.</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=80861
2016-11-08T17:01:10Z
Sage Weil
sage@newdream.net
<ul></ul><p>Are you able to reproduce this locally?</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=80879
2016-11-09T01:05:35Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><p>yes, constantly. and reverting the 933a1da6d7517b8215c0cc720e47374adedf381e helps.</p>
<p>ceph_test_objectstore --gtest_filter=ObjectStore/StoreTest.Synthetic/1</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=81098
2016-11-10T19:32:47Z
Sage Weil
sage@newdream.net
<ul></ul><p>I've tried a few different machines now but I can't reproduce this.</p>
<p>Can you generate a filestore = 20 log for me?</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=81351
2016-11-15T08:09:08Z
Kefu Chai
tchaikov@gmail.com
<ul><li><strong>File</strong> <i>17743.tgz</i> added</li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=81355
2016-11-15T08:58:31Z
Kefu Chai
tchaikov@gmail.com
<ul><li><strong>File</strong> deleted (<del><i>17743.tgz</i></del>)</li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=81356
2016-11-15T09:04:34Z
Kefu Chai
tchaikov@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/download/2578/ceph_test_objectstore.log.xz">ceph_test_objectstore.log.xz</a> added</li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=81359
2016-11-15T09:25:34Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><p>Sage, i will try to fix this if you don't have enough bandwidth today.</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=81429
2016-11-16T13:00:25Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><p>i just tested on ext4 the problem disappears. and seems it is reproducible on btrfs.</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=81430
2016-11-16T13:03:37Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><p><a class="external" href="http://pulpito.ceph.com/kchai-2016-11-13_07:03:13-rados-wip-kefu-testing---basic-smithi/544085/">http://pulpito.ceph.com/kchai-2016-11-13_07:03:13-rados-wip-kefu-testing---basic-smithi/544085/</a><br /><a class="external" href="http://pulpito.ceph.com/samuelj-2016-11-16_21:33:31-rados-wip-sam-working---basic-smithi/554921/">http://pulpito.ceph.com/samuelj-2016-11-16_21:33:31-rados-wip-sam-working---basic-smithi/554921/</a></p>
<p>both of them (smithi072, smithi037) are using ext4. weird...</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=81683
2016-11-21T16:10:16Z
Samuel Just
sjust@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Kefu Chai</i></li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82056
2016-11-27T16:19:33Z
Kefu Chai
tchaikov@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/download/2605/consoleText.txt.gz">consoleText.txt.gz</a> added</li></ul><p>"ceph_test_objectstore --gtest_filter=\*/0" also, see</p>
<p><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/14959/consoleFull#-824916114d63714d2-c8d8-41fc-a9d4-8dee30be4c32">https://jenkins.ceph.com/job/ceph-pull-requests/14959/consoleFull#-824916114d63714d2-c8d8-41fc-a9d4-8dee30be4c32</a></p>
<pre>
0> 2016-11-26 20:48:19.042533 7fd54a19a6c0 -1 *** Caught signal (Aborted) **
in thread 7fd54a19a6c0 thread_name:ceph_test_objec
ceph version 11.0.2-1993-g9cb008d (9cb008dedce2094f54bfeffb698f3092d5af0233)
1: (()+0x682fa2) [0x7fd549caffa2]
2: (()+0x10330) [0x7fd5491fa330]
3: (gsignal()+0x37) [0x7fd5471bdc37]
4: (abort()+0x148) [0x7fd5471c1028]
5: (()+0x2fbf6) [0x7fd5471b6bf6]
6: (()+0x2fca2) [0x7fd5471b6ca2]
7: (()+0x32dd08) [0x7fd54995ad08]
8: (doSyntheticTest(boost::scoped_ptr<ObjectStore>&, int, unsigned long, unsigned long, unsigned long)+0x55e) [0x7fd54990c25e]
9: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x33) [0x7fd549ca81f3]
10: (testing::Test::Run()+0xb7) [0x7fd549c9bc67]
11: (testing::TestInfo::Run()+0x9e) [0x7fd549c9bd0e]
12: (testing::TestCase::Run()+0xa5) [0x7fd549c9be15]
13: (testing::internal::UnitTestImpl::RunAllTests()+0x248) [0x7fd549c9c0c8]
14: (testing::UnitTest::Run()+0x54) [0x7fd549c9c384]
15: (main()+0x35f) [0x7fd549864b0f]
16: (__libc_start_main()+0xf5) [0x7fd5471a8f45]
17: (()+0x2a97b6) [0x7fd5498d67b6]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82057
2016-11-27T16:19:50Z
Kefu Chai
tchaikov@gmail.com
<ul><li><strong>Subject</strong> changed from <i>ceph_test_objectstore --gtest_filter=-*/2'" crashes in qa run </i> to <i>ceph_test_objectstore" crashes in qa run </i></li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82111
2016-11-29T08:09:56Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><p>i am able to reproduce the above failure using "ctest -R test_objectstore_memstore.sh -V --repeat-until-fail 400".</p>
<pre>
Total Test time (real) = 10373.15 sec
The following tests FAILED:
10 - test_objectstore_memstore.sh (Failed)
Errors while running CTest
</pre>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82156
2016-11-29T16:19:49Z
Samuel Just
sjust@redhat.com
<ul><li><strong>Duplicated by</strong> <i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/18027">Bug #18027</a>: test_objectstore_memstore.sh transient failure</i> added</li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82391
2016-12-02T10:58:57Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Subject</strong> changed from <i>ceph_test_objectstore" crashes in qa run </i> to <i>ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run </i></li></ul><p>Updating the title so that it matches when looking for test_objectstore_memstore.sh failures</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82402
2016-12-02T13:40:54Z
Loïc Dachary
loic@dachary.org
<ul></ul><p>repeating the failure as instructed by Kefu and trying to get a core but cannot (3a9bcaa4aa6042587886c0eaae0ce4eeeb8f8164)<br /><pre>
$ ctest -R test_objectstore_memstore.sh --repeat-until-fail 4000
tore_memstore.sh --repeat-until-fail 4000
Test project /slow/loic/ceph-ubuntu-14.04-loic/build
Start 10: test_objectstore_memstore.sh
1/1 Test #10: test_objectstore_memstore.sh ..... Passed 90.86 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 90.92 sec
</pre></p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82406
2016-12-02T14:15:48Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>File</strong> <a href="/attachments/download/2614/consoleText.gz">consoleText.gz</a> added</li></ul><p><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/15167/console">https://jenkins.ceph.com/job/ceph-pull-requests/15167/console</a></p>
<pre>
7/154 Test #10: test_objectstore_memstore.sh ............***Failed 43.45 sec
...
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/store_test.cc:3750: Failure
Expected: r
Which is: 28697
To be equal to: 0
...
[ FAILED ] ObjectStore/StoreTest.Synthetic/0, where GetParam() = "memstore" (34805 ms)
</pre>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82414
2016-12-02T15:21:35Z
Loïc Dachary
loic@dachary.org
<ul></ul><p><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/15166/console">https://jenkins.ceph.com/job/ceph-pull-requests/15166/console</a></p>
<pre>
3/154 Test #10: test_objectstore_memstore.sh ............***Failed 19.38 sec
...
0> 2016-12-01 13:37:34.675656 7ff2a4f18700 -1 *** Caught signal (Aborted) **
in thread 7ff2a4f18700 thread_name:fn_anonymous
ceph version 11.0.2-2148-g99305be (99305be55d0e60e5f6d06aebe4f6a8c1f5832e60)
1: (()+0x682922) [0x7ff2aa45d922]
2: (()+0x10330) [0x7ff2a99a8330]
3: (gsignal()+0x37) [0x7ff2a796bc37]
4: (abort()+0x148) [0x7ff2a796f028]
5: (()+0x2fbf6) [0x7ff2a7964bf6]
6: (()+0x2fca2) [0x7ff2a7964ca2]
7: (SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)+0x367) [0x7ff2aa100607]
8: (Context::complete(int)+0x9) [0x7ff2aa0ddcc9]
9: (Finisher::finisher_thread_entry()+0x1e6) [0x7ff2aa2fd456]
10: (()+0x8184) [0x7ff2a99a0184]
11: (clone()+0x6d) [0x7ff2a7a2f37d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82417
2016-12-02T15:54:16Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fix Under Review</i></li><li><strong>Assignee</strong> changed from <i>Kefu Chai</i> to <i>Loïc Dachary</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/12281">https://github.com/ceph/ceph/pull/12281</a></p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82581
2016-12-05T21:38:47Z
Loïc Dachary
loic@dachary.org
<ul></ul><p><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/15371/console">https://jenkins.ceph.com/job/ceph-pull-requests/15371/console</a> fails despite <a class="external" href="https://github.com/ceph/ceph/pull/12281">https://github.com/ceph/ceph/pull/12281</a></p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82582
2016-12-05T21:40:46Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>File</strong> <a href="/attachments/download/2625/consoleText.3.gz">consoleText.3.gz</a> added</li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82647
2016-12-07T09:26:19Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>File</strong> <a href="/attachments/download/2628/consoleText.4.gz">consoleText.4.gz</a> added</li></ul><p><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/15414/">https://jenkins.ceph.com/job/ceph-pull-requests/15414/</a></p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=82764
2016-12-08T14:45:54Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Need More Info</i></li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=84123
2017-01-12T15:51:26Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><p><a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/16883/consoleFull#-108728127277933967-90d1-4877-8d60-89cb08ef4eb1">https://jenkins.ceph.com/job/ceph-pull-requests/16883/consoleFull#-108728127277933967-90d1-4877-8d60-89cb08ef4eb1</a></p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=89452
2017-04-15T10:00:40Z
Nathan Cutler
ncutler@suse.cz
<ul></ul><p>This failure is plaguing kraken backports - see e.g.</p>
<ul>
<li><a class="external" href="https://github.com/ceph/ceph/pull/14517">https://github.com/ceph/ceph/pull/14517</a></li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/14540">https://github.com/ceph/ceph/pull/14540</a></li>
</ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=92764
2017-06-14T03:52:03Z
Greg Farnum
gfarnum@redhat.com
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>RADOS</i></li></ul>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=94289
2017-07-04T16:46:56Z
Nathan Cutler
ncutler@suse.cz
<ul></ul><p>Happened on another kraken backport: <a class="external" href="https://github.com/ceph/ceph/pull/16108">https://github.com/ceph/ceph/pull/16108</a></p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=94696
2017-07-07T15:38:52Z
Sage Weil
sage@newdream.net
<ul></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/16215">https://github.com/ceph/ceph/pull/16215</a> ?</p>
RADOS - Bug #17743: ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)
https://tracker.ceph.com/issues/17743?journal_id=94775
2017-07-09T02:40:39Z
Sage Weil
sage@newdream.net
<ul><li><strong>Subject</strong> changed from <i>ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run </i> to <i>ceph_test_objectstore & test_objectstore_memstore.sh crashes in qa run (kraken)</i></li><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>Won't Fix</i></li></ul><p>see <a class="external" href="https://github.com/ceph/ceph/pull/16215">https://github.com/ceph/ceph/pull/16215</a> (disabled the memstore tests on kraken)</p>