Ceph » RADOS

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
#1  0x00007ffff383640a in __GI_abort () at abort.c:89
#2  0x00007ffff382de47 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55555675da78 "bl_eq(state->contents[hoid].data, r2)",
    file=file@entry=0x55555675c490 "/var/ceph/ceph/src/test/objectstore/store_test.cc", line=line@entry=3330,
    function=function@entry=0x55555675f5e0 <SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)::__PRETTY_FUNCTION__> "virtual void SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)") at assert.c:92
#3  0x00007ffff382def2 in __GI___assert_fail (assertion=0x55555675da78 "bl_eq(state->contents[hoid].data, r2)",
    file=0x55555675c490 "/var/ceph/ceph/src/test/objectstore/store_test.cc", line=3330,
    function=0x55555675f5e0 <SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)::__PRETTY_FUNCTION__> "virtual void SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)") at assert.c:101
#4  0x0000555556036e19 in SyntheticWorkloadState::C_SyntheticOnReadable::finish (this=0x55555fce9170, r=290422) at /var/ceph/ceph/src/test/objectstore/store_test.cc:3330
#5  0x000055555602c64b in Context::complete (this=0x55555fce9170, r=0) at /var/ceph/ceph/src/include/Context.h:64
#6  0x000055555630e146 in Finisher::finisher_thread_entry (this=0x55555f023de0) at /var/ceph/ceph/src/common/Finisher.cc:68
#7  0x00005555560718d0 in Finisher::FinisherThread::entry (this=0x55555f023f38) at /var/ceph/ceph/src/common/Finisher.h:66
#8  0x00005555563d2dbb in Thread::entry_wrapper (this=0x55555f023f38) at /var/ceph/ceph/src/common/Thread.cc:89
#9  0x00005555563d2cf0 in Thread::_entry_func (arg=0x55555f023f38) at /var/ceph/ceph/src/common/Thread.cc:69
#10 0x00007ffff7bc4464 in start_thread (arg=0x7fffd3fff700) at pthread_create.c:333
#11 0x00007ffff38ea9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

git bisect shows that 933a1da6d7517b8215c0cc720e47374adedf381e is the offending commit.

Are you able to reproduce this locally?

yes, constantly. and reverting the 933a1da6d7517b8215c0cc720e47374adedf381e helps.

ceph_test_objectstore --gtest_filter=ObjectStore/StoreTest.Synthetic/1

I've tried a few different machines now but I can't reproduce this.

Can you generate a filestore = 20 log for me?

Sage, i will try to fix this if you don't have enough bandwidth today.

i just tested on ext4 the problem disappears. and seems it is reproducible on btrfs.

http://pulpito.ceph.com/kchai-2016-11-13_07:03:13-rados-wip-kefu-testing---basic-smithi/544085/
http://pulpito.ceph.com/samuelj-2016-11-16_21:33:31-rados-wip-sam-working---basic-smithi/554921/

both of them (smithi072, smithi037) are using ext4. weird...

i am able to reproduce the above failure using "ctest -R test_objectstore_memstore.sh -V --repeat-until-fail 400".

Total Test time (real) = 10373.15 sec
The following tests FAILED:
         10 - test_objectstore_memstore.sh (Failed)
Errors while running CTest

repeating the failure as instructed by Kefu and trying to get a core but cannot (3a9bcaa4aa6042587886c0eaae0ce4eeeb8f8164)

$ ctest -R test_objectstore_memstore.sh --repeat-until-fail 4000
tore_memstore.sh --repeat-until-fail 4000
Test project /slow/loic/ceph-ubuntu-14.04-loic/build
    Start 10: test_objectstore_memstore.sh
1/1 Test #10: test_objectstore_memstore.sh .....   Passed   90.86 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  90.92 sec

https://jenkins.ceph.com/job/ceph-pull-requests/15166/console

  3/154 Test  #10: test_objectstore_memstore.sh ............***Failed   19.38 sec

...
     0> 2016-12-01 13:37:34.675656 7ff2a4f18700 -1 *** Caught signal (Aborted) **
 in thread 7ff2a4f18700 thread_name:fn_anonymous

 ceph version 11.0.2-2148-g99305be (99305be55d0e60e5f6d06aebe4f6a8c1f5832e60)
 1: (()+0x682922) [0x7ff2aa45d922]
 2: (()+0x10330) [0x7ff2a99a8330]
 3: (gsignal()+0x37) [0x7ff2a796bc37]
 4: (abort()+0x148) [0x7ff2a796f028]
 5: (()+0x2fbf6) [0x7ff2a7964bf6]
 6: (()+0x2fca2) [0x7ff2a7964ca2]
 7: (SyntheticWorkloadState::C_SyntheticOnReadable::finish(int)+0x367) [0x7ff2aa100607]
 8: (Context::complete(int)+0x9) [0x7ff2aa0ddcc9]
 9: (Finisher::finisher_thread_entry()+0x1e6) [0x7ff2aa2fd456]
 10: (()+0x8184) [0x7ff2a99a0184]
 11: (clone()+0x6d) [0x7ff2a7a2f37d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

https://jenkins.ceph.com/job/ceph-pull-requests/15371/console fails despite https://github.com/ceph/ceph/pull/12281

https://jenkins.ceph.com/job/ceph-pull-requests/16883/consoleFull#-108728127277933967-90d1-4877-8d60-89cb08ef4eb1

This failure is plaguing kraken backports - see e.g.

• https://github.com/ceph/ceph/pull/14517
• https://github.com/ceph/ceph/pull/14540

Happened on another kraken backport: https://github.com/ceph/ceph/pull/16108

https://github.com/ceph/ceph/pull/16215 ?