Bug #3690
closedosd crashed in FileStore::_do_transaction
0%
Description
ceph version: 0.55.1-360-g6356739 (635673928a6b4dae6d4712cacad81cbac6412dc3)
I had a cluster[burnupi15, burnupi19, burnupi20] running on argonaut and then upgraded to bobtail and started running tests on the cluster from two different clients. from client burnupi13[bobtail version of ceph-fuse], was running fsstress.sh and from burnupi14[argonaut version of ceph-fuse], was running bonnie.sh and hit the core pasted below on 2 of the osds [osd.1 on burnupi15 and osd.4 on burnupi19],
ceph version 0.55.1-360-g6356739 (635673928a6b4dae6d4712cacad81cbac6412dc3)
1: /usr/bin/ceph-osd() [0x7839ca]
2: (()+0xfcb0) [0x7fa716788cb0]
3: (gsignal()+0x35) [0x7fa714a5c425]
4: (abort()+0x17b) [0x7fa714a5fb8b]
5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7fa7153ae69d]
6: (()+0xb5846) [0x7fa7153ac846]
7: (()+0xb5873) [0x7fa7153ac873]
8: (()+0xb596e) [0x7fa7153ac96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x82ea7f]
10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int)+0x912) [0x71a8d2]
11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x4c) [0x7209cc]
12: (FileStore::_do_op(FileStore::OpSequencer*)+0x1b1) [0x6f0d21]
13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4bc) [0x823ecc]
14: (ThreadPool::WorkThread::entry()+0x10) [0x825cd0]
15: (()+0x7e9a) [0x7fa716780e9a]
16: (clone()+0x6d) [0x7fa714b19cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.--- begin dump of recent events ---
4> 2012-12-27 17:15:41.120097 7fa6e0ed8700 1 - 10.214.134.22:6802/37265 >> :/0 pipe(0x492dd80 sd=24 :6802 pgs=0 cs=0 l=0).accept sd=24
3> 2012-12-27 17:15:41.132294 7fa6e0ed8700 1 - 10.214.134.22:6802/37265 >> :/0 pipe(0x492d240 sd=24 :6802 pgs=0 cs=0 l=0).accept sd=24
2> 2012-12-27 17:15:41.141558 7fa6e0ed8700 1 - 10.214.134.22:6802/37265 >> :/0 pipe(0x492d480 sd=24 :6802 pgs=0 cs=0 l=0).accept sd=24
-1> 2012-12-27 17:15:41.148697 7fa70cffd700 -1 ** Caught signal (Aborted) *
in thread 7fa70cffd700
ceph version 0.55.1-360-g6356739 (635673928a6b4dae6d4712cacad81cbac6412dc3)
1: /usr/bin/ceph-osd() [0x7839ca]
2: (()+0xfcb0) [0x7fa716788cb0]
3: (gsignal()+0x35) [0x7fa714a5c425]
4: (abort()+0x17b) [0x7fa714a5fb8b]
5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7fa7153ae69d]
6: (()+0xb5846) [0x7fa7153ac846]
7: (()+0xb5873) [0x7fa7153ac873]
8: (()+0xb596e) [0x7fa7153ac96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x82ea7f]
10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int)+0x912) [0x71a8d2]
11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x4c) [0x7209cc]
12: (FileStore::_do_op(FileStore::OpSequencer*)+0x1b1) [0x6f0d21]
13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4bc) [0x823ecc]
14: (ThreadPool::WorkThread::entry()+0x10) [0x825cd0]
15: (()+0x7e9a) [0x7fa716780e9a]
16: (clone()+0x6d) [0x7fa714b19cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.ceph.conf:
ubuntu@burnupi15:~$ sudo cat /etc/ceph/ceph.conf
[global]
auth client required = none
auth cluster required = none
auth service required = none
[osd]
osd journal size = 1000
filestore xattr use omap = true
[osd.1]
host = burnupi15
[osd.2]
host = burnupi15
[osd.3]
host = burnupi19
[osd.4]
host = burnupi19
[osd.5]
host = burnupi20
[osd.6]
host = burnupi20
[mon.a]
host = burnupi15
mon addr = 10.214.134.22:6789
[mon.b]
host = burnupi19
mon addr = 10.214.134.14:6789
[mon.c]
host = burnupi20
mon addr = 10.214.134.12:6789
[mds.a]
host = burnupi20
[client.radosgw.gateway]
host = burnupi15
keyring = /etc/ceph/keyring.radosgw.gateway
rgw socket path = /tmp/radosgw.sock
log file = /var/log/ceph/radosgw.log