client: failed xlist assert on put_inode()
teuthology archives: /a/teuthology-2013-09-11_01:30:firstname.lastname@example.org/log/ceph-client.0.4245.log
Blogbench, update and reboot the servers, run fsstress, and it crashed with
0> 2013-09-11 08:23:19.091279 7f30e95d0700 -1 ./include/xlist.h: In function 'xlist<T>::~xlist() [with T = ObjectCacher::Object*]' thread 7f30e95d0700 time 2013-09-11 08:23:19.089433 ./include/xlist.h: 69: FAILED assert(_size == 0) ceph version 0.61.8-27-gb16f812 (b16f812362ccb1d9bdd4900d155e248d695ef0d7) 1: (Inode::~Inode()+0x28a) [0x4d44ca] 2: (Client::put_inode(Inode*, int)+0x2c7) [0x48a697] 3: (Context::complete(int)+0xa) [0x4cbe7a] 4: (C_Gather::sub_finish(Context*, int)+0x117) [0x4d1497] 5: (C_Gather::C_GatherSub::finish(int)+0x12) [0x4d1662] 6: (Context::complete(int)+0xa) [0x4cbe7a] 7: (finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x95) [0x600255] 8: (ObjectCacher::bh_write_commit(long, sobject_t, long, unsigned long, unsigned long, int)+0x59f) [0x5fc95f] 9: (ObjectCacher::C_WriteCommit::finish(int)+0x6b) [0x60379b] 10: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe55) [0x60f0c5] 11: (Client::ms_dispatch(Message*)+0x8b) [0x4cb89b] 12: (DispatchQueue::entry()+0x3f1) [0x646f81] 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x5860cd] 14: (()+0x7e9a) [0x7f30eeb32e9a] 15: (clone()+0x6d) [0x7f30ed2e8ccd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
I assume the xlist in question is the xlist of Objects in the oset associated with the Inode (at least, I can't find any other xlists that get destroyed via the Inode destructor). Perhaps we've got an issue with the ObjectCacher write holding the last reference to the Inode, and then when the write is completed the Client drops the Inode, but the ObjectCacher hasn't dropped its own references yet? The bit of log we have doesn't contradict this, but I'm not very familiar with the ObjectCacher right now.