Ceph

This occurred on wip_bobtail_f in a local vstart 11-osd cluster which I was trying to use to reproduce #4052 by causing some thrashing and some osds to stay down for a while while there were some rados benches and several unused pools.

2013-02-08 12:16:30.453731 7f79222ee780 10 journal op_apply_finish 1758 open_ops 1 -> 0, max_applied_seq 1757 -> 1758
2013-02-08 12:16:30.453735 7f79222ee780  3 journal journal_replay: r = 0, op_seq now 1758
2013-02-08 12:16:30.453747 7f79222ee780  2 journal read_entry 100585472 : seq 1759 263 bytes
2013-02-08 12:16:30.453750 7f79222ee780  3 journal journal_replay: applying op seq 1759
2013-02-08 12:16:30.453760 7f79222ee780 10 journal op_apply_start 1759 open_ops 0 -> 1
2013-02-08 12:16:30.453763 7f79222ee780 10 filestore(dev/osd2) _do_transaction on 0x3463a20
2013-02-08 12:16:30.453827 7f79222ee780 10 filestore(dev/osd2) _check_replay_guard object has 1892.0.9 > current pos 1759.0.0, now or in future, SKIPPING REPLAY
2013-02-08 12:16:30.453890 7f79222ee780 10 filestore(dev/osd2) _check_replay_guard object has 1895.0.8 > current pos 1759.0.1, now or in future, SKIPPING REPLAY
2013-02-08 12:16:30.453911 7f79222ee780 10 filestore(dev/osd2) _check_replay_guard object has 1753.0.0 < current pos 1759.0.2, in past, will replay
2013-02-08 12:16:30.454098 7f79222ee780  0 filestore(dev/osd2)  error (39) Directory not empty not handled on operation 21 (1759.0.2, or op 2, counting from 0)
2013-02-08 12:16:30.454111 7f79222ee780  0 filestore(dev/osd2) ENOTEMPTY suggests garbage data in osd data dir
2013-02-08 12:16:30.454112 7f79222ee780  0 filestore(dev/osd2)  transaction dump:
{ "ops": [
        { "op_num": 0,
          "op_name": "remove",
          "collection": "FORREMOVAL_15_8.21",
          "oid": "b0c75721\/benchmark_data_vit_17179_object70\/head\/\/8"},
        { "op_num": 1,
          "op_name": "remove",
          "collection": "FORREMOVAL_15_8.21",
          "oid": "9327c821\/benchmark_data_vit_17179_object79\/head\/\/8"},
        { "op_num": 2,
          "op_name": "rmcoll",
          "collection": "FORREMOVAL_15_8.21"}]}
2013-02-08 12:16:30.461801 7f79222ee780 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int)' thread 7f79222
ee780 time 2013-02-08 12:16:30.454158
os/FileStore.cc: 2667: FAILED assert(0 == "unexpected error")

 ceph version 0.56-663-g46d7dbd (46d7dbd3472f26926c6d048bfc3c150074bfd283)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xf86351]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int)+0x3e90) [0xee971a]
 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0xfc) [0xee3b56]
 4: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x31) [0xf036bf]
 5: (JournalingObjectStore::journal_replay(unsigned long)+0x97c) [0xf11520]
 6: (FileStore::mount()+0x519f) [0xee07eb]
 7: (OSD::do_convertfs(ObjectStore*)+0x32) [0xd0576e]
 8: (OSD::convertfs(std::string const&, std::string const&)+0x87) [0xd067d3]
 9: (main()+0x3869) [0xc2efc5]
 10: (__libc_start_main()+0xfd) [0x7f792008dc4d]
 11: ./ceph-osd() [0xc2b459]

Full log is attached.