Ceph

Test run on 36 OSD configuration using 0.93-982-g6678e12. Several OSDs failed the same way:

2015-04-01 14:35:45.091984 7effb060f700 -1 common/Mutex.cc: In function 'Mutex::~Mutex()' thread 7effb060f700 time 2015-04-01 14:35:45.087190
common/Mutex.cc: 71: FAILED assert(nlock == 0)

 ceph version 0.93-982-g6678e12 (6678e120aaf031115343be9cd0b233f21af31af3)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xcabea5]
 2: /usr/bin/ceph-osd() [0xc5aba6]
 3: (NewStore::OpSequencer::~OpSequencer()+0xa4) [0xa37c34]
 4: (std::tr1::_Sp_counted_base_impl<ObjectStore::Sequencer*, SharedPtrRegistry<spg_t, ObjectStore::Sequencer>::OnRemoval, (__gnu_cxx::_Lock_policy)2>::_M_disp
ose()+0x7b) [0x867e5b]
 5: (std::tr1::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x39) [0x7184e9]
 6: (PG::~PG()+0xa6) [0x811f86]
 7: (ReplicatedPG::~ReplicatedPG()+0x9) [0x9256d9]
 8: (intrusive_ptr_release(PG*)+0x3d) [0x804e0d]
 9: (ObjectStore::C_DeleteTransactionHolder<boost::intrusive_ptr<PG> >::~C_DeleteTransactionHolder()+0x20) [0x708d20]
 10: (std::tr1::_Sp_counted_base_impl<RunOnDelete*, std::tr1::_Sp_deleter<RunOnDelete>, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x1a) [0x7aca6a]
 11: (std::tr1::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x39) [0x7184e9]
 12: (Wrapper<std::tr1::shared_ptr<RunOnDelete> >::~Wrapper()+0x20) [0x7acdf0]
 13: (Finisher::finisher_thread_entry()+0x168) [0xbd10a8]
 14: (()+0x7ee5) [0x7effbafb4ee5]
 15: (clone()+0x6d) [0x7effb9a92b8d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
 -1907> 2015-04-01 14:35:30.994525 7effbc860980  5 asok(0x4758000) register_command perfcounters_dump hook 0x46b8050
 -1906> 2015-04-01 14:35:30.994600 7effbc860980  5 asok(0x4758000) register_command 1 hook 0x46b8050
 -1905> 2015-04-01 14:35:30.994611 7effbc860980  5 asok(0x4758000) register_command perf dump hook 0x46b8050
 -1904> 2015-04-01 14:35:30.994621 7effbc860980  5 asok(0x4758000) register_command perfcounters_schema hook 0x46b8050
 -1903> 2015-04-01 14:35:30.994629 7effbc860980  5 asok(0x4758000) register_command 2 hook 0x46b8050
 -1902> 2015-04-01 14:35:30.994633 7effbc860980  5 asok(0x4758000) register_command perf schema hook 0x46b8050
 -1901> 2015-04-01 14:35:30.994637 7effbc860980  5 asok(0x4758000) register_command perf reset hook 0x46b8050
 -1900> 2015-04-01 14:35:30.994641 7effbc860980  5 asok(0x4758000) register_command config show hook 0x46b8050
 -1899> 2015-04-01 14:35:30.994645 7effbc860980  5 asok(0x4758000) register_command config set hook 0x46b8050
 -1898> 2015-04-01 14:35:30.994649 7effbc860980  5 asok(0x4758000) register_command config get hook 0x46b8050
 -1897> 2015-04-01 14:35:30.994657 7effbc860980  5 asok(0x4758000) register_command config diff hook 0x46b8050
 -1896> 2015-04-01 14:35:30.994663 7effbc860980  5 asok(0x4758000) register_command log flush hook 0x46b8050
 -1895> 2015-04-01 14:35:30.994669 7effbc860980  5 asok(0x4758000) register_command log dump hook 0x46b8050
 -1894> 2015-04-01 14:35:30.994673 7effbc860980  5 asok(0x4758000) register_command log reopen hook 0x46b8050
 -1893> 2015-04-01 14:35:30.998897 7effbc860980 -1 WARNING: the following dangerous and experimental features are enabled: newstore,rocksdb
 -1892> 2015-04-01 14:35:30.999033 7effbc860980 -1 WARNING: the following dangerous and experimental features are enabled: newstore,rocksdb
 -1891> 2015-04-01 14:35:30.999312 7effbc860980  0 ceph version 0.93-982-g6678e12 (6678e120aaf031115343be9cd0b233f21af31af3), process ceph-osd, pid 1363
 -1890> 2015-04-01 14:35:30.999340 7effbc860980 -1 WARNING: experimental feature 'newstore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

 -1889> 2015-04-01 14:35:31.002798 7effbc860980  1 finished global_init_daemonize
 -1888> 2015-04-01 14:35:31.005760 7effbc860980 -1 WARNING: the following dangerous and experimental features are enabled: newstore,rocksdb
 -1887> 2015-04-01 14:35:31.020462 7effbc860980  1 newstore(/tmp/cbt/mnt/osd-device-0-data) mount path /tmp/cbt/mnt/osd-device-0-data
 -1886> 2015-04-01 14:35:31.020620 7effbc860980 -1 WARNING: experimental feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

Full OSD log attached.