Ceph

See #14103

Seems similar in
Run: http://pulpito.ceph.com/teuthology-2016-01-04_19:00:01-rados-jewel-distro-basic-smithi/
Job: 13535
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-01-04_19:00:01-rados-jewel-distro-basic-smithi/13535/teuthology.log

2016-01-05T17:56:00.004 INFO:tasks.ceph.osd.2.smithi017.stderr:os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f8070fb4700 time 2016-01-05 20:56:00.013636
2016-01-05T17:56:00.005 INFO:tasks.ceph.osd.2.smithi017.stderr:os/FileStore.cc: 2890: FAILED assert(0 == "unexpected error")
2016-01-05T17:56:00.020 INFO:tasks.ceph.osd.2.smithi017.stderr: ceph version 10.0.1-612-g7bcb744 (7bcb744d6b76aea3aebf065edfc231d6b5c42d2f)
2016-01-05T17:56:00.020 INFO:tasks.ceph.osd.2.smithi017.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f807f50e6d5]
2016-01-05T17:56:00.021 INFO:tasks.ceph.osd.2.smithi017.stderr: 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xa95) [0x7f807f23c455]
2016-01-05T17:56:00.021 INFO:tasks.ceph.osd.2.smithi017.stderr: 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x7f807f242184]
2016-01-05T17:56:00.021 INFO:tasks.ceph.osd.2.smithi017.stderr: 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x194) [0x7f807f242334]
2016-01-05T17:56:00.021 INFO:tasks.ceph.osd.2.smithi017.stderr: 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa7e) [0x7f807f4ffbce]
2016-01-05T17:56:00.022 INFO:tasks.ceph.osd.2.smithi017.stderr: 6: (ThreadPool::WorkThread::entry()+0x10) [0x7f807f500ab0]
2016-01-05T17:56:00.022 INFO:tasks.ceph.osd.2.smithi017.stderr: 7: (()+0x7df5) [0x7f807d553df5]
2016-01-05T17:56:00.022 INFO:tasks.ceph.osd.2.smithi017.stderr: 8: (clone()+0x6d) [0x7f807bdfc1ad]
2016-01-05T17:56:00.022 INFO:tasks.ceph.osd.2.smithi017.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

And job ['13370']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-01-04_19:00:01-rados-jewel-distro-basic-smithi/13370/teuthology.log

We are also reproducing this bug consistently. It does seems related to an osd running out of space, but the failing osd is also having drive error, as seen in kernel logs: "Buffer I/O error on device" then "Add. Sense: Unrecovered read error" and "end_request: critical medium error".

Not sure if it's related, but this ceph cluster is running a pool with only one replica too.

Anyway, once the osd fail with this exact stack trace, the whole cluster seems to be stuck until the osd is manually removed. Can the severity be raised from minor ?

Tristan- oh, you have 1 replica. Of course it will hang if an OSD fails.