Ceph

is_missing_object(snapdir)->wait_for_unreadable

2014-03-18T12:48:27.770 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3
2014-03-18T12:48:27.771 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False
2014-03-18T12:48:27.771 DEBUG:teuthology.orchestra.run:Running [10.214.132.4]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3'
2014-03-18T12:48:30.097 INFO:teuthology.task.rados.rados.0.out:[10.214.132.4]: 2602: finishing write tid 5 to plana7429837-19
2014-03-18T12:48:30.098 INFO:teuthology.task.rados.rados.0.out:[10.214.132.4]: 2602: finishing write tid 6 to plana7429837-19
2014-03-18T12:48:30.367 INFO:teuthology.task.rados.rados.0.out:[10.214.132.4]: 2602: finishing write tid 1 to plana7429837-19
2014-03-18T12:48:30.368 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: Error: finished tid 1 when last_acked_tid was 6
2014-03-18T12:48:30.508 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: ./test/osd/RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)' thread 7f738effd700 time 2014-03-18 12:48:30.366951
2014-03-18T12:48:30.508 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: ./test/osd/RadosModel.h: 811: FAILED assert(0)
2014-03-18T12:48:30.528 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: ceph version 0.77-902-g35aed15 (35aed15f9ba0b8016d191f3b35be621771053508)
2014-03-18T12:48:30.528 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: 1: (WriteOp::_finish(TestOp::CallbackInfo*)+0x318) [0x419dc8]
2014-03-18T12:48:30.528 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: 2: (write_callback(void*, void*)+0x21) [0x4274b1]
2014-03-18T12:48:30.529 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: 3: (librados::C_AioSafe::finish(int)+0x1d) [0x7f73992d553d]
2014-03-18T12:48:30.529 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: 4: (Context::complete(int)+0x9) [0x7f73992b2f89]
2014-03-18T12:48:30.529 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: 5: (Finisher::finisher_thread_entry()+0x1c0) [0x7f7399365b70]
2014-03-18T12:48:30.529 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: 6: (()+0x7e9a) [0x7f7398f0ce9a]
2014-03-18T12:48:30.530 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: 7: (clone()+0x6d) [0x7f73987233fd]
2014-03-18T12:48:30.530 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-03-18T12:48:30.530 INFO:teuthology.task.rados.rados.0.err:[10.214.132.4]: terminate called after throwing an instance of 'ceph::FailedAssertion'
2014-03-18T12:48:30.977 ERROR:teuthology.run_tasks:Manager failed: rados
Traceback (most recent call last):
File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 84, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 24, in exit
self.gen.next()
File "/home/teuthworker/teuthology-firefly/teuthology/task/rados.py", line 170, in task
running.get()
File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 331, in get
raise self._exception
CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --ec-pool --op read 100 --op write 0 --op delete 50 --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op copy_from 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op append 100 --pool unique_pool_0'