-9> 2013-10-28 01:08:21.613140 7fa518a02700 5 filestore(dev/osd0) queue_transactions existing osr(0.4 0x2fb8400)/0x2fb8400 -8> 2013-10-28 01:08:21.613143 7fa518a02700 2 filestore(dev/osd0) waiting 51 > 50 ops || 316098 > 104857600 -7> 2013-10-28 01:08:21.613139 7fa518001700 10 osd.0 pg_epoch: 5 pg[0.0( v 5'2297 (0'0,5'2297] local-les=5 n=2297 ec=1 les/c 5/5 4/4/4) [0] r=0 lpr=4 luod=5'2295 lcod 5'2295 mlcod 5'2295 active+degraded] new_repop mode was rmw(wr=7) -6> 2013-10-28 01:08:21.613145 7fa518001700 10 osd.0 pg_epoch: 5 pg[0.0( v 5'2297 (0'0,5'2297] local-les=5 n=2297 ec=1 les/c 5/5 4/4/4) [0] r=0 lpr=4 luod=5'2295 lcod 5'2295 mlcod 5'2295 active+degraded] new_repop mode now rmw(wr=8) (start_write) -5> 2013-10-28 01:08:21.613152 7fa518001700 7 osd.0 pg_epoch: 5 pg[0.0( v 5'2297 (0'0,5'2297] local-les=5 n=2297 ec=1 les/c 5/5 4/4/4) [0] r=0 lpr=4 luod=5'2295 lcod 5'2295 mlcod 5'2295 active+degraded] issue_repop rep_tid 17925 o c8b96078/benchmark_data_lab142.schoonerinfotech.net_7752_object17924/head//0 -4> 2013-10-28 01:08:21.613159 7fa518001700 10 osd.0 pg_epoch: 5 pg[0.0( v 5'2297 (0'0,5'2297] local-les=5 n=2297 ec=1 les/c 5/5 4/4/4) [0] r=0 lpr=4 luod=5'2295 lcod 5'2295 mlcod 5'2295 active+degraded] eval_repop repgather(0x31dabe0 5'2297 rep_tid=17925 wfack=0 wfdisk=0 op=osd_op(client.4100.0:17925 benchmark_data_lab142.schoonerinfotech.net_7752_object17924 [write 0~4096] 0.c8b96078 e5) v4) wants=ad -3> 2013-10-28 01:08:21.613167 7fa518001700 10 osd.0 pg_epoch: 5 pg[0.0( v 5'2297 (0'0,5'2297] local-les=5 n=2297 ec=1 les/c 5/5 4/4/4) [0] r=0 lpr=4 luod=5'2295 lcod 5'2295 mlcod 5'2295 active+degraded] apply_repop applying update on repgather(0x31dabe0 5'2297 rep_tid=17925 wfack=0 wfdisk=0 op=osd_op(client.4100.0:17925 benchmark_data_lab142.schoonerinfotech.net_7752_object17924 [write 0~4096] 0.c8b96078 e5) v4) -2> 2013-10-28 01:08:21.613176 7fa518001700 5 filestore(dev/osd0) queue_transactions existing osr(0.0 0x2fb8260)/0x2fb8260 -1> 2013-10-28 01:08:21.613178 7fa518001700 2 filestore(dev/osd0) waiting 51 > 50 ops || 316098 > 104857600 0> 2013-10-28 01:08:21.613400 7fa52422d700 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int)' thread 7fa52422d700 time 2013-10-28 01:08:21.612245 os/FileStore.cc: 2816: FAILED assert(0 == "unexpected error") ceph version 0.67.4-11-g299ddd3 (299ddd31b29e332dc5e76bc4f871e4769698665d) 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int)+0xc50) [0x7bf200] 2: (FileStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x71) [0x7c5df1] 3: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x276) [0x7c6086] 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x551) [0x8bb011] 5: (ThreadPool::WorkThread::entry()+0x10) [0x8be040] 6: /lib64/libpthread.so.0() [0x3ca8607851] 7: (clone()+0x6d) [0x3ca7ee890d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 25/25 osd 0/ 5 optracker 20/20 objclass 20/20 filestore 20/20 journal 1/ 1 ms 1/ 5 mon