Project

General

Profile

Actions
Copy link

Bug #6761

closed

emperor's "dirty" flag is being interpreted as "lost" by Dumpling OSDs

Added by Corin Langosch over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Samuel Just
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

All my systems run ubuntu 12.10. I was running dumpling for a few months without any errors. My kvm guests use qemu-rbd and run on the same machines as the mons and osds.

I just upgraded all my monitors (3) and one osd (out of 14) to emporer. The cluster is healthy and seems to be running fine. A few minutes after upgrading a few of my qemu (kvm) machines just died. There are no core dumps, no logs.

When I start one of the died kvm machines using the command line it dies again after a few seconds (using vnc i can see that linux boots up in the vm and then the vnc disconnects). On the command line I get http://pastie.org/8477535 This clearly looks like a ceph bug.

I already tried to pass "cache=none" but it doesn't help.

Another thing I noticed: I restarted the one upgraded osd to see if it's related to it. When the upgraded osd was down (not out, no rebalance active!) the kvm guest still didn't startup. I then started the osd again. Just a few moments later I discovered another already running kvm guest died (one which died during the upgrade before but which I was able to restart).

I relly hope someone can help me, as it's a production cluster and I need to get the virtual machines running again asap.

Thank you!

Files

Download all files
export.txt (5.07 MB) export.txt Corin Langosch, 11/13/2013 09:55 AM
data.tar.gz (45.6 MB) data.tar.gz Corin Langosch, 11/13/2013 12:24 PM
Actions
Copy link
 #1

Updated by Corin Langosch over 10 years ago

  1. gdb --args /opt/qemu/1.4.0/bin/qemu-system-x86_64 -smp sockets=1,cores=2 -m 512 -vga cirrus -drive id=drive14183,if=none,cache=none,aio=native,format=raw,media=disk,file=rbd:numatrix-ssd/3a9294d9-6408-4415-8bd4-2dc143d5b10e -device virtio-blk-pci,id=hdrive14183,addr=0x12,drive=drive14183
    GNU gdb (GDB) 7.5-ubuntu
    Copyright (C) 2012 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law. Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/&gt;...
    Reading symbols from /opt/qemu/1.4.0/bin/qemu-system-x86_64...(no debugging symbols found)...done.
    (gdb) handle SIGUSR1 noprin
    Signal Stop Print Pass to program Description
    SIGUSR1 No No Yes User defined signal 1
    (gdb) run
    Starting program: /opt/qemu/1.4.0/bin/qemu-system-x86_64 -smp sockets=1,cores=2 -m 512 -vga cirrus -drive id=drive14183,if=none,cache=none,aio=native,format=raw,media=disk,file=rbd:numatrix-ssd/3a9294d9-6408-4415-8bd4-2dc143d5b10e -device virtio-blk-pci,id=hdrive14183,addr=0x12,drive=drive14183
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    [New Thread 0x7fffdf19e700 (LWP 14046)]
    [New Thread 0x7fffde4f8700 (LWP 14047)]
    [New Thread 0x7fffddcf7700 (LWP 14048)]
    [New Thread 0x7fffdd4f6700 (LWP 14049)]
    [New Thread 0x7fffdccf5700 (LWP 14050)]
    [New Thread 0x7fffdc4f4700 (LWP 14051)]
    [New Thread 0x7ffff7fdc700 (LWP 14052)]
    [New Thread 0x7fffdbcf3700 (LWP 14055)]
    [New Thread 0x7fffcbfff700 (LWP 14056)]
    [New Thread 0x7fffcb7fe700 (LWP 14057)]
    [New Thread 0x7fffcaffd700 (LWP 14058)]
    [New Thread 0x7fffca7fc700 (LWP 14059)]
    [New Thread 0x7fffdbbf2700 (LWP 14060)]
    [New Thread 0x7fffdbaf1700 (LWP 14061)]
    [New Thread 0x7fffdb9f0700 (LWP 14062)]
    [New Thread 0x7fffdb8ef700 (LWP 14063)]
    [New Thread 0x7fffdb7ee700 (LWP 14064)]
    [New Thread 0x7fffdb6ed700 (LWP 14065)]
    [New Thread 0x7fffdb5ec700 (LWP 14066)]
    [New Thread 0x7fffdacea700 (LWP 14067)]
    [New Thread 0x7fffd989a700 (LWP 14068)]
    VNC server running on `127.0.0.1:5900'
    [New Thread 0x7fffd8d58700 (LWP 14069)]
    [New Thread 0x7fffd8c57700 (LWP 14070)]
    [New Thread 0x7fffd8b56700 (LWP 14071)]
    [New Thread 0x7fffd8a55700 (LWP 14074)]
    [Thread 0x7fffdacea700 (LWP 14067) exited]
    terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
    what(): buffer::end_of_buffer

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffcb7fe700 (LWP 14057)]
0x00007ffff4501425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff4501425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff4504b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff3febe2d in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff3fe9f26 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff3fe9f53 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff3fea17e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff673e17f in ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned int, unsigned int) () from /usr/lib/librados.so.2
#7 0x00007ffff747a665 in ObjectCacher::_readx(ObjectCacher::OSDRead*, ObjectCacher::ObjectSet*, Context*, bool) () from /usr/lib/librbd.so.1
#8 0x00007ffff7482e10 in ObjectCacher::C_RetryRead::finish(int) () from /usr/lib/librbd.so.1
#9 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#10 0x00007ffff747dd64 in finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int) () from /usr/lib/librbd.so.1
#11 0x00007ffff74770b8 in ObjectCacher::bh_read_finish(long, sobject_t, unsigned long, long, unsigned long, ceph::buffer::list&, int, bool) () from /usr/lib/librbd.so.1
#12 0x00007ffff7481f5d in ObjectCacher::C_ReadFinish::finish(int) () from /usr/lib/librbd.so.1
#13 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#14 0x00007ffff7443b15 in librbd::C_Request::finish(int) () from /usr/lib/librbd.so.1
#15 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#16 0x00007ffff6696e5d in librados::C_AioComplete::finish(int) () from /usr/lib/librados.so.2
#17 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#18 0x00007ffff6723528 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2
#19 0x00007ffff4892e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007ffff45bf3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()
(gdb)

Actions
Copy link
 #2

Updated by Corin Langosch over 10 years ago

Same but nicely formatted:

root@r-ch105:/var/log# gdb --args /opt/qemu/1.4.0/bin/qemu-system-x86_64 -smp sockets=1,cores=2 -m 512 -vga cirrus -drive id=drive14183,if=none,cache=none,aio=native,format=raw,media=disk,file=rbd:numatrix-ssd/3a9294d9-6408-4415-8bd4-2dc143d5b10e -device virtio-blk-pci,id=hdrive14183,addr=0x12,drive=drive14183
GNU gdb (GDB) 7.5-ubuntu
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying" 
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/qemu/1.4.0/bin/qemu-system-x86_64...(no debugging symbols found)...done.
(gdb) handle SIGUSR1 noprin
Signal        Stop    Print    Pass to program    Description
SIGUSR1       No    No    Yes        User defined signal 1
(gdb) run
Starting program: /opt/qemu/1.4.0/bin/qemu-system-x86_64 -smp sockets=1,cores=2 -m 512 -vga cirrus -drive id=drive14183,if=none,cache=none,aio=native,format=raw,media=disk,file=rbd:numatrix-ssd/3a9294d9-6408-4415-8bd4-2dc143d5b10e -device virtio-blk-pci,id=hdrive14183,addr=0x12,drive=drive14183
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffdf19e700 (LWP 14046)]
[New Thread 0x7fffde4f8700 (LWP 14047)]
[New Thread 0x7fffddcf7700 (LWP 14048)]
[New Thread 0x7fffdd4f6700 (LWP 14049)]
[New Thread 0x7fffdccf5700 (LWP 14050)]
[New Thread 0x7fffdc4f4700 (LWP 14051)]
[New Thread 0x7ffff7fdc700 (LWP 14052)]
[New Thread 0x7fffdbcf3700 (LWP 14055)]
[New Thread 0x7fffcbfff700 (LWP 14056)]
[New Thread 0x7fffcb7fe700 (LWP 14057)]
[New Thread 0x7fffcaffd700 (LWP 14058)]
[New Thread 0x7fffca7fc700 (LWP 14059)]
[New Thread 0x7fffdbbf2700 (LWP 14060)]
[New Thread 0x7fffdbaf1700 (LWP 14061)]
[New Thread 0x7fffdb9f0700 (LWP 14062)]
[New Thread 0x7fffdb8ef700 (LWP 14063)]
[New Thread 0x7fffdb7ee700 (LWP 14064)]
[New Thread 0x7fffdb6ed700 (LWP 14065)]
[New Thread 0x7fffdb5ec700 (LWP 14066)]
[New Thread 0x7fffdacea700 (LWP 14067)]
[New Thread 0x7fffd989a700 (LWP 14068)]
VNC server running on `127.0.0.1:5900'
[New Thread 0x7fffd8d58700 (LWP 14069)]
[New Thread 0x7fffd8c57700 (LWP 14070)]
[New Thread 0x7fffd8b56700 (LWP 14071)]
[New Thread 0x7fffd8a55700 (LWP 14074)]
[Thread 0x7fffdacea700 (LWP 14067) exited]
terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffcb7fe700 (LWP 14057)]
0x00007ffff4501425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff4501425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff4504b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff3febe2d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff3fe9f26 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff3fe9f53 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff3fea17e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff673e17f in ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned int, unsigned int) () from /usr/lib/librados.so.2
#7  0x00007ffff747a665 in ObjectCacher::_readx(ObjectCacher::OSDRead*, ObjectCacher::ObjectSet*, Context*, bool) () from /usr/lib/librbd.so.1
#8  0x00007ffff7482e10 in ObjectCacher::C_RetryRead::finish(int) () from /usr/lib/librbd.so.1
#9  0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#10 0x00007ffff747dd64 in finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int) () from /usr/lib/librbd.so.1
#11 0x00007ffff74770b8 in ObjectCacher::bh_read_finish(long, sobject_t, unsigned long, long, unsigned long, ceph::buffer::list&, int, bool) () from /usr/lib/librbd.so.1
#12 0x00007ffff7481f5d in ObjectCacher::C_ReadFinish::finish(int) () from /usr/lib/librbd.so.1
#13 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#14 0x00007ffff7443b15 in librbd::C_Request::finish(int) () from /usr/lib/librbd.so.1
#15 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#16 0x00007ffff6696e5d in librados::C_AioComplete::finish(int) () from /usr/lib/librados.so.2
#17 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#18 0x00007ffff6723528 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2
#19 0x00007ffff4892e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007ffff45bf3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()
Actions
Copy link
 #3

Updated by Corin Langosch over 10 years ago

This is the backtrace when I try to start from another rbd image. Looks the same to me.

root@r-ch105:/var/log# gdb --args /opt/qemu/1.4.0/bin/qemu-system-x86_64 -smp sockets=1,cores=2 -m 512 -vga cirrus -drive id=drive14183,if=none,cache=none,aio=native,format=raw,media=disk,file=rbd:numatrix-ssd/65d7e585-29bc-4bc7-8241-77f9bc4d5263 -device virtio-blk-pci,id=hdrive14183,addr=0x12,drive=drive14183
GNU gdb (GDB) 7.5-ubuntu
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying" 
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/qemu/1.4.0/bin/qemu-system-x86_64...(no debugging symbols found)...done.
(gdb) handle SIGUSR1 noprin
Signal        Stop    Print    Pass to program    Description
SIGUSR1       No    No    Yes        User defined signal 1
(gdb) run
Starting program: /opt/qemu/1.4.0/bin/qemu-system-x86_64 -smp sockets=1,cores=2 -m 512 -vga cirrus -drive id=drive14183,if=none,cache=none,aio=native,format=raw,media=disk,file=rbd:numatrix-ssd/65d7e585-29bc-4bc7-8241-77f9bc4d5263 -device virtio-blk-pci,id=hdrive14183,addr=0x12,drive=drive14183
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffdf19e700 (LWP 14466)]
[New Thread 0x7fffde4f8700 (LWP 14467)]
[New Thread 0x7fffddcf7700 (LWP 14468)]
[New Thread 0x7fffdd4f6700 (LWP 14469)]
[New Thread 0x7fffdccf5700 (LWP 14470)]
[New Thread 0x7fffdc4f4700 (LWP 14471)]
[New Thread 0x7ffff7fdc700 (LWP 14472)]
[New Thread 0x7fffdbcf3700 (LWP 14475)]
[New Thread 0x7fffcbfff700 (LWP 14476)]
[New Thread 0x7fffcb7fe700 (LWP 14477)]
[New Thread 0x7fffcaffd700 (LWP 14478)]
[New Thread 0x7fffca7fc700 (LWP 14479)]
[New Thread 0x7fffdbbf2700 (LWP 14480)]
[New Thread 0x7fffdbaf1700 (LWP 14481)]
[New Thread 0x7fffdb9f0700 (LWP 14482)]
[New Thread 0x7fffdb8ef700 (LWP 14483)]
[New Thread 0x7fffdb7ee700 (LWP 14484)]
[New Thread 0x7fffdaeec700 (LWP 14485)]
[New Thread 0x7fffd9ae2700 (LWP 14486)]
[New Thread 0x7fffd99e1700 (LWP 14487)]
[New Thread 0x7fffd989a700 (LWP 14488)]
VNC server running on `127.0.0.1:5900'
[New Thread 0x7fffd8d5f700 (LWP 14489)]
[New Thread 0x7fffd8c5e700 (LWP 14490)]
[New Thread 0x7fffd8b5d700 (LWP 14491)]
[New Thread 0x7fffd8a5c700 (LWP 14494)]
[Thread 0x7fffdaeec700 (LWP 14485) exited]
terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffcb7fe700 (LWP 14477)]
0x00007ffff4501425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff4501425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff4504b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff3febe2d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff3fe9f26 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff3fe9f53 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff3fea17e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff673e17f in ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned int, unsigned int) () from /usr/lib/librados.so.2
#7  0x00007ffff747a665 in ObjectCacher::_readx(ObjectCacher::OSDRead*, ObjectCacher::ObjectSet*, Context*, bool) () from /usr/lib/librbd.so.1
#8  0x00007ffff7482e10 in ObjectCacher::C_RetryRead::finish(int) () from /usr/lib/librbd.so.1
#9  0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#10 0x00007ffff747dd64 in finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int) () from /usr/lib/librbd.so.1
#11 0x00007ffff74770b8 in ObjectCacher::bh_read_finish(long, sobject_t, unsigned long, long, unsigned long, ceph::buffer::list&, int, bool) () from /usr/lib/librbd.so.1
#12 0x00007ffff7481f5d in ObjectCacher::C_ReadFinish::finish(int) () from /usr/lib/librbd.so.1
#13 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#14 0x00007ffff7443b15 in librbd::C_Request::finish(int) () from /usr/lib/librbd.so.1
#15 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#16 0x00007ffff6696e5d in librados::C_AioComplete::finish(int) () from /usr/lib/librados.so.2
#17 0x00007ffff741c459 in Context::complete(int) () from /usr/lib/librbd.so.1
#18 0x00007ffff6723528 in Finisher::finisher_thread_entry() () from /usr/lib/librados.so.2
#19 0x00007ffff4892e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007ffff45bf3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()
(gdb) continue
Continuing.
[Thread 0x7fffd99e1700 (LWP 14487) exited]
[Thread 0x7fffd8b5d700 (LWP 14491) exited]
[Thread 0x7fffd989a700 (LWP 14488) exited]
[Thread 0x7fffd8a5c700 (LWP 14494) exited]
[Thread 0x7fffd8c5e700 (LWP 14490) exited]
[Thread 0x7fffd8d5f700 (LWP 14489) exited]
[Thread 0x7fffd9ae2700 (LWP 14486) exited]
[Thread 0x7fffdb7ee700 (LWP 14484) exited]
[Thread 0x7fffdb8ef700 (LWP 14483) exited]
[Thread 0x7fffdb9f0700 (LWP 14482) exited]
[Thread 0x7fffdbaf1700 (LWP 14481) exited]
[Thread 0x7fffdbbf2700 (LWP 14480) exited]
[Thread 0x7fffca7fc700 (LWP 14479) exited]
[Thread 0x7fffcaffd700 (LWP 14478) exited]
[Thread 0x7fffcb7fe700 (LWP 14477) exited]
[Thread 0x7fffcbfff700 (LWP 14476) exited]
[Thread 0x7fffdbcf3700 (LWP 14475) exited]
[Thread 0x7ffff7fdc700 (LWP 14472) exited]
[Thread 0x7fffdc4f4700 (LWP 14471) exited]
[Thread 0x7fffdd4f6700 (LWP 14469) exited]
[Thread 0x7fffddcf7700 (LWP 14468) exited]
[Thread 0x7fffde4f8700 (LWP 14467) exited]
[Thread 0x7fffdf19e700 (LWP 14466) exited]
[Thread 0x7fffee9a6940 (LWP 14463) exited]

Program terminated with signal SIGABRT, Aborted.
The program no longer exists.
Actions
Copy link
 #4

Updated by Corin Langosch over 10 years ago

I just tried to export the image a file so I can boot the vm using that. But it fails to:

root@r-ch105:/xfs-drive1/a# rbd export numatrix-ssd/65d7e585-29bc-4bc7-8241-77f9bc4d5263 test.img
Exporting image: 60% complete...terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 7f1429340700
 ceph version 0.72 (5832e2603c7db5d40b433d0953408993a9b7c217)
 1: rbd() [0x41d200]
 2: (()+0xfcb0) [0x7f142f78fcb0]
 3: (gsignal()+0x35) [0x7f142eed6425]
 4: (abort()+0x17b) [0x7f142eed9b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f142f4d8e2d]
 6: (()+0x5ef26) [0x7f142f4d6f26]
 7: (()+0x5ef53) [0x7f142f4d6f53]
 8: (()+0x5f17e) [0x7f142f4d717e]
 9: (ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned int, unsigned int)+0x24f) [0x7f142fcb817f]
 10: (ObjectCacher::_readx(ObjectCacher::OSDRead*, ObjectCacher::ObjectSet*, Context*, bool)+0x1545) [0x7f14309f6665]
 11: (ObjectCacher::C_RetryRead::finish(int)+0x20) [0x7f14309fee10]
 12: (Context::complete(int)+0x9) [0x7f1430998459]
 13: (finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0xa4) [0x7f14309f9d64]
 14: (ObjectCacher::bh_read_finish(long, sobject_t, unsigned long, long, unsigned long, ceph::buffer::list&, int, bool)+0x798) [0x7f14309f30b8]
 15: (ObjectCacher::C_ReadFinish::finish(int)+0x7d) [0x7f14309fdf5d]
 16: (Context::complete(int)+0x9) [0x7f1430998459]
 17: (librbd::C_Request::finish(int)+0x85) [0x7f14309bfb15]
 18: (Context::complete(int)+0x9) [0x7f1430998459]
 19: (librados::C_AioComplete::finish(int)+0x1d) [0x7f142fc10e5d]
 20: (Context::complete(int)+0x9) [0x7f1430998459]
 21: (Finisher::finisher_thread_entry()+0x1c8) [0x7f142fc9d528]
 22: (()+0x7e9a) [0x7f142f787e9a]
 23: (clone()+0x6d) [0x7f142ef943fd]
2013-11-13 16:42:30.080066 7f1429340700 -1 *** Caught signal (Aborted) **
 in thread 7f1429340700

 ceph version 0.72 (5832e2603c7db5d40b433d0953408993a9b7c217)
 1: rbd() [0x41d200]
 2: (()+0xfcb0) [0x7f142f78fcb0]
 3: (gsignal()+0x35) [0x7f142eed6425]
 4: (abort()+0x17b) [0x7f142eed9b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f142f4d8e2d]
 6: (()+0x5ef26) [0x7f142f4d6f26]
 7: (()+0x5ef53) [0x7f142f4d6f53]
 8: (()+0x5f17e) [0x7f142f4d717e]
 9: (ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned int, unsigned int)+0x24f) [0x7f142fcb817f]
 10: (ObjectCacher::_readx(ObjectCacher::OSDRead*, ObjectCacher::ObjectSet*, Context*, bool)+0x1545) [0x7f14309f6665]
 11: (ObjectCacher::C_RetryRead::finish(int)+0x20) [0x7f14309fee10]
 12: (Context::complete(int)+0x9) [0x7f1430998459]
 13: (finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0xa4) [0x7f14309f9d64]
 14: (ObjectCacher::bh_read_finish(long, sobject_t, unsigned long, long, unsigned long, ceph::buffer::list&, int, bool)+0x798) [0x7f14309f30b8]
 15: (ObjectCacher::C_ReadFinish::finish(int)+0x7d) [0x7f14309fdf5d]
 16: (Context::complete(int)+0x9) [0x7f1430998459]
 17: (librbd::C_Request::finish(int)+0x85) [0x7f14309bfb15]
 18: (Context::complete(int)+0x9) [0x7f1430998459]
 19: (librados::C_AioComplete::finish(int)+0x1d) [0x7f142fc10e5d]
 20: (Context::complete(int)+0x9) [0x7f1430998459]
 21: (Finisher::finisher_thread_entry()+0x1c8) [0x7f142fc9d528]
 22: (()+0x7e9a) [0x7f142f787e9a]
 23: (clone()+0x6d) [0x7f142ef943fd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -500> 2013-11-13 16:42:08.958603 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1305 rbd_data.296683d1b58ba.0000000000000512 [read 0~4194304] 5.fa23c925 e5250) v4 -- ?+0 0x11c07a0 con 0x11a7940
  -499> 2013-11-13 16:42:09.024519 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 275 ==== osd_op_reply(1305 rbd_data.296683d1b58ba.0000000000000512 [read 0~4194304] v5236'26520 uv26520 ondisk = 0) v4 ==== 138+0+4194304 (4259892484 0 162970265) 0x7f13fc0009f0 con 0x11a7940
  -498> 2013-11-13 16:42:09.032179 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1306 rbd_data.296683d1b58ba.0000000000000513 [read 0~4194304] 5.1e352112 e5250) v4 -- ?+0 0x11c07a0 con 0x11a12f0
  -497> 2013-11-13 16:42:09.098829 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 290 ==== osd_op_reply(1306 rbd_data.296683d1b58ba.0000000000000513 [read 0~4194304] v5236'55036 uv55036 ondisk = 0) v4 ==== 138+0+4194304 (4070628570 0 3134788234) 0x7f14040011c0 con 0x11a12f0
  -496> 2013-11-13 16:42:09.104416 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1307 rbd_data.296683d1b58ba.0000000000000514 [read 0~4194304] 5.c0b1b68b e5250) v4 -- ?+0 0x11c07a0 con 0x11a6460
  -495> 2013-11-13 16:42:09.169679 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 244 ==== osd_op_reply(1307 rbd_data.296683d1b58ba.0000000000000514 [read 0~4194304] v5236'14242 uv14242 ondisk = 0) v4 ==== 138+0+4194304 (2445661948 0 2216436031) 0x7f13f0000ad0 con 0x11a6460
  -494> 2013-11-13 16:42:09.175670 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1308 rbd_data.296683d1b58ba.0000000000000515 [read 0~4194304] 5.e894390 e5250) v4 -- ?+0 0x11a6ad0 con 0x11a12f0
  -493> 2013-11-13 16:42:09.243315 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 291 ==== osd_op_reply(1308 rbd_data.296683d1b58ba.0000000000000515 [read 0~4194304] v5236'1182305 uv1182305 ondisk = 0) v4 ==== 138+0+4194304 (1150204718 0 4159557935) 0x7f14040011c0 con 0x11a12f0
  -492> 2013-11-13 16:42:09.249644 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1309 rbd_data.296683d1b58ba.0000000000000516 [read 0~4194304] 5.c448ea6d e5250) v4 -- ?+0 0x11a6ad0 con 0x11a6460
  -491> 2013-11-13 16:42:09.318865 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 245 ==== osd_op_reply(1309 rbd_data.296683d1b58ba.0000000000000516 [read 0~4194304] v5236'1169215 uv1169215 ondisk = 0) v4 ==== 138+0+4194304 (3503801598 0 2266876311) 0x7f13f0000a90 con 0x11a6460
  -490> 2013-11-13 16:42:09.326655 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1310 rbd_data.296683d1b58ba.0000000000000517 [read 0~4194304] 5.4d4cad94 e5250) v4 -- ?+0 0x11c07a0 con 0x11a12f0
  -489> 2013-11-13 16:42:09.394641 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 292 ==== osd_op_reply(1310 rbd_data.296683d1b58ba.0000000000000517 [read 0~4194304] v5236'291388 uv291388 ondisk = 0) v4 ==== 138+0+4194304 (1361511243 0 1385507238) 0x7f14040011c0 con 0x11a12f0
  -488> 2013-11-13 16:42:09.403630 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1311 rbd_data.296683d1b58ba.0000000000000518 [read 0~4194304] 5.4a8877b1 e5250) v4 -- ?+0 0x11b0bc0 con 0x11a5ae0
  -487> 2013-11-13 16:42:10.181098 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 241 ==== osd_op_reply(1311 rbd_data.296683d1b58ba.0000000000000518 [read 0~4194304] v0'0 uv97847 ondisk = 0) v6 ==== 198+0+4194304 (1965919162 0 923620778) 0x7f13f8000ba0 con 0x11a5ae0
  -486> 2013-11-13 16:42:10.190150 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1312 rbd_data.296683d1b58ba.0000000000000519 [read 0~4194304] 5.a034a5d1 e5250) v4 -- ?+0 0x11a4950 con 0x11a7940
  -485> 2013-11-13 16:42:10.257972 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 276 ==== osd_op_reply(1312 rbd_data.296683d1b58ba.0000000000000519 [read 0~4194304] v5236'30836 uv30836 ondisk = 0) v4 ==== 138+0+4194304 (751015828 0 2211394779) 0x7f13fc0009f0 con 0x11a7940
  -484> 2013-11-13 16:42:10.266236 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1313 rbd_data.296683d1b58ba.000000000000051a [read 0~4194304] 5.88fadf96 e5250) v4 -- ?+0 0x11a6ad0 con 0x11a6460
  -483> 2013-11-13 16:42:10.323903 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 246 ==== osd_op_reply(1313 rbd_data.296683d1b58ba.000000000000051a [read 0~4194304] v5236'16310 uv16310 ondisk = 0) v4 ==== 138+0+4194304 (2470072296 0 3861427743) 0x7f13f00009f0 con 0x11a6460
  -482> 2013-11-13 16:42:10.331064 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1314 rbd_data.296683d1b58ba.000000000000051b [read 0~4194304] 5.24f8fa46 e5250) v4 -- ?+0 0x11a6ad0 con 0x11a5ae0
  -481> 2013-11-13 16:42:10.974099 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 242 ==== osd_op_reply(1314 rbd_data.296683d1b58ba.000000000000051b [read 0~4194304] v0'0 uv500558 ondisk = 0) v6 ==== 198+0+4194304 (2250253386 0 2940394961) 0x7f13f8000ba0 con 0x11a5ae0
  -480> 2013-11-13 16:42:10.981861 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1315 rbd_data.296683d1b58ba.000000000000051c [read 0~4194304] 5.b54376a2 e5250) v4 -- ?+0 0x11a6ad0 con 0x11a6460
  -479> 2013-11-13 16:42:11.231626 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 247 ==== osd_op_reply(1315 rbd_data.296683d1b58ba.000000000000051c [read 0~4194304] v5236'41954 uv41954 ondisk = 0) v4 ==== 138+0+4194304 (941180526 0 3227142522) 0x7f13f00009f0 con 0x11a6460
  -478> 2013-11-13 16:42:11.240455 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1316 rbd_data.296683d1b58ba.000000000000051d [read 0~4194304] 5.8ebf99cf e5250) v4 -- ?+0 0x11a4950 con 0x11a12f0
  -477> 2013-11-13 16:42:11.440076 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 293 ==== osd_op_reply(1316 rbd_data.296683d1b58ba.000000000000051d [read 0~4194304] v5236'12176 uv12176 ondisk = 0) v4 ==== 138+0+4194304 (3764281623 0 2211394779) 0x7f14040011c0 con 0x11a12f0
  -476> 2013-11-13 16:42:11.447275 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1317 rbd_data.296683d1b58ba.000000000000051e [read 0~4194304] 5.c9e95b89 e5250) v4 -- ?+0 0x11a4950 con 0x11a7940
  -475> 2013-11-13 16:42:11.566098 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 277 ==== osd_op_reply(1317 rbd_data.296683d1b58ba.000000000000051e [read 0~4194304] v5236'32437 uv32437 ondisk = 0) v4 ==== 138+0+4194304 (2834451428 0 2737647200) 0x7f13fc0009f0 con 0x11a7940
  -474> 2013-11-13 16:42:11.574210 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1318 rbd_data.296683d1b58ba.000000000000051f [read 0~4194304] 5.2dbe3378 e5250) v4 -- ?+0 0x11c07a0 con 0x11a6460
  -473> 2013-11-13 16:42:11.661940 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 248 ==== osd_op_reply(1318 rbd_data.296683d1b58ba.000000000000051f [read 0~4194304] v5236'20369 uv20369 ondisk = 0) v4 ==== 138+0+4194304 (580103491 0 749644836) 0x7f13f00009f0 con 0x11a6460
  -472> 2013-11-13 16:42:11.667381 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1319 rbd_data.296683d1b58ba.0000000000000520 [read 0~4194304] 5.77a83fe8 e5250) v4 -- ?+0 0x11a4950 con 0x11a5ae0
  -471> 2013-11-13 16:42:11.691602 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 243 ==== osd_op_reply(1319 rbd_data.296683d1b58ba.0000000000000520 [read 0~4194304] v0'0 uv75217 ondisk = 0) v6 ==== 198+0+4194304 (243027741 0 2112406506) 0x7f13f8000be0 con 0x11a5ae0
  -470> 2013-11-13 16:42:11.700272 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1320 rbd_data.296683d1b58ba.0000000000000521 [read 0~4194304] 5.8053e366 e5250) v4 -- ?+0 0x11c07a0 con 0x11a7940
  -469> 2013-11-13 16:42:11.766117 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 278 ==== osd_op_reply(1320 rbd_data.296683d1b58ba.0000000000000521 [read 0~4194304] v5241'94933 uv94933 ondisk = 0) v4 ==== 138+0+4194304 (723392482 0 1367093712) 0x7f13fc0009f0 con 0x11a7940
  -468> 2013-11-13 16:42:11.771160 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1321 rbd_data.296683d1b58ba.0000000000000522 [read 0~4194304] 5.449fe2b3 e5250) v4 -- ?+0 0x11a4950 con 0x11a5ae0
  -467> 2013-11-13 16:42:11.801450 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 244 ==== osd_op_reply(1321 rbd_data.296683d1b58ba.0000000000000522 [read 0~4194304] v0'0 uv94011 ondisk = 0) v6 ==== 198+0+4194304 (3508681155 0 968688184) 0x7f13f8000be0 con 0x11a5ae0
  -466> 2013-11-13 16:42:11.808762 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1322 rbd_data.296683d1b58ba.0000000000000523 [read 0~4194304] 5.c9e6df3e e5250) v4 -- ?+0 0x11c07a0 con 0x11a12f0
  -465> 2013-11-13 16:42:11.876755 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 294 ==== osd_op_reply(1322 rbd_data.296683d1b58ba.0000000000000523 [read 0~4194304] v5236'23365 uv23365 ondisk = 0) v4 ==== 138+0+4194304 (3240168289 0 2841048947) 0x7f1404000ac0 con 0x11a12f0
  -464> 2013-11-13 16:42:11.884064 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1323 rbd_data.296683d1b58ba.0000000000000524 [read 0~4194304] 5.49d3d21f e5250) v4 -- ?+0 0x11a4950 con 0x11a6460
  -463> 2013-11-13 16:42:11.952285 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 249 ==== osd_op_reply(1323 rbd_data.296683d1b58ba.0000000000000524 [read 0~4194304] v5236'27206 uv27206 ondisk = 0) v4 ==== 138+0+4194304 (317620091 0 1945563910) 0x7f13f00009f0 con 0x11a6460
  -462> 2013-11-13 16:42:11.956821 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1324 rbd_data.296683d1b58ba.0000000000000525 [read 0~4194304] 5.d9fb69b7 e5250) v4 -- ?+0 0x11c07a0 con 0x11a5ae0
  -461> 2013-11-13 16:42:11.985367 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 245 ==== osd_op_reply(1324 rbd_data.296683d1b58ba.0000000000000525 [read 0~4194304] v0'0 uv3436 ondisk = 0) v6 ==== 198+0+4194304 (3187334041 0 3124345078) 0x7f13f8000be0 con 0x11a5ae0
  -460> 2013-11-13 16:42:11.992220 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1325 rbd_data.296683d1b58ba.0000000000000526 [read 0~4194304] 5.58bf6255 e5250) v4 -- ?+0 0x11a4950 con 0x11a0240
  -459> 2013-11-13 16:42:12.086595 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 259 ==== osd_op_reply(1325 rbd_data.296683d1b58ba.0000000000000526 [read 0~4194304] v5236'20975 uv20975 ondisk = 0) v4 ==== 138+0+4194304 (2957681410 0 2827592404) 0x7f140c001090 con 0x11a0240
  -458> 2013-11-13 16:42:12.091416 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1326 rbd_data.296683d1b58ba.0000000000000527 [read 0~4194304] 5.f89181e0 e5250) v4 -- ?+0 0x11c07a0 con 0x11a7940
  -457> 2013-11-13 16:42:12.159624 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 279 ==== osd_op_reply(1326 rbd_data.296683d1b58ba.0000000000000527 [read 0~4194304] v5236'16644623 uv16644623 ondisk = 0) v4 ==== 138+0+4194304 (2534009234 0 1184624301) 0x7f13fc0009f0 con 0x11a7940
  -456> 2013-11-13 16:42:12.167697 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1327 rbd_data.296683d1b58ba.0000000000000528 [read 0~4194304] 5.91a0873 e5250) v4 -- ?+0 0x11a4950 con 0x11a12f0
  -455> 2013-11-13 16:42:12.238766 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 295 ==== osd_op_reply(1327 rbd_data.296683d1b58ba.0000000000000528 [read 0~4194304] v5236'5211 uv5211 ondisk = 0) v4 ==== 138+0+4194304 (313408761 0 1495790315) 0x7f1404000ac0 con 0x11a12f0
  -454> 2013-11-13 16:42:12.248349 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1328 rbd_data.296683d1b58ba.0000000000000529 [read 0~4194304] 5.aaee3ade e5250) v4 -- ?+0 0x11c07a0 con 0x11a7940
  -453> 2013-11-13 16:42:12.316843 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 280 ==== osd_op_reply(1328 rbd_data.296683d1b58ba.0000000000000529 [read 0~4194304] v5236'205156 uv205156 ondisk = 0) v4 ==== 138+0+4194304 (2506041009 0 3836167202) 0x7f13fc0009f0 con 0x11a7940
  -452> 2013-11-13 16:42:12.326001 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1329 rbd_data.296683d1b58ba.000000000000052a [read 0~4194304] 5.18cbbcaf e5250) v4 -- ?+0 0x11a4950 con 0x11a0240
  -451> 2013-11-13 16:42:12.414545 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 260 ==== osd_op_reply(1329 rbd_data.296683d1b58ba.000000000000052a [read 0~4194304] v5236'41811 uv41811 ondisk = 0) v4 ==== 138+0+4194304 (2463392122 0 792273728) 0x7f140c001130 con 0x11a0240
  -450> 2013-11-13 16:42:12.421844 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1330 rbd_data.296683d1b58ba.000000000000052b [read 0~4194304] 5.1fdc4b46 e5250) v4 -- ?+0 0x11c07a0 con 0x11a12f0
  -449> 2013-11-13 16:42:12.493290 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 296 ==== osd_op_reply(1330 rbd_data.296683d1b58ba.000000000000052b [read 0~4194304] v5236'27604 uv27604 ondisk = 0) v4 ==== 138+0+4194304 (146525356 0 1756839249) 0x7f1404000ac0 con 0x11a12f0
  -448> 2013-11-13 16:42:12.498658 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1331 rbd_data.296683d1b58ba.000000000000052c [read 0~4194304] 5.75e8e1af e5250) v4 -- ?+0 0x11a6ba0 con 0x11a7940
  -447> 2013-11-13 16:42:12.568546 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 281 ==== osd_op_reply(1331 rbd_data.296683d1b58ba.000000000000052c [read 0~4194304] v5236'7662 uv7662 ondisk = 0) v4 ==== 138+0+4194304 (509420443 0 2207834095) 0x7f13fc0009f0 con 0x11a7940
  -446> 2013-11-13 16:42:12.576519 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1332 rbd_data.296683d1b58ba.000000000000052d [read 0~4194304] 5.a8395b29 e5250) v4 -- ?+0 0x11a6ba0 con 0x11a7940
  -445> 2013-11-13 16:42:12.644040 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 282 ==== osd_op_reply(1332 rbd_data.296683d1b58ba.000000000000052d [read 0~4194304] v5236'825152 uv825152 ondisk = 0) v4 ==== 138+0+4194304 (4287954114 0 115836676) 0x7f13fc0009f0 con 0x11a7940
  -444> 2013-11-13 16:42:12.651745 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1333 rbd_data.296683d1b58ba.000000000000052e [read 0~4194304] 5.5e34c880 e5250) v4 -- ?+0 0x11c07a0 con 0x11a7940
  -443> 2013-11-13 16:42:12.720028 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 283 ==== osd_op_reply(1333 rbd_data.296683d1b58ba.000000000000052e [read 0~4194304] v5236'30490 uv30490 ondisk = 0) v4 ==== 138+0+4194304 (2527040908 0 1949603136) 0x7f13fc0009f0 con 0x11a7940
  -442> 2013-11-13 16:42:12.728024 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1334 rbd_data.296683d1b58ba.000000000000052f [read 0~4194304] 5.1df6684f e5250) v4 -- ?+0 0x11c07a0 con 0x11a5ae0
  -441> 2013-11-13 16:42:12.754922 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 246 ==== osd_op_reply(1334 rbd_data.296683d1b58ba.000000000000052f [read 0~4194304] v0'0 uv51337 ondisk = 0) v6 ==== 198+0+4194304 (81573884 0 2874155382) 0x7f13f8000ba0 con 0x11a5ae0
  -440> 2013-11-13 16:42:12.760189 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1335 rbd_data.296683d1b58ba.0000000000000530 [read 0~4194304] 5.66e9a398 e5250) v4 -- ?+0 0x11c07a0 con 0x11a5ae0
  -439> 2013-11-13 16:42:12.792524 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 247 ==== osd_op_reply(1335 rbd_data.296683d1b58ba.0000000000000530 [read 0~4194304] v0'0 uv48045 ondisk = 0) v6 ==== 198+0+4194304 (150424103 0 2377912680) 0x7f13f8000ba0 con 0x11a5ae0
  -438> 2013-11-13 16:42:12.800414 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1336 rbd_data.296683d1b58ba.0000000000000531 [read 0~4194304] 5.a5364bc6 e5250) v4 -- ?+0 0x11c07a0 con 0x11a5ae0
  -437> 2013-11-13 16:42:12.836857 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 248 ==== osd_op_reply(1336 rbd_data.296683d1b58ba.0000000000000531 [read 0~4194304] v0'0 uv30856 ondisk = 0) v6 ==== 198+0+4194304 (2981169580 0 4015970684) 0x7f13f8000be0 con 0x11a5ae0
  -436> 2013-11-13 16:42:12.845223 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1337 rbd_data.296683d1b58ba.0000000000000532 [read 0~4194304] 5.53bab3eb e5250) v4 -- ?+0 0x11c07a0 con 0x11a6460
  -435> 2013-11-13 16:42:12.914794 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 250 ==== osd_op_reply(1337 rbd_data.296683d1b58ba.0000000000000532 [read 0~4194304] v5164'6631 uv6631 ondisk = 0) v4 ==== 138+0+4194304 (2974125929 0 612210) 0x7f13f00009f0 con 0x11a6460
  -434> 2013-11-13 16:42:12.922066 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1338 rbd_data.296683d1b58ba.0000000000000533 [read 0~4194304] 5.75c0eac8 e5250) v4 -- ?+0 0x11b34e0 con 0x11a0240
  -433> 2013-11-13 16:42:12.928168 7f142ac44700 10 monclient: tick
  -432> 2013-11-13 16:42:12.928204 7f142ac44700 10 monclient: renew subs? (now: 2013-11-13 16:42:12.928204; renew after: 2013-11-13 16:42:49.926492) -- no
  -431> 2013-11-13 16:42:12.995659 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 261 ==== osd_op_reply(1338 rbd_data.296683d1b58ba.0000000000000533 [read 0~4194304] v5164'100685 uv100685 ondisk = 0) v4 ==== 138+0+4194304 (2808475897 0 3817026346) 0x7f140c000fc0 con 0x11a0240
  -430> 2013-11-13 16:42:13.003443 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1339 rbd_data.296683d1b58ba.0000000000000534 [read 0~4194304] 5.bc542ca2 e5250) v4 -- ?+0 0x11a4950 con 0x11a12f0
  -429> 2013-11-13 16:42:13.073228 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 297 ==== osd_op_reply(1339 rbd_data.296683d1b58ba.0000000000000534 [read 0~4194304] v5164'834541 uv834541 ondisk = 0) v4 ==== 138+0+4194304 (3137293446 0 1406553329) 0x7f1404000ac0 con 0x11a12f0
  -428> 2013-11-13 16:42:13.081083 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1340 rbd_data.296683d1b58ba.0000000000000535 [read 0~4194304] 5.2b4ea039 e5250) v4 -- ?+0 0x11b34e0 con 0x11a12f0
  -427> 2013-11-13 16:42:13.151745 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 298 ==== osd_op_reply(1340 rbd_data.296683d1b58ba.0000000000000535 [read 0~4194304] v5236'94556 uv94556 ondisk = 0) v4 ==== 138+0+4194304 (2875400413 0 3165726067) 0x7f1404000ac0 con 0x11a12f0
  -426> 2013-11-13 16:42:13.159491 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1341 rbd_data.296683d1b58ba.0000000000000536 [read 0~4194304] 5.c9aec31 e5250) v4 -- ?+0 0x11a4950 con 0x11a5ae0
  -425> 2013-11-13 16:42:13.179859 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 249 ==== osd_op_reply(1341 rbd_data.296683d1b58ba.0000000000000536 [read 0~4194304] v0'0 uv10666 ondisk = 0) v6 ==== 198+0+4194304 (1702318585 0 952682217) 0x7f13f8000be0 con 0x11a5ae0
  -424> 2013-11-13 16:42:13.187479 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1342 rbd_data.296683d1b58ba.0000000000000537 [read 0~4194304] 5.f0b38b67 e5250) v4 -- ?+0 0x11b34e0 con 0x11a5ae0
  -423> 2013-11-13 16:42:13.221016 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 250 ==== osd_op_reply(1342 rbd_data.296683d1b58ba.0000000000000537 [read 0~4194304] v0'0 uv90820 ondisk = 0) v6 ==== 198+0+4194304 (356106781 0 304041727) 0x7f13f8000be0 con 0x11a5ae0
  -422> 2013-11-13 16:42:13.229503 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1343 rbd_data.296683d1b58ba.0000000000000538 [read 0~4194304] 5.287d165c e5250) v4 -- ?+0 0x11c2880 con 0x11a7940
  -421> 2013-11-13 16:42:13.298996 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 284 ==== osd_op_reply(1343 rbd_data.296683d1b58ba.0000000000000538 [read 0~4194304] v5164'84745 uv84745 ondisk = 0) v4 ==== 138+0+4194304 (4122035088 0 8645671) 0x7f13fc0009f0 con 0x11a7940
  -420> 2013-11-13 16:42:13.307268 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1344 rbd_data.296683d1b58ba.0000000000000539 [read 0~4194304] 5.1166eb52 e5250) v4 -- ?+0 0x11b34e0 con 0x11a6460
  -419> 2013-11-13 16:42:13.377210 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 251 ==== osd_op_reply(1344 rbd_data.296683d1b58ba.0000000000000539 [read 0~4194304] v5164'55207 uv55207 ondisk = 0) v4 ==== 138+0+4194304 (3513325156 0 952712409) 0x7f13f00009f0 con 0x11a6460
  -418> 2013-11-13 16:42:13.382954 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1345 rbd_data.296683d1b58ba.000000000000053a [read 0~4194304] 5.dae6f222 e5250) v4 -- ?+0 0x11c2880 con 0x11a6460
  -417> 2013-11-13 16:42:13.448747 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 252 ==== osd_op_reply(1345 rbd_data.296683d1b58ba.000000000000053a [read 0~4194304] v5236'21880 uv21880 ondisk = 0) v4 ==== 138+0+4194304 (161052986 0 1794366411) 0x7f13f00009f0 con 0x11a6460
  -416> 2013-11-13 16:42:13.453816 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1346 rbd_data.296683d1b58ba.000000000000053b [read 0~4194304] 5.59b36674 e5250) v4 -- ?+0 0x11a4950 con 0x11a0240
  -415> 2013-11-13 16:42:13.570383 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 262 ==== osd_op_reply(1346 rbd_data.296683d1b58ba.000000000000053b [read 0~4194304] v5236'12721 uv12721 ondisk = 0) v4 ==== 138+0+4194304 (2564616957 0 3497984470) 0x7f140c001000 con 0x11a0240
  -414> 2013-11-13 16:42:13.578364 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1347 rbd_data.296683d1b58ba.000000000000053c [read 0~4194304] 5.e1b267de e5250) v4 -- ?+0 0x11b34e0 con 0x11a12f0
  -413> 2013-11-13 16:42:13.648003 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 299 ==== osd_op_reply(1347 rbd_data.296683d1b58ba.000000000000053c [read 0~4194304] v5236'7907 uv7907 ondisk = 0) v4 ==== 138+0+4194304 (2874540529 0 3728625515) 0x7f1404000ac0 con 0x11a12f0
  -412> 2013-11-13 16:42:13.655210 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1348 rbd_data.296683d1b58ba.000000000000053d [read 0~4194304] 5.3ef96225 e5250) v4 -- ?+0 0x11b34e0 con 0x11a6460
  -411> 2013-11-13 16:42:13.724114 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 253 ==== osd_op_reply(1348 rbd_data.296683d1b58ba.000000000000053d [read 0~4194304] v5236'38095 uv38095 ondisk = 0) v4 ==== 138+0+4194304 (526247002 0 2332698278) 0x7f13f00009f0 con 0x11a6460
  -410> 2013-11-13 16:42:13.731415 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1349 rbd_data.296683d1b58ba.000000000000053e [read 0~4194304] 5.39b7b3eb e5250) v4 -- ?+0 0x11a4950 con 0x11a6460
  -409> 2013-11-13 16:42:13.800292 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 254 ==== osd_op_reply(1349 rbd_data.296683d1b58ba.000000000000053e [read 0~4194304] v5236'6729 uv6729 ondisk = 0) v4 ==== 138+0+4194304 (3894999398 0 2791629290) 0x7f13f00009f0 con 0x11a6460
  -408> 2013-11-13 16:42:13.808180 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1350 rbd_data.296683d1b58ba.000000000000053f [read 0~4194304] 5.b459ab25 e5250) v4 -- ?+0 0x11c2880 con 0x11a6460
  -407> 2013-11-13 16:42:13.879262 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 255 ==== osd_op_reply(1350 rbd_data.296683d1b58ba.000000000000053f [read 0~4194304] v5236'40980 uv40980 ondisk = 0) v4 ==== 138+0+4194304 (2549150011 0 3209644568) 0x7f13f00009f0 con 0x11a6460
  -406> 2013-11-13 16:42:13.887029 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1351 rbd_data.296683d1b58ba.0000000000000540 [read 0~4194304] 5.7c0f1f05 e5250) v4 -- ?+0 0x11a4950 con 0x11a7940
  -405> 2013-11-13 16:42:13.957246 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 285 ==== osd_op_reply(1351 rbd_data.296683d1b58ba.0000000000000540 [read 0~4194304] v5236'72740 uv72740 ondisk = 0) v4 ==== 138+0+4194304 (3472848143 0 2934020114) 0x7f13fc0009f0 con 0x11a7940
  -404> 2013-11-13 16:42:13.965565 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1352 rbd_data.296683d1b58ba.0000000000000541 [read 0~4194304] 5.9f6ee7f5 e5250) v4 -- ?+0 0x11c2880 con 0x11a12f0
  -403> 2013-11-13 16:42:14.034943 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 300 ==== osd_op_reply(1352 rbd_data.296683d1b58ba.0000000000000541 [read 0~4194304] v5236'156645 uv156645 ondisk = 0) v4 ==== 138+0+4194304 (111867437 0 1551300291) 0x7f1404000ac0 con 0x11a12f0
  -402> 2013-11-13 16:42:14.042875 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1353 rbd_data.296683d1b58ba.0000000000000542 [read 0~4194304] 5.785f67a e5250) v4 -- ?+0 0x11a4950 con 0x11a6460
  -401> 2013-11-13 16:42:14.113030 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 256 ==== osd_op_reply(1353 rbd_data.296683d1b58ba.0000000000000542 [read 0~4194304] v5236'20259 uv20259 ondisk = 0) v4 ==== 138+0+4194304 (3809780714 0 1610275227) 0x7f13f00009f0 con 0x11a6460
  -400> 2013-11-13 16:42:14.120892 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1354 rbd_data.296683d1b58ba.0000000000000543 [read 0~4194304] 5.f8cf4194 e5250) v4 -- ?+0 0x11c2880 con 0x11a6460
  -399> 2013-11-13 16:42:14.190637 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 257 ==== osd_op_reply(1354 rbd_data.296683d1b58ba.0000000000000543 [read 0~4194304] v5236'40865 uv40865 ondisk = 0) v4 ==== 138+0+4194304 (622123842 0 2399053430) 0x7f13f00009f0 con 0x11a6460
  -398> 2013-11-13 16:42:14.198856 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1355 rbd_data.296683d1b58ba.0000000000000544 [read 0~4194304] 5.acff6b79 e5250) v4 -- ?+0 0x11a4950 con 0x11a6460
  -397> 2013-11-13 16:42:14.273989 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 258 ==== osd_op_reply(1355 rbd_data.296683d1b58ba.0000000000000544 [read 0~4194304] v5236'66994 uv66994 ondisk = 0) v4 ==== 138+0+4194304 (3760618180 0 1607091843) 0x7f13f00009f0 con 0x11a6460
  -396> 2013-11-13 16:42:14.282191 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1356 rbd_data.296683d1b58ba.0000000000000545 [read 0~4194304] 5.d246a44e e5250) v4 -- ?+0 0x11c2880 con 0x11a5ae0
  -395> 2013-11-13 16:42:14.311920 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 251 ==== osd_op_reply(1356 rbd_data.296683d1b58ba.0000000000000545 [read 0~4194304] v0'0 uv69060 ondisk = 0) v6 ==== 198+0+4194304 (1242799233 0 3726601486) 0x7f13f8000a20 con 0x11a5ae0
  -394> 2013-11-13 16:42:14.320211 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1357 rbd_data.296683d1b58ba.0000000000000546 [read 0~4194304] 5.9125cd3a e5250) v4 -- ?+0 0x11c2880 con 0x11a6460
  -393> 2013-11-13 16:42:14.546406 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 259 ==== osd_op_reply(1357 rbd_data.296683d1b58ba.0000000000000546 [read 0~4194304] v5236'41430 uv41430 ondisk = 0) v4 ==== 138+0+4194304 (4175290579 0 1134956211) 0x7f13f00009f0 con 0x11a6460
  -392> 2013-11-13 16:42:14.554064 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1358 rbd_data.296683d1b58ba.0000000000000547 [read 0~4194304] 5.15b346eb e5250) v4 -- ?+0 0x11c2880 con 0x11a12f0
  -391> 2013-11-13 16:42:14.620190 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 301 ==== osd_op_reply(1358 rbd_data.296683d1b58ba.0000000000000547 [read 0~4194304] v5236'6099 uv6099 ondisk = 0) v4 ==== 138+0+4194304 (3898838661 0 2086688214) 0x7f1404000ac0 con 0x11a12f0
  -390> 2013-11-13 16:42:14.626625 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1359 rbd_data.296683d1b58ba.0000000000000548 [read 0~4194304] 5.d1ef4a9d e5250) v4 -- ?+0 0x11c2880 con 0x11a5ae0
  -389> 2013-11-13 16:42:14.662813 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 252 ==== osd_op_reply(1359 rbd_data.296683d1b58ba.0000000000000548 [read 0~4194304] v0'0 uv17137 ondisk = 0) v6 ==== 198+0+4194304 (597224853 0 4007585893) 0x7f13f8000e70 con 0x11a5ae0
  -388> 2013-11-13 16:42:14.667814 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1360 rbd_data.296683d1b58ba.0000000000000549 [read 0~4194304] 5.aebde031 e5250) v4 -- ?+0 0x11c2880 con 0x11a12f0
  -387> 2013-11-13 16:42:14.751941 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 302 ==== osd_op_reply(1360 rbd_data.296683d1b58ba.0000000000000549 [read 0~4194304] v5236'7033 uv7033 ondisk = 0) v4 ==== 138+0+4194304 (3047255309 0 2128519106) 0x7f1404000ac0 con 0x11a12f0
  -386> 2013-11-13 16:42:14.758774 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1361 rbd_data.296683d1b58ba.000000000000054a [read 0~4194304] 5.1ef9a98 e5250) v4 -- ?+0 0x11c2880 con 0x11a5ae0
  -385> 2013-11-13 16:42:14.789797 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 253 ==== osd_op_reply(1361 rbd_data.296683d1b58ba.000000000000054a [read 0~4194304] v0'0 uv14815 ondisk = 0) v6 ==== 198+0+4194304 (46307243 0 1430728268) 0x7f13f8000eb0 con 0x11a5ae0
  -384> 2013-11-13 16:42:14.797832 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1362 rbd_data.296683d1b58ba.000000000000054b [read 0~4194304] 5.538c98b7 e5250) v4 -- ?+0 0x11c2880 con 0x11a0240
  -383> 2013-11-13 16:42:14.972887 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 263 ==== osd_op_reply(1362 rbd_data.296683d1b58ba.000000000000054b [read 0~4194304] v5236'24302 uv24302 ondisk = 0) v4 ==== 138+0+4194304 (2435016209 0 3550704778) 0x7f140c000b40 con 0x11a0240
  -382> 2013-11-13 16:42:14.980981 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1363 rbd_data.296683d1b58ba.000000000000054c [read 0~4194304] 5.e56813fe e5250) v4 -- ?+0 0x11b0330 con 0x11a12f0
  -381> 2013-11-13 16:42:15.049819 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 303 ==== osd_op_reply(1363 rbd_data.296683d1b58ba.000000000000054c [read 0~4194304] v5236'6057 uv6057 ondisk = 0) v4 ==== 138+0+4194304 (2443169450 0 3078057348) 0x7f1404000ac0 con 0x11a12f0
  -380> 2013-11-13 16:42:15.057889 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1364 rbd_data.296683d1b58ba.000000000000054d [read 0~4194304] 5.80bf045 e5250) v4 -- ?+0 0x11b0330 con 0x11a6460
  -379> 2013-11-13 16:42:15.125452 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 260 ==== osd_op_reply(1364 rbd_data.296683d1b58ba.000000000000054d [read 0~4194304] v5236'1016 uv1016 ondisk = 0) v4 ==== 138+0+4194304 (1113669876 0 4045136438) 0x7f13f00009f0 con 0x11a6460
  -378> 2013-11-13 16:42:15.133592 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1365 rbd_data.296683d1b58ba.000000000000054e [read 0~4194304] 5.fad4f489 e5250) v4 -- ?+0 0x11b7020 con 0x11a12f0
  -377> 2013-11-13 16:42:15.201783 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 304 ==== osd_op_reply(1365 rbd_data.296683d1b58ba.000000000000054e [read 0~4194304] v5236'79934 uv79934 ondisk = 0) v4 ==== 138+0+4194304 (3664228370 0 357745627) 0x7f1404000ac0 con 0x11a12f0
  -376> 2013-11-13 16:42:15.209656 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1366 rbd_data.296683d1b58ba.000000000000054f [read 0~4194304] 5.16091e08 e5250) v4 -- ?+0 0x11b7020 con 0x11a12f0
  -375> 2013-11-13 16:42:15.278987 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 305 ==== osd_op_reply(1366 rbd_data.296683d1b58ba.000000000000054f [read 0~4194304] v5236'30411 uv30411 ondisk = 0) v4 ==== 138+0+4194304 (138589108 0 216939556) 0x7f1404000ac0 con 0x11a12f0
  -374> 2013-11-13 16:42:15.287435 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1367 rbd_data.296683d1b58ba.0000000000000550 [read 0~4194304] 5.a82469e0 e5250) v4 -- ?+0 0x11b7020 con 0x11a0240
  -373> 2013-11-13 16:42:15.383841 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 264 ==== osd_op_reply(1367 rbd_data.296683d1b58ba.0000000000000550 [read 0~4194304] v5236'19642 uv19642 ondisk = 0) v4 ==== 138+0+4194304 (1342297628 0 2195054656) 0x7f140c000b80 con 0x11a0240
  -372> 2013-11-13 16:42:15.389114 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1368 rbd_data.296683d1b58ba.0000000000000551 [read 0~4194304] 5.4b87d89d e5250) v4 -- ?+0 0x11b7020 con 0x11a12f0
  -371> 2013-11-13 16:42:15.459846 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 306 ==== osd_op_reply(1368 rbd_data.296683d1b58ba.0000000000000551 [read 0~4194304] v5236'23800 uv23800 ondisk = 0) v4 ==== 138+0+4194304 (2658499906 0 3875324095) 0x7f1404000ac0 con 0x11a12f0
  -370> 2013-11-13 16:42:15.465382 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1369 rbd_data.296683d1b58ba.0000000000000552 [read 0~4194304] 5.ab34e7f5 e5250) v4 -- ?+0 0x11b7020 con 0x11a12f0
  -369> 2013-11-13 16:42:15.533790 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 307 ==== osd_op_reply(1369 rbd_data.296683d1b58ba.0000000000000552 [read 0~4194304] v5236'154306 uv154306 ondisk = 0) v4 ==== 138+0+4194304 (4238167130 0 3762918755) 0x7f1404000ac0 con 0x11a12f0
  -368> 2013-11-13 16:42:15.542103 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1370 rbd_data.296683d1b58ba.0000000000000553 [read 0~4194304] 5.cbd11a61 e5250) v4 -- ?+0 0x11b7020 con 0x11a0240
  -367> 2013-11-13 16:42:15.611937 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 265 ==== osd_op_reply(1370 rbd_data.296683d1b58ba.0000000000000553 [read 0~4194304] v5236'5064 uv5064 ondisk = 0) v4 ==== 138+0+4194304 (2007317819 0 3149747179) 0x7f140c001090 con 0x11a0240
  -366> 2013-11-13 16:42:15.620109 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1371 rbd_data.296683d1b58ba.0000000000000554 [read 0~4194304] 5.de5678e9 e5250) v4 -- ?+0 0x11b7020 con 0x11a6460
  -365> 2013-11-13 16:42:15.690066 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 261 ==== osd_op_reply(1371 rbd_data.296683d1b58ba.0000000000000554 [read 0~4194304] v5236'39758 uv39758 ondisk = 0) v4 ==== 138+0+4194304 (628532520 0 413634612) 0x7f13f00009f0 con 0x11a6460
  -364> 2013-11-13 16:42:15.697936 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1372 rbd_data.296683d1b58ba.0000000000000555 [read 0~4194304] 5.9361be0a e5250) v4 -- ?+0 0x11b7020 con 0x11a0240
  -363> 2013-11-13 16:42:15.772374 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 266 ==== osd_op_reply(1372 rbd_data.296683d1b58ba.0000000000000555 [read 0~4194304] v5236'9600 uv9600 ondisk = 0) v4 ==== 138+0+4194304 (1831648560 0 2126609876) 0x7f140c000b80 con 0x11a0240
  -362> 2013-11-13 16:42:15.779256 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1373 rbd_data.296683d1b58ba.0000000000000556 [read 0~4194304] 5.1cc4e670 e5250) v4 -- ?+0 0x11b7020 con 0x11a0240
  -361> 2013-11-13 16:42:15.858657 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 267 ==== osd_op_reply(1373 rbd_data.296683d1b58ba.0000000000000556 [read 0~4194304] v5236'10302 uv10302 ondisk = 0) v4 ==== 138+0+4194304 (3723055225 0 2992107121) 0x7f140c000b80 con 0x11a0240
  -360> 2013-11-13 16:42:15.867123 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1374 rbd_data.296683d1b58ba.0000000000000557 [read 0~4194304] 5.de8f0b8f e5250) v4 -- ?+0 0x11b7020 con 0x11a12f0
  -359> 2013-11-13 16:42:15.936342 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 308 ==== osd_op_reply(1374 rbd_data.296683d1b58ba.0000000000000557 [read 0~4194304] v5236'9945 uv9945 ondisk = 0) v4 ==== 138+0+4194304 (3330282550 0 138169141) 0x7f1404000ac0 con 0x11a12f0
  -358> 2013-11-13 16:42:15.944275 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1375 rbd_data.296683d1b58ba.0000000000000558 [read 0~4194304] 5.a8c88a91 e5250) v4 -- ?+0 0x11b8490 con 0x11a6460
  -357> 2013-11-13 16:42:16.053214 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 262 ==== osd_op_reply(1375 rbd_data.296683d1b58ba.0000000000000558 [read 0~4194304] v5236'15425 uv15425 ondisk = 0) v4 ==== 138+0+4194304 (1912354393 0 4294017218) 0x7f13f00009f0 con 0x11a6460
  -356> 2013-11-13 16:42:16.061624 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1376 rbd_data.296683d1b58ba.0000000000000559 [read 0~4194304] 5.96d1af0c e5250) v4 -- ?+0 0x11b8490 con 0x11a6460
  -355> 2013-11-13 16:42:16.144693 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 263 ==== osd_op_reply(1376 rbd_data.296683d1b58ba.0000000000000559 [read 0~4194304] v5236'448941 uv448941 ondisk = 0) v4 ==== 138+0+4194304 (2918698926 0 614459443) 0x7f13f00009f0 con 0x11a6460
  -354> 2013-11-13 16:42:16.152387 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1377 rbd_data.296683d1b58ba.000000000000055a [read 0~4194304] 5.eb38ce9f e5250) v4 -- ?+0 0x11b8090 con 0x11a12f0
  -353> 2013-11-13 16:42:16.234428 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 309 ==== osd_op_reply(1377 rbd_data.296683d1b58ba.000000000000055a [read 0~4194304] v5236'7231 uv7231 ondisk = 0) v4 ==== 138+0+4194304 (1361852335 0 1760912194) 0x7f1404000ac0 con 0x11a12f0
  -352> 2013-11-13 16:42:16.242341 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1378 rbd_data.296683d1b58ba.000000000000055b [read 0~4194304] 5.11e3a493 e5250) v4 -- ?+0 0x11b8090 con 0x11a12f0
  -351> 2013-11-13 16:42:16.313253 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 310 ==== osd_op_reply(1378 rbd_data.296683d1b58ba.000000000000055b [read 0~4194304] v5236'753 uv753 ondisk = 0) v4 ==== 138+0+4194304 (3906672622 0 3377861953) 0x7f1404000ac0 con 0x11a12f0
  -350> 2013-11-13 16:42:16.322104 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1379 rbd_data.296683d1b58ba.000000000000055c [read 0~4194304] 5.c256d97f e5250) v4 -- ?+0 0x11b8090 con 0x11a5ae0
  -349> 2013-11-13 16:42:16.341174 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 254 ==== osd_op_reply(1379 rbd_data.296683d1b58ba.000000000000055c [read 0~4194304] v0'0 uv139876 ondisk = 0) v6 ==== 198+0+4194304 (3749996538 0 2806290283) 0x7f13f8000aa0 con 0x11a5ae0
  -348> 2013-11-13 16:42:16.349035 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1380 rbd_data.296683d1b58ba.000000000000055d [read 0~4194304] 5.bd8fafe5 e5250) v4 -- ?+0 0x11b8090 con 0x11a5ae0
  -347> 2013-11-13 16:42:16.386105 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 255 ==== osd_op_reply(1380 rbd_data.296683d1b58ba.000000000000055d [read 0~4194304] v0'0 uv28466 ondisk = 0) v6 ==== 198+0+4194304 (2933303243 0 3539966728) 0x7f13f8000ff0 con 0x11a5ae0
  -346> 2013-11-13 16:42:16.392734 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1381 rbd_data.296683d1b58ba.000000000000055e [read 0~4194304] 5.c796f42b e5250) v4 -- ?+0 0x11b8090 con 0x11a7940
  -345> 2013-11-13 16:42:16.462209 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 286 ==== osd_op_reply(1381 rbd_data.296683d1b58ba.000000000000055e [read 0~4194304] v5236'31640 uv31640 ondisk = 0) v4 ==== 138+0+4194304 (1955023770 0 104835999) 0x7f13fc0009f0 con 0x11a7940
  -344> 2013-11-13 16:42:16.466707 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1382 rbd_data.296683d1b58ba.000000000000055f [read 0~4194304] 5.504a017 e5250) v4 -- ?+0 0x11b8cb0 con 0x11a12f0
  -343> 2013-11-13 16:42:16.534110 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 311 ==== osd_op_reply(1382 rbd_data.296683d1b58ba.000000000000055f [read 0~4194304] v5236'28014 uv28014 ondisk = 0) v4 ==== 138+0+4194304 (3652490608 0 2501335588) 0x7f1404000ac0 con 0x11a12f0
  -342> 2013-11-13 16:42:16.539904 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1383 rbd_data.296683d1b58ba.0000000000000560 [read 0~4194304] 5.181f99a8 e5250) v4 -- ?+0 0x11b8090 con 0x11a7940
  -341> 2013-11-13 16:42:16.619241 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 287 ==== osd_op_reply(1383 rbd_data.296683d1b58ba.0000000000000560 [read 0~4194304] v5236'29119 uv29119 ondisk = 0) v4 ==== 138+0+4194304 (100382731 0 256853926) 0x7f13fc0009f0 con 0x11a7940
  -340> 2013-11-13 16:42:16.626645 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1384 rbd_data.296683d1b58ba.0000000000000561 [read 0~4194304] 5.bdb60359 e5250) v4 -- ?+0 0x11b8cb0 con 0x11a6460
  -339> 2013-11-13 16:42:16.742003 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 264 ==== osd_op_reply(1384 rbd_data.296683d1b58ba.0000000000000561 [read 0~4194304] v5236'50463 uv50463 ondisk = 0) v4 ==== 138+0+4194304 (877860549 0 3256983597) 0x7f13f00009f0 con 0x11a6460
  -338> 2013-11-13 16:42:16.750079 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1385 rbd_data.296683d1b58ba.0000000000000562 [read 0~4194304] 5.6aa1c7d8 e5250) v4 -- ?+0 0x11b8cb0 con 0x11a0240
  -337> 2013-11-13 16:42:16.905215 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 268 ==== osd_op_reply(1385 rbd_data.296683d1b58ba.0000000000000562 [read 0~4194304] v5236'804 uv804 ondisk = 0) v4 ==== 138+0+4194304 (865288047 0 1748695940) 0x7f140c000b40 con 0x11a0240
  -336> 2013-11-13 16:42:16.910383 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1386 rbd_data.296683d1b58ba.0000000000000563 [read 0~4194304] 5.2b2958ec e5250) v4 -- ?+0 0x11b8cb0 con 0x11a0240
  -335> 2013-11-13 16:42:17.047034 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 269 ==== osd_op_reply(1386 rbd_data.296683d1b58ba.0000000000000563 [read 0~4194304] v5236'25258 uv25258 ondisk = 0) v4 ==== 138+0+4194304 (3562618049 0 1549360334) 0x7f140c000b40 con 0x11a0240
  -334> 2013-11-13 16:42:17.055640 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1387 rbd_data.296683d1b58ba.0000000000000564 [read 0~4194304] 5.70314789 e5250) v4 -- ?+0 0x11b8cb0 con 0x11a7940
  -333> 2013-11-13 16:42:17.124925 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 288 ==== osd_op_reply(1387 rbd_data.296683d1b58ba.0000000000000564 [read 0~4194304] v5236'121653 uv121653 ondisk = 0) v4 ==== 138+0+4194304 (3791782560 0 2357752520) 0x7f13fc0009f0 con 0x11a7940
  -332> 2013-11-13 16:42:17.132846 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1388 rbd_data.296683d1b58ba.0000000000000565 [read 0~4194304] 5.83333208 e5250) v4 -- ?+0 0x11b8cb0 con 0x11a5ae0
  -331> 2013-11-13 16:42:17.164507 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 256 ==== osd_op_reply(1388 rbd_data.296683d1b58ba.0000000000000565 [read 0~4194304] v0'0 uv113494 ondisk = 0) v6 ==== 198+0+4194304 (133665085 0 492256710) 0x7f13f8000ff0 con 0x11a5ae0
  -330> 2013-11-13 16:42:17.172727 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1389 rbd_data.296683d1b58ba.0000000000000566 [read 0~4194304] 5.f719fcaa e5250) v4 -- ?+0 0x11b8cb0 con 0x11a5ae0
  -329> 2013-11-13 16:42:17.202111 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 257 ==== osd_op_reply(1389 rbd_data.296683d1b58ba.0000000000000566 [read 0~4194304] v0'0 uv73285 ondisk = 0) v6 ==== 198+0+4194304 (546531114 0 1844369124) 0x7f13f8000ff0 con 0x11a5ae0
  -328> 2013-11-13 16:42:17.209545 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1390 rbd_data.296683d1b58ba.0000000000000567 [read 0~4194304] 5.a5a6c9ea e5250) v4 -- ?+0 0x11b8cb0 con 0x11a0240
  -327> 2013-11-13 16:42:17.340655 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 270 ==== osd_op_reply(1390 rbd_data.296683d1b58ba.0000000000000567 [read 0~4194304] v5236'20254 uv20254 ondisk = 0) v4 ==== 138+0+4194304 (3504642801 0 2379474437) 0x7f140c000b40 con 0x11a0240
  -326> 2013-11-13 16:42:17.348750 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1391 rbd_data.296683d1b58ba.0000000000000568 [read 0~4194304] 5.7a0d3115 e5250) v4 -- ?+0 0x11b8cb0 con 0x11a0240
  -325> 2013-11-13 16:42:17.463632 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 271 ==== osd_op_reply(1391 rbd_data.296683d1b58ba.0000000000000568 [read 0~4194304] v5236'52045 uv52045 ondisk = 0) v4 ==== 138+0+4194304 (1869664480 0 3432792463) 0x7f140c000b80 con 0x11a0240
  -324> 2013-11-13 16:42:17.471736 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1392 rbd_data.296683d1b58ba.0000000000000569 [read 0~4194304] 5.b66870bb e5250) v4 -- ?+0 0x11b8cb0 con 0x11a0240
  -323> 2013-11-13 16:42:17.541273 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 272 ==== osd_op_reply(1392 rbd_data.296683d1b58ba.0000000000000569 [read 0~4194304] v5236'42890 uv42890 ondisk = 0) v4 ==== 138+0+4194304 (1382405326 0 1815874062) 0x7f140c000b80 con 0x11a0240
  -322> 2013-11-13 16:42:17.549281 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1393 rbd_data.296683d1b58ba.000000000000056a [read 0~4194304] 5.dbb3cab8 e5250) v4 -- ?+0 0x11b8cb0 con 0x11a12f0
  -321> 2013-11-13 16:42:17.617773 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 312 ==== osd_op_reply(1393 rbd_data.296683d1b58ba.000000000000056a [read 0~4194304] v5236'680 uv680 ondisk = 0) v4 ==== 138+0+4194304 (2193966356 0 3600748411) 0x7f1404000ac0 con 0x11a12f0
  -320> 2013-11-13 16:42:17.623064 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1394 rbd_data.296683d1b58ba.000000000000056b [read 0~4194304] 5.17d5844b e5250) v4 -- ?+0 0x11b9770 con 0x11a0240
  -319> 2013-11-13 16:42:17.695363 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 273 ==== osd_op_reply(1394 rbd_data.296683d1b58ba.000000000000056b [read 0~4194304] v5236'115471 uv115471 ondisk = 0) v4 ==== 138+0+4194304 (4027383790 0 619989344) 0x7f140c000b80 con 0x11a0240
  -318> 2013-11-13 16:42:17.703711 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1395 rbd_data.296683d1b58ba.000000000000056c [read 0~4194304] 5.cfb9f426 e5250) v4 -- ?+0 0x11b8e50 con 0x11a5ae0
  -317> 2013-11-13 16:42:17.743444 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 258 ==== osd_op_reply(1395 rbd_data.296683d1b58ba.000000000000056c [read 0~4194304] v0'0 uv677016 ondisk = 0) v6 ==== 198+0+4194304 (1991023077 0 3130021279) 0x7f13f8000ff0 con 0x11a5ae0
  -316> 2013-11-13 16:42:17.751693 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1396 rbd_data.296683d1b58ba.000000000000056d [read 0~4194304] 5.cb8d998a e5250) v4 -- ?+0 0x11b8e50 con 0x11a6460
  -315> 2013-11-13 16:42:17.823684 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 265 ==== osd_op_reply(1396 rbd_data.296683d1b58ba.000000000000056d [read 0~4194304] v5236'19788 uv19788 ondisk = 0) v4 ==== 138+0+4194304 (1818929022 0 1250042176) 0x7f13f00009f0 con 0x11a6460
  -314> 2013-11-13 16:42:17.831366 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1397 rbd_data.296683d1b58ba.000000000000056e [read 0~4194304] 5.c66345a2 e5250) v4 -- ?+0 0x11b9770 con 0x11a7940
  -313> 2013-11-13 16:42:17.899852 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 289 ==== osd_op_reply(1397 rbd_data.296683d1b58ba.000000000000056e [read 0~4194304] v5236'6574 uv6574 ondisk = 0) v4 ==== 138+0+4194304 (703720595 0 581401910) 0x7f13fc0009f0 con 0x11a7940
  -312> 2013-11-13 16:42:17.907698 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1398 rbd_data.296683d1b58ba.000000000000056f [read 0~4194304] 5.2dff65d3 e5250) v4 -- ?+0 0x11b9770 con 0x11a12f0
  -311> 2013-11-13 16:42:17.977541 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 313 ==== osd_op_reply(1398 rbd_data.296683d1b58ba.000000000000056f [read 0~4194304] v5236'14542 uv14542 ondisk = 0) v4 ==== 138+0+4194304 (2850139400 0 2944700854) 0x7f1404000ac0 con 0x11a12f0
  -310> 2013-11-13 16:42:17.986696 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1399 rbd_data.296683d1b58ba.0000000000000570 [read 0~4194304] 5.4a10d55a e5250) v4 -- ?+0 0x11b9770 con 0x11a7940
  -309> 2013-11-13 16:42:18.054882 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 290 ==== osd_op_reply(1399 rbd_data.296683d1b58ba.0000000000000570 [read 0~4194304] v5236'11255 uv11255 ondisk = 0) v4 ==== 138+0+4194304 (2782810662 0 268666269) 0x7f13fc0009f0 con 0x11a7940
  -308> 2013-11-13 16:42:18.063714 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1400 rbd_data.296683d1b58ba.0000000000000571 [read 0~4194304] 5.c771e532 e5250) v4 -- ?+0 0x11b9770 con 0x11a12f0
  -307> 2013-11-13 16:42:18.133542 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 314 ==== osd_op_reply(1400 rbd_data.296683d1b58ba.0000000000000571 [read 0~4194304] v5236'10145 uv10145 ondisk = 0) v4 ==== 138+0+4194304 (901132603 0 4036073161) 0x7f1404000ac0 con 0x11a12f0
  -306> 2013-11-13 16:42:18.139389 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1401 rbd_data.296683d1b58ba.0000000000000572 [read 0~4194304] 5.58468c87 e5250) v4 -- ?+0 0x11b9770 con 0x11a7940
  -305> 2013-11-13 16:42:18.208462 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 291 ==== osd_op_reply(1401 rbd_data.296683d1b58ba.0000000000000572 [read 0~4194304] v5236'1737956 uv1737956 ondisk = 0) v4 ==== 138+0+4194304 (3680606488 0 3546861696) 0x7f13fc0009f0 con 0x11a7940
  -304> 2013-11-13 16:42:18.214559 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1402 rbd_data.296683d1b58ba.0000000000000573 [read 0~4194304] 5.bbd97cc9 e5250) v4 -- ?+0 0x11b8e50 con 0x11a6460
  -303> 2013-11-13 16:42:18.283742 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 266 ==== osd_op_reply(1402 rbd_data.296683d1b58ba.0000000000000573 [read 0~4194304] v5236'41313 uv41313 ondisk = 0) v4 ==== 138+0+4194304 (829004961 0 3716649354) 0x7f13f00009f0 con 0x11a6460
  -302> 2013-11-13 16:42:18.292805 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1403 rbd_data.296683d1b58ba.0000000000000574 [read 0~4194304] 5.1539e08d e5250) v4 -- ?+0 0x11b8e50 con 0x11a7940
  -301> 2013-11-13 16:42:18.362869 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 292 ==== osd_op_reply(1403 rbd_data.296683d1b58ba.0000000000000574 [read 0~4194304] v5236'756 uv756 ondisk = 0) v4 ==== 138+0+4194304 (3692285340 0 1460957031) 0x7f13fc0009f0 con 0x11a7940
  -300> 2013-11-13 16:42:18.371213 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1404 rbd_data.296683d1b58ba.0000000000000575 [read 0~4194304] 5.9cef013d e5250) v4 -- ?+0 0x11b9770 con 0x11a5ae0
  -299> 2013-11-13 16:42:18.408578 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 259 ==== osd_op_reply(1404 rbd_data.296683d1b58ba.0000000000000575 [read 0~4194304] v0'0 uv66183 ondisk = 0) v6 ==== 198+0+4194304 (2633955205 0 4123005009) 0x7f13f8000e70 con 0x11a5ae0
  -298> 2013-11-13 16:42:18.413734 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1405 rbd_data.296683d1b58ba.0000000000000576 [read 0~4194304] 5.a1df8cb2 e5250) v4 -- ?+0 0x11b8e50 con 0x11a5ae0
  -297> 2013-11-13 16:42:18.441372 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 260 ==== osd_op_reply(1405 rbd_data.296683d1b58ba.0000000000000576 [read 0~4194304] v0'0 uv62560 ondisk = 0) v6 ==== 198+0+4194304 (1446182707 0 301748961) 0x7f13f8000e70 con 0x11a5ae0
  -296> 2013-11-13 16:42:18.448781 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1406 rbd_data.296683d1b58ba.0000000000000577 [read 0~4194304] 5.f2889a85 e5250) v4 -- ?+0 0x11b8e50 con 0x11a5ae0
  -295> 2013-11-13 16:42:18.473816 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 261 ==== osd_op_reply(1406 rbd_data.296683d1b58ba.0000000000000577 [read 0~4194304] v0'0 uv23784 ondisk = 0) v6 ==== 198+0+4194304 (2720235006 0 40929222) 0x7f13f8000e70 con 0x11a5ae0
  -294> 2013-11-13 16:42:18.481423 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1407 rbd_data.296683d1b58ba.0000000000000578 [read 0~4194304] 5.ff881c14 e5250) v4 -- ?+0 0x11b9770 con 0x11a6460
  -293> 2013-11-13 16:42:18.551131 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 267 ==== osd_op_reply(1407 rbd_data.296683d1b58ba.0000000000000578 [read 0~4194304] v5236'15307 uv15307 ondisk = 0) v4 ==== 138+0+4194304 (1510229302 0 1164981665) 0x7f13f00009f0 con 0x11a6460
  -292> 2013-11-13 16:42:18.559005 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1408 rbd_data.296683d1b58ba.0000000000000579 [read 0~4194304] 5.78c9050e e5250) v4 -- ?+0 0x11b8e50 con 0x11a7940
  -291> 2013-11-13 16:42:18.626868 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 293 ==== osd_op_reply(1408 rbd_data.296683d1b58ba.0000000000000579 [read 0~4194304] v5236'32211 uv32211 ondisk = 0) v4 ==== 138+0+4194304 (2005080013 0 2116273472) 0x7f13fc0009f0 con 0x11a7940
  -290> 2013-11-13 16:42:18.635238 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1409 rbd_data.296683d1b58ba.000000000000057a [read 0~4194304] 5.f2ae8a18 e5250) v4 -- ?+0 0x11b9770 con 0x11a5ae0
  -289> 2013-11-13 16:42:18.659130 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 262 ==== osd_op_reply(1409 rbd_data.296683d1b58ba.000000000000057a [read 0~4194304] v0'0 uv44285 ondisk = 0) v6 ==== 198+0+4194304 (314011848 0 1631662982) 0x7f13f8000e70 con 0x11a5ae0
  -288> 2013-11-13 16:42:18.667036 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1410 rbd_data.296683d1b58ba.000000000000057b [read 0~4194304] 5.2fea8097 e5250) v4 -- ?+0 0x11b9770 con 0x11a6460
  -287> 2013-11-13 16:42:18.735936 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 268 ==== osd_op_reply(1410 rbd_data.296683d1b58ba.000000000000057b [read 0~4194304] v5236'101439 uv101439 ondisk = 0) v4 ==== 138+0+4194304 (1673029482 0 344059011) 0x7f13f00009f0 con 0x11a6460
  -286> 2013-11-13 16:42:18.743684 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.8:6800/24537 -- osd_op(client.314766.0:1411 rbd_data.296683d1b58ba.000000000000057c [read 0~4194304] 5.e27ada1a e5250) v4 -- ?+0 0x11b9770 con 0x11a12f0
  -285> 2013-11-13 16:42:18.812124 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.12 10.0.0.8:6800/24537 315 ==== osd_op_reply(1411 rbd_data.296683d1b58ba.000000000000057c [read 0~4194304] v5236'62216 uv62216 ondisk = 0) v4 ==== 138+0+4194304 (1682302162 0 2604929398) 0x7f1404000ac0 con 0x11a12f0
  -284> 2013-11-13 16:42:18.820173 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1412 rbd_data.296683d1b58ba.000000000000057d [read 0~4194304] 5.174fd028 e5250) v4 -- ?+0 0x11b9770 con 0x11a6460
  -283> 2013-11-13 16:42:18.889562 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 269 ==== osd_op_reply(1412 rbd_data.296683d1b58ba.000000000000057d [read 0~4194304] v5236'9337 uv9337 ondisk = 0) v4 ==== 138+0+4194304 (3845046897 0 2451232555) 0x7f13f00009f0 con 0x11a6460
  -282> 2013-11-13 16:42:18.894294 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1413 rbd_data.296683d1b58ba.000000000000057e [read 0~4194304] 5.5e947e75 e5250) v4 -- ?+0 0x11b9770 con 0x11a7940
  -281> 2013-11-13 16:42:18.957763 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 294 ==== osd_op_reply(1413 rbd_data.296683d1b58ba.000000000000057e [read 0~4194304] v5236'1430 uv1430 ondisk = 0) v4 ==== 138+0+4194304 (3545135902 0 137756266) 0x7f13fc0009f0 con 0x11a7940
  -280> 2013-11-13 16:42:18.962262 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1414 rbd_data.296683d1b58ba.000000000000057f [read 0~4194304] 5.6b4db7db e5250) v4 -- ?+0 0x11b9770 con 0x11a7940
  -279> 2013-11-13 16:42:19.028114 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 295 ==== osd_op_reply(1414 rbd_data.296683d1b58ba.000000000000057f [read 0~4194304] v5236'456993 uv456993 ondisk = 0) v4 ==== 138+0+4194304 (3750484866 0 1580630782) 0x7f13fc0009f0 con 0x11a7940
  -278> 2013-11-13 16:42:19.035259 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.6:6800/21239 -- osd_op(client.314766.0:1415 rbd_data.296683d1b58ba.0000000000000580 [read 0~4194304] 5.91d9846f e5250) v4 -- ?+0 0x11b9770 con 0x11a6460
  -277> 2013-11-13 16:42:19.101695 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.6 10.0.0.6:6800/21239 270 ==== osd_op_reply(1415 rbd_data.296683d1b58ba.0000000000000580 [read 0~4194304] v5236'105982 uv105982 ondisk = 0) v4 ==== 138+0+4194304 (2635554450 0 2664781799) 0x7f13f00009f0 con 0x11a6460
  -276> 2013-11-13 16:42:19.109651 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.4:6800/3857 -- osd_op(client.314766.0:1416 rbd_data.296683d1b58ba.0000000000000581 [read 0~4194304] 5.a7456e30 e5250) v4 -- ?+0 0x11b9770 con 0x11a0240
  -275> 2013-11-13 16:42:19.181101 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.0 10.0.0.4:6800/3857 274 ==== osd_op_reply(1416 rbd_data.296683d1b58ba.0000000000000581 [read 0~4194304] v5236'55977 uv55977 ondisk = 0) v4 ==== 138+0+4194304 (1491455738 0 2822296379) 0x7f140c0009f0 con 0x11a0240
  -274> 2013-11-13 16:42:19.188766 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1417 rbd_data.296683d1b58ba.0000000000000582 [read 0~4194304] 5.d9343950 e5250) v4 -- ?+0 0x11b9770 con 0x11a5ae0
  -273> 2013-11-13 16:42:19.225156 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 263 ==== osd_op_reply(1417 rbd_data.296683d1b58ba.0000000000000582 [read 0~4194304] v0'0 uv10330 ondisk = 0) v6 ==== 198+0+4194304 (2078910139 0 1691932911) 0x7f13f8000c80 con 0x11a5ae0
  -272> 2013-11-13 16:42:19.232393 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1418 rbd_data.296683d1b58ba.0000000000000583 [read 0~4194304] 5.5f951a3d e5250) v4 -- ?+0 0x11b9770 con 0x11a5ae0
  -271> 2013-11-13 16:42:19.268843 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 264 ==== osd_op_reply(1418 rbd_data.296683d1b58ba.0000000000000583 [read 0~4194304] v0'0 uv30833 ondisk = 0) v6 ==== 198+0+4194304 (2272326671 0 1167011551) 0x7f13f8000c40 con 0x11a5ae0
  -270> 2013-11-13 16:42:19.276223 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.5:6800/8449 -- osd_op(client.314766.0:1419 rbd_data.296683d1b58ba.0000000000000584 [read 0~4194304] 5.97385367 e5250) v4 -- ?+0 0x11b9770 con 0x11a7940
  -269> 2013-11-13 16:42:19.344330 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.3 10.0.0.5:6800/8449 296 ==== osd_op_reply(1419 rbd_data.296683d1b58ba.0000000000000584 [read 0~4194304] v5236'17961 uv17961 ondisk = 0) v4 ==== 138+0+4194304 (1181412217 0 3483094531) 0x7f13fc0009f0 con 0x11a7940
  -268> 2013-11-13 16:42:19.351613 7f142d5bc780  1 -- 10.0.0.7:0/1014604 --> 10.0.0.7:6800/12719 -- osd_op(client.314766.0:1420 rbd_data.296683d1b58ba.0000000000000585 [read 0~4194304] 5.5e4916a9 e5250) v4 -- ?+0 0x11b9770 con 0x11a5ae0
  -267> 2013-11-13 16:42:19.393209 7f142bc46700  1 -- 10.0.0.7:0/1014604 <== osd.9 10.0.0.7:6800/12719 265 ==== osd_op_reply(1420 rbd_data.296683d1b58ba.0000000000000585 [read 0~4194304] v0'0 uv53333 ondisk = 0) v6 ==== 198+0+4194304 (3897101456
Actions
Copy link
 #5

Updated by Corin Langosch over 10 years ago

Same happens for another image that cannot be booted anymore:

root@r-ch105:/xfs-drive1/a# rbd export numatrix-ssd/3a9294d9-6408-4415-8bd4-2dc143d5b10e test.img
Exporting image: 42% complete...terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 7f90fdd78700
 ceph version 0.72 (5832e2603c7db5d40b433d0953408993a9b7c217)
 1: rbd() [0x41d200]
 2: (()+0xfcb0) [0x7f91041c7cb0]
 3: (gsignal()+0x35) [0x7f910390e425]
 4: (abort()+0x17b) [0x7f9103911b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9103f10e2d]
 6: (()+0x5ef26) [0x7f9103f0ef26]
 7: (()+0x5ef53) [0x7f9103f0ef53]
 8: (()+0x5f17e) [0x7f9103f0f17e]
 9: (ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned int, unsigned int)+0x24f) [0x7f91046f017f]
 10: (ObjectCacher::_readx(ObjectCacher::OSDRead*, ObjectCacher::ObjectSet*, Context*, bool)+0x1545) [0x7f910542e665]
 11: (ObjectCacher::C_RetryRead::finish(int)+0x20) [0x7f9105436e10]
 12: (Context::complete(int)+0x9) [0x7f91053d0459]
 13: (finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0xa4) [0x7f9105431d64]
 14: (ObjectCacher::bh_read_finish(long, sobject_t, unsigned long, long, unsigned long, ceph::buffer::list&, int, bool)+0x798) [0x7f910542b0b8]
 15: (ObjectCacher::C_ReadFinish::finish(int)+0x7d) [0x7f9105435f5d]
 16: (Context::complete(int)+0x9) [0x7f91053d0459]
 17: (librbd::C_Request::finish(int)+0x85) [0x7f91053f7b15]
 18: (Context::complete(int)+0x9) [0x7f91053d0459]
 19: (librados::C_AioComplete::finish(int)+0x1d) [0x7f9104648e5d]
 20: (Context::complete(int)+0x9) [0x7f91053d0459]
 21: (Finisher::finisher_thread_entry()+0x1c8) [0x7f91046d5528]
 22: (()+0x7e9a) [0x7f91041bfe9a]
 23: (clone()+0x6d) [0x7f91039cc3fd]
2013-11-13 16:48:36.682149 7f90fdd78700 -1 *** Caught signal (Aborted) **
 in thread 7f90fdd78700

 ceph version 0.72 (5832e2603c7db5d40b433d0953408993a9b7c217)
 1: rbd() [0x41d200]
 2: (()+0xfcb0) [0x7f91041c7cb0]
 3: (gsignal()+0x35) [0x7f910390e425]
 4: (abort()+0x17b) [0x7f9103911b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9103f10e2d]
 6: (()+0x5ef26) [0x7f9103f0ef26]
 7: (()+0x5ef53) [0x7f9103f0ef53]
 8: (()+0x5f17e) [0x7f9103f0f17e]
 9: (ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned int, unsigned int)+0x24f) [0x7f91046f017f]
 10: (ObjectCacher::_readx(ObjectCacher::OSDRead*, ObjectCacher::ObjectSet*, Context*, bool)+0x1545) [0x7f910542e665]
 11: (ObjectCacher::C_RetryRead::finish(int)+0x20) [0x7f9105436e10]
 12: (Context::complete(int)+0x9) [0x7f91053d0459]
 13: (finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0xa4) [0x7f9105431d64]
 14: (ObjectCacher::bh_read_finish(long, sobject_t, unsigned long, long, unsigned long, ceph::buffer::list&, int, bool)+0x798) [0x7f910542b0b8]
 15: (ObjectCacher::C_ReadFinish::finish(int)+0x7d) [0x7f9105435f5d]
 16: (Context::complete(int)+0x9) [0x7f91053d0459]
 17: (librbd::C_Request::finish(int)+0x85) [0x7f91053f7b15]
 18: (Context::complete(int)+0x9) [0x7f91053d0459]
 19: (librados::C_AioComplete::finish(int)+0x1d) [0x7f9104648e5d]
 20: (Context::complete(int)+0x9) [0x7f91053d0459]
 21: (Finisher::finisher_thread_entry()+0x1c8) [0x7f91046d5528]
 22: (()+0x7e9a) [0x7f91041bfe9a]
 23: (clone()+0x6d) [0x7f91039cc3fd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -500> 2013-11-13 16:48:15.682366 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:852 rbd_data.1934e2eb141f2.000000000000034d [read 0~4194304] 5.f5488758 e5250) v4 -- ?+0 0x1ce6c80 con 0x1cd2b70
  -499> 2013-11-13 16:48:15.748392 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 172 ==== osd_op_reply(852 rbd_data.1934e2eb141f2.000000000000034d [read 0~4194304] v4376'3432 uv3432 ondisk = 0) v4 ==== 138+0+4194304 (3535933777 0 3425040618) 0x7f90e4000aa0 con 0x1cd2b70
  -498> 2013-11-13 16:48:15.753594 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:853 rbd_data.1934e2eb141f2.000000000000034e [read 0~4194304] 5.62a0cad2 e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -497> 2013-11-13 16:48:15.830418 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 187 ==== osd_op_reply(853 rbd_data.1934e2eb141f2.000000000000034e [read 0~4194304] v4376'20758 uv20758 ondisk = 0) v4 ==== 138+0+4194304 (3797012802 0 1278071264) 0x7f90bc0009f0 con 0x1cda050
  -496> 2013-11-13 16:48:15.835355 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:854 rbd_data.1934e2eb141f2.000000000000034f [read 0~4194304] 5.cbeff00f e5250) v4 -- ?+0 0x1ce6c80 con 0x1cd2270
  -495> 2013-11-13 16:48:15.899694 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 169 ==== osd_op_reply(854 rbd_data.1934e2eb141f2.000000000000034f [read 0~4194304] v4376'15523 uv15523 ondisk = 0) v4 ==== 138+0+4194304 (3225148216 0 478230131) 0x7f90ec000a30 con 0x1cd2270
  -494> 2013-11-13 16:48:15.904200 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:855 rbd_data.1934e2eb141f2.0000000000000350 [read 0~4194304] 5.850cfe44 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2270
  -493> 2013-11-13 16:48:15.968686 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 170 ==== osd_op_reply(855 rbd_data.1934e2eb141f2.0000000000000350 [read 0~4194304] v4376'6206 uv6206 ondisk = 0) v4 ==== 138+0+4194304 (3917622402 0 4282792992) 0x7f90ec0009f0 con 0x1cd2270
  -492> 2013-11-13 16:48:15.973320 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:856 rbd_data.1934e2eb141f2.0000000000000351 [read 0~4194304] 5.321f2652 e5250) v4 -- ?+0 0x1ce6c80 con 0x1cd4b50
  -491> 2013-11-13 16:48:16.014275 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 150 ==== osd_op_reply(856 rbd_data.1934e2eb141f2.0000000000000351 [read 0~4194304] v0'0 uv3172 ondisk = 0) v6 ==== 198+0+4194304 (1580151107 0 503441328) 0x7f90c4000a60 con 0x1cd4b50
  -490> 2013-11-13 16:48:16.019135 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:857 rbd_data.1934e2eb141f2.0000000000000352 [read 0~4194304] 5.59273f4f e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -489> 2013-11-13 16:48:16.098900 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 188 ==== osd_op_reply(857 rbd_data.1934e2eb141f2.0000000000000352 [read 0~4194304] v4376'165858 uv165858 ondisk = 0) v4 ==== 138+0+4194304 (3463453512 0 3450292362) 0x7f90bc0009f0 con 0x1cda050
  -488> 2013-11-13 16:48:16.103858 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:858 rbd_data.1934e2eb141f2.0000000000000353 [read 0~4194304] 5.85c509b2 e5250) v4 -- ?+0 0x1ce6c80 con 0x1cd2b70
  -487> 2013-11-13 16:48:16.173893 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 173 ==== osd_op_reply(858 rbd_data.1934e2eb141f2.0000000000000353 [read 0~4194304] v4376'5520 uv5520 ondisk = 0) v4 ==== 138+0+4194304 (2600437408 0 3498296002) 0x7f90e4000aa0 con 0x1cd2b70
  -486> 2013-11-13 16:48:16.178719 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:859 rbd_data.1934e2eb141f2.0000000000000354 [read 0~4194304] 5.eae3a2eb e5250) v4 -- ?+0 0x1ce3190 con 0x1cd4b50
  -485> 2013-11-13 16:48:16.215982 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 151 ==== osd_op_reply(859 rbd_data.1934e2eb141f2.0000000000000354 [read 0~4194304] v0'0 uv1578 ondisk = 0) v6 ==== 198+0+4194304 (2789450139 0 790443634) 0x7f90c4000aa0 con 0x1cd4b50
  -484> 2013-11-13 16:48:16.220934 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:860 rbd_data.1934e2eb141f2.0000000000000355 [read 0~4194304] 5.cfaa74bf e5250) v4 -- ?+0 0x1ce6c80 con 0x1cd2270
  -483> 2013-11-13 16:48:16.288984 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 171 ==== osd_op_reply(860 rbd_data.1934e2eb141f2.0000000000000355 [read 0~4194304] v4376'8392 uv8392 ondisk = 0) v4 ==== 138+0+4194304 (138703755 0 2299211863) 0x7f90ec0009f0 con 0x1cd2270
  -482> 2013-11-13 16:48:16.293826 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:861 rbd_data.1934e2eb141f2.0000000000000356 [read 0~4194304] 5.d3b70f54 e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -481> 2013-11-13 16:48:16.405837 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 189 ==== osd_op_reply(861 rbd_data.1934e2eb141f2.0000000000000356 [read 0~4194304] v4376'4757 uv4757 ondisk = 0) v4 ==== 138+0+4194304 (3583623913 0 2519861258) 0x7f90bc0009f0 con 0x1cda050
  -480> 2013-11-13 16:48:16.410975 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:862 rbd_data.1934e2eb141f2.0000000000000357 [read 0~4194304] 5.9056babe e5250) v4 -- ?+0 0x1ce6c80 con 0x1cd2270
  -479> 2013-11-13 16:48:16.477803 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 172 ==== osd_op_reply(862 rbd_data.1934e2eb141f2.0000000000000357 [read 0~4194304] v4376'343 uv343 ondisk = 0) v4 ==== 138+0+4194304 (3745024695 0 1387340009) 0x7f90ec0009f0 con 0x1cd2270
  -478> 2013-11-13 16:48:16.482756 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:863 rbd_data.1934e2eb141f2.0000000000000358 [read 0~4194304] 5.9dc14fd1 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2b70
  -477> 2013-11-13 16:48:16.551485 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 174 ==== osd_op_reply(863 rbd_data.1934e2eb141f2.0000000000000358 [read 0~4194304] v4376'111402 uv111402 ondisk = 0) v4 ==== 138+0+4194304 (3273049844 0 881403769) 0x7f90e4000aa0 con 0x1cd2b70
  -476> 2013-11-13 16:48:16.556652 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:864 rbd_data.1934e2eb141f2.0000000000000359 [read 0~4194304] 5.ffead275 e5250) v4 -- ?+0 0x1ce6c80 con 0x1cd4b50
  -475> 2013-11-13 16:48:16.592167 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 152 ==== osd_op_reply(864 rbd_data.1934e2eb141f2.0000000000000359 [read 0~4194304] v0'0 uv210 ondisk = 0) v6 ==== 198+0+4194304 (1403160565 0 1754690108) 0x7f90c4000ae0 con 0x1cd4b50
  -474> 2013-11-13 16:48:16.597023 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:865 rbd_data.1934e2eb141f2.000000000000035a [read 0~4194304] 5.90b1222f e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -473> 2013-11-13 16:48:16.676018 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 190 ==== osd_op_reply(865 rbd_data.1934e2eb141f2.000000000000035a [read 0~4194304] v4376'2243 uv2243 ondisk = 0) v4 ==== 138+0+4194304 (56808853 0 2632491416) 0x7f90bc0009f0 con 0x1cda050
  -472> 2013-11-13 16:48:16.681116 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:866 rbd_data.1934e2eb141f2.000000000000035b [read 0~4194304] 5.208976f9 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2270
  -471> 2013-11-13 16:48:16.747641 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 173 ==== osd_op_reply(866 rbd_data.1934e2eb141f2.000000000000035b [read 0~4194304] v4376'12625 uv12625 ondisk = 0) v4 ==== 138+0+4194304 (1301336309 0 2771415547) 0x7f90ec0009f0 con 0x1cd2270
  -470> 2013-11-13 16:48:16.752521 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:867 rbd_data.1934e2eb141f2.000000000000035c [read 0~4194304] 5.d6325b88 e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -469> 2013-11-13 16:48:16.837084 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 191 ==== osd_op_reply(867 rbd_data.1934e2eb141f2.000000000000035c [read 0~4194304] v4376'362 uv362 ondisk = 0) v4 ==== 138+0+4194304 (3642410531 0 2890875800) 0x7f90bc0009f0 con 0x1cda050
  -468> 2013-11-13 16:48:16.842427 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:868 rbd_data.1934e2eb141f2.000000000000035d [read 0~4194304] 5.76242899 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2b70
  -467> 2013-11-13 16:48:16.909409 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 175 ==== osd_op_reply(868 rbd_data.1934e2eb141f2.000000000000035d [read 0~4194304] v4376'9508 uv9508 ondisk = 0) v4 ==== 138+0+4194304 (3842075333 0 2338956908) 0x7f90e4000aa0 con 0x1cd2b70
  -466> 2013-11-13 16:48:16.916578 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:869 rbd_data.1934e2eb141f2.000000000000035e [read 0~4194304] 5.95a42d94 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2270
  -465> 2013-11-13 16:48:16.983141 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 174 ==== osd_op_reply(869 rbd_data.1934e2eb141f2.000000000000035e [read 0~4194304] v4376'134199 uv134199 ondisk = 0) v4 ==== 138+0+4194304 (3220569391 0 3334994492) 0x7f90ec0009f0 con 0x1cd2270
  -464> 2013-11-13 16:48:16.989983 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:870 rbd_data.1934e2eb141f2.000000000000035f [read 0~4194304] 5.11951c35 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2270
  -463> 2013-11-13 16:48:17.056267 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 175 ==== osd_op_reply(870 rbd_data.1934e2eb141f2.000000000000035f [read 0~4194304] v4376'80631 uv80631 ondisk = 0) v4 ==== 138+0+4194304 (3992236494 0 2715992) 0x7f90ec0009f0 con 0x1cd2270
  -462> 2013-11-13 16:48:17.062637 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:871 rbd_data.1934e2eb141f2.0000000000000360 [read 0~4194304] 5.2c8e2b0c e5250) v4 -- ?+0 0x1ce3190 con 0x1cd4b50
  -461> 2013-11-13 16:48:17.101738 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 153 ==== osd_op_reply(871 rbd_data.1934e2eb141f2.0000000000000360 [read 0~4194304] v0'0 uv13144 ondisk = 0) v6 ==== 198+0+4194304 (939536383 0 3541082134) 0x7f90c4000ae0 con 0x1cd4b50
  -460> 2013-11-13 16:48:17.109096 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:872 rbd_data.1934e2eb141f2.0000000000000361 [read 0~4194304] 5.22e44596 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd37d0
  -459> 2013-11-13 16:48:17.156854 7f90ff67c700 10 monclient: tick
  -458> 2013-11-13 16:48:17.156897 7f90ff67c700 10 monclient: renew subs? (now: 2013-11-13 16:48:17.156897; renew after: 2013-11-13 16:49:34.155455) -- no
  -457> 2013-11-13 16:48:17.178581 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 178 ==== osd_op_reply(872 rbd_data.1934e2eb141f2.0000000000000361 [read 0~4194304] v4376'28232 uv28232 ondisk = 0) v4 ==== 138+0+4194304 (1166224997 0 281993596) 0x7f90d4000be0 con 0x1cd37d0
  -456> 2013-11-13 16:48:17.185902 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:873 rbd_data.1934e2eb141f2.0000000000000362 [read 0~4194304] 5.2ffd82e3 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2b70
  -455> 2013-11-13 16:48:17.255133 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 176 ==== osd_op_reply(873 rbd_data.1934e2eb141f2.0000000000000362 [read 0~4194304] v4376'17688 uv17688 ondisk = 0) v4 ==== 138+0+4194304 (3664046845 0 3825645554) 0x7f90e4000aa0 con 0x1cd2b70
  -454> 2013-11-13 16:48:17.262748 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:874 rbd_data.1934e2eb141f2.0000000000000363 [read 0~4194304] 5.5701c5db e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -453> 2013-11-13 16:48:17.342298 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 192 ==== osd_op_reply(874 rbd_data.1934e2eb141f2.0000000000000363 [read 0~4194304] v4376'17254 uv17254 ondisk = 0) v4 ==== 138+0+4194304 (3502979223 0 855605932) 0x7f90bc0009f0 con 0x1cda050
  -452> 2013-11-13 16:48:17.349712 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:875 rbd_data.1934e2eb141f2.0000000000000364 [read 0~4194304] 5.5a441ae9 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2b70
  -451> 2013-11-13 16:48:17.418242 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 177 ==== osd_op_reply(875 rbd_data.1934e2eb141f2.0000000000000364 [read 0~4194304] v4376'27138 uv27138 ondisk = 0) v4 ==== 138+0+4194304 (1591840621 0 404186716) 0x7f90e4000aa0 con 0x1cd2b70
  -450> 2013-11-13 16:48:17.425382 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:876 rbd_data.1934e2eb141f2.0000000000000365 [read 0~4194304] 5.50ebd1bc e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2270
  -449> 2013-11-13 16:48:17.492802 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 176 ==== osd_op_reply(876 rbd_data.1934e2eb141f2.0000000000000365 [read 0~4194304] v3309'30321 uv30321 ondisk = 0) v4 ==== 138+0+4194304 (117804126 0 3473988332) 0x7f90ec0009f0 con 0x1cd2270
  -448> 2013-11-13 16:48:17.499906 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:877 rbd_data.1934e2eb141f2.0000000000000366 [read 0~4194304] 5.6befb13d e5250) v4 -- ?+0 0x1ce3190 con 0x1cd4b50
  -447> 2013-11-13 16:48:17.538872 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 154 ==== osd_op_reply(877 rbd_data.1934e2eb141f2.0000000000000366 [read 0~4194304] v0'0 uv32740 ondisk = 0) v6 ==== 198+0+4194304 (668925573 0 152071460) 0x7f90c4000ae0 con 0x1cd4b50
  -446> 2013-11-13 16:48:17.546983 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:878 rbd_data.1934e2eb141f2.0000000000000367 [read 0~4194304] 5.6ca29562 e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -445> 2013-11-13 16:48:17.654685 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 193 ==== osd_op_reply(878 rbd_data.1934e2eb141f2.0000000000000367 [read 0~4194304] v4376'2037 uv2037 ondisk = 0) v4 ==== 138+0+4194304 (3925564624 0 207085766) 0x7f90bc0009f0 con 0x1cda050
  -444> 2013-11-13 16:48:17.661854 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:879 rbd_data.1934e2eb141f2.0000000000000368 [read 0~4194304] 5.7a98e713 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd4b50
  -443> 2013-11-13 16:48:17.699613 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 155 ==== osd_op_reply(879 rbd_data.1934e2eb141f2.0000000000000368 [read 0~4194304] v0'0 uv7247 ondisk = 0) v6 ==== 198+0+4194304 (4160394407 0 2962059593) 0x7f90c4000b20 con 0x1cd4b50
  -442> 2013-11-13 16:48:17.708181 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:880 rbd_data.1934e2eb141f2.0000000000000369 [read 0~4194304] 5.ae104b2c e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -441> 2013-11-13 16:48:17.823996 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 194 ==== osd_op_reply(880 rbd_data.1934e2eb141f2.0000000000000369 [read 0~4194304] v4376'6665 uv6665 ondisk = 0) v4 ==== 138+0+4194304 (1325467642 0 253353578) 0x7f90bc0009f0 con 0x1cda050
  -440> 2013-11-13 16:48:17.830453 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:881 rbd_data.1934e2eb141f2.000000000000036a [read 0~4194304] 5.d77c99cd e5250) v4 -- ?+0 0x1ce1800 con 0x1cd37d0
  -439> 2013-11-13 16:48:17.895581 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 179 ==== osd_op_reply(881 rbd_data.1934e2eb141f2.000000000000036a [read 0~4194304] v4376'6076 uv6076 ondisk = 0) v4 ==== 138+0+4194304 (933187038 0 3842113856) 0x7f90d4000be0 con 0x1cd37d0
  -438> 2013-11-13 16:48:17.902804 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:882 rbd_data.1934e2eb141f2.000000000000036b [read 0~4194304] 5.4a20aac8 e5250) v4 -- ?+0 0x1ce1800 con 0x1cda050
  -437> 2013-11-13 16:48:18.000239 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 195 ==== osd_op_reply(882 rbd_data.1934e2eb141f2.000000000000036b [read 0~4194304] v4376'68389 uv68389 ondisk = 0) v4 ==== 138+0+4194304 (2960193946 0 2628093577) 0x7f90bc0009f0 con 0x1cda050
  -436> 2013-11-13 16:48:18.007336 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:883 rbd_data.1934e2eb141f2.000000000000036c [read 0~4194304] 5.ba3ca7f9 e5250) v4 -- ?+0 0x1ce1800 con 0x1cda050
  -435> 2013-11-13 16:48:18.105809 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 196 ==== osd_op_reply(883 rbd_data.1934e2eb141f2.000000000000036c [read 0~4194304] v4376'286154 uv286154 ondisk = 0) v4 ==== 138+0+4194304 (1347456066 0 3214894013) 0x7f90bc0009f0 con 0x1cda050
  -434> 2013-11-13 16:48:18.113664 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:884 rbd_data.1934e2eb141f2.000000000000036d [read 0~4194304] 5.3294349f e5250) v4 -- ?+0 0x1cf4c40 con 0x1cd37d0
  -433> 2013-11-13 16:48:18.180744 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 180 ==== osd_op_reply(884 rbd_data.1934e2eb141f2.000000000000036d [read 0~4194304] v4376'10325 uv10325 ondisk = 0) v4 ==== 138+0+4194304 (1631688210 0 3562004898) 0x7f90d4000be0 con 0x1cd37d0
  -432> 2013-11-13 16:48:18.188050 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:885 rbd_data.1934e2eb141f2.000000000000036e [read 0~4194304] 5.9721759b e5250) v4 -- ?+0 0x1ce3190 con 0x1cda050
  -431> 2013-11-13 16:48:18.269719 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 197 ==== osd_op_reply(885 rbd_data.1934e2eb141f2.000000000000036e [read 0~4194304] v4376'16894 uv16894 ondisk = 0) v4 ==== 138+0+4194304 (3959084472 0 1270893514) 0x7f90bc0009f0 con 0x1cda050
  -430> 2013-11-13 16:48:18.274477 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:886 rbd_data.1934e2eb141f2.000000000000036f [read 0~4194304] 5.f0201240 e5250) v4 -- ?+0 0x1cf4b00 con 0x1cd2b70
  -429> 2013-11-13 16:48:18.345721 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 178 ==== osd_op_reply(886 rbd_data.1934e2eb141f2.000000000000036f [read 0~4194304] v4376'60280 uv60280 ondisk = 0) v4 ==== 138+0+4194304 (3608621838 0 945368009) 0x7f90e4000aa0 con 0x1cd2b70
  -428> 2013-11-13 16:48:18.352532 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:887 rbd_data.1934e2eb141f2.0000000000000370 [read 0~4194304] 5.b4c641e8 e5250) v4 -- ?+0 0x1ce3190 con 0x1cd4b50
  -427> 2013-11-13 16:48:18.392812 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 156 ==== osd_op_reply(887 rbd_data.1934e2eb141f2.0000000000000370 [read 0~4194304] v0'0 uv12198 ondisk = 0) v6 ==== 198+0+4194304 (2309999178 0 2093904980) 0x7f90c4000b20 con 0x1cd4b50
  -426> 2013-11-13 16:48:18.400858 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:888 rbd_data.1934e2eb141f2.0000000000000371 [read 0~4194304] 5.ad56ec7e e5250) v4 -- ?+0 0x1ce3190 con 0x1cd2b70
  -425> 2013-11-13 16:48:18.469397 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 179 ==== osd_op_reply(888 rbd_data.1934e2eb141f2.0000000000000371 [read 0~4194304] v4376'1597 uv1597 ondisk = 0) v4 ==== 138+0+4194304 (2745248894 0 4172664429) 0x7f90e4000aa0 con 0x1cd2b70
  -424> 2013-11-13 16:48:18.476721 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:889 rbd_data.1934e2eb141f2.0000000000000372 [read 0~4194304] 5.d7ed4c2d e5250) v4 -- ?+0 0x1ceb700 con 0x1cda050
  -423> 2013-11-13 16:48:18.555291 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 198 ==== osd_op_reply(889 rbd_data.1934e2eb141f2.0000000000000372 [read 0~4194304] v4376'9203 uv9203 ondisk = 0) v4 ==== 138+0+4194304 (4047856227 0 2589517754) 0x7f90bc0009f0 con 0x1cda050
  -422> 2013-11-13 16:48:18.562341 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:890 rbd_data.1934e2eb141f2.0000000000000373 [read 0~4194304] 5.ac1b412e e5250) v4 -- ?+0 0x1cea610 con 0x1cd2b70
  -421> 2013-11-13 16:48:18.632110 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 180 ==== osd_op_reply(890 rbd_data.1934e2eb141f2.0000000000000373 [read 0~4194304] v4376'2753 uv2753 ondisk = 0) v4 ==== 138+0+4194304 (3015707942 0 4133972012) 0x7f90e4000aa0 con 0x1cd2b70
  -420> 2013-11-13 16:48:18.639244 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:891 rbd_data.1934e2eb141f2.0000000000000374 [read 0~4194304] 5.a9e313f3 e5250) v4 -- ?+0 0x1cea610 con 0x1cd2b70
  -419> 2013-11-13 16:48:18.708153 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 181 ==== osd_op_reply(891 rbd_data.1934e2eb141f2.0000000000000374 [read 0~4194304] v4376'5050 uv5050 ondisk = 0) v4 ==== 138+0+4194304 (519559309 0 1436218756) 0x7f90e4000aa0 con 0x1cd2b70
  -418> 2013-11-13 16:48:18.715463 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:892 rbd_data.1934e2eb141f2.0000000000000375 [read 0~4194304] 5.c5bd431f e5250) v4 -- ?+0 0x1cea610 con 0x1cd2270
  -417> 2013-11-13 16:48:18.785263 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 177 ==== osd_op_reply(892 rbd_data.1934e2eb141f2.0000000000000375 [read 0~4194304] v4376'13027 uv13027 ondisk = 0) v4 ==== 138+0+4194304 (2172520843 0 3007652890) 0x7f90ec0009f0 con 0x1cd2270
  -416> 2013-11-13 16:48:18.792363 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:893 rbd_data.1934e2eb141f2.0000000000000376 [read 0~4194304] 5.e5e35cc7 e5250) v4 -- ?+0 0x1cea610 con 0x1cd4b50
  -415> 2013-11-13 16:48:18.831941 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 157 ==== osd_op_reply(893 rbd_data.1934e2eb141f2.0000000000000376 [read 0~4194304] v0'0 uv1686 ondisk = 0) v6 ==== 198+0+4194304 (2925531923 0 901291772) 0x7f90c4000ae0 con 0x1cd4b50
  -414> 2013-11-13 16:48:18.839387 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:894 rbd_data.1934e2eb141f2.0000000000000377 [read 0~4194304] 5.bb76a710 e5250) v4 -- ?+0 0x1cea610 con 0x1cd37d0
  -413> 2013-11-13 16:48:18.907227 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 181 ==== osd_op_reply(894 rbd_data.1934e2eb141f2.0000000000000377 [read 0~4194304] v4376'358 uv358 ondisk = 0) v4 ==== 138+0+4194304 (2519301773 0 2513321335) 0x7f90d4000be0 con 0x1cd37d0
  -412> 2013-11-13 16:48:18.913768 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:895 rbd_data.1934e2eb141f2.0000000000000378 [read 0~4194304] 5.ecf72cfb e5250) v4 -- ?+0 0x1cebf20 con 0x1cda050
  -411> 2013-11-13 16:48:18.992880 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 199 ==== osd_op_reply(895 rbd_data.1934e2eb141f2.0000000000000378 [read 0~4194304] v4376'3839 uv3839 ondisk = 0) v4 ==== 138+0+4194304 (1478895141 0 434692117) 0x7f90bc0009f0 con 0x1cda050
  -410> 2013-11-13 16:48:18.997552 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:896 rbd_data.1934e2eb141f2.0000000000000379 [read 0~4194304] 5.65831fc8 e5250) v4 -- ?+0 0x1cebf20 con 0x1cd4b50
  -409> 2013-11-13 16:48:19.035088 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 158 ==== osd_op_reply(896 rbd_data.1934e2eb141f2.0000000000000379 [read 0~4194304] v0'0 uv5705 ondisk = 0) v6 ==== 198+0+4194304 (3178025743 0 791254642) 0x7f90c4000ae0 con 0x1cd4b50
  -408> 2013-11-13 16:48:19.041963 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:897 rbd_data.1934e2eb141f2.000000000000037a [read 0~4194304] 5.8d48bfc e5250) v4 -- ?+0 0x1cdb120 con 0x1cd4b50
  -407> 2013-11-13 16:48:19.075237 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 159 ==== osd_op_reply(897 rbd_data.1934e2eb141f2.000000000000037a [read 0~4194304] v0'0 uv73055 ondisk = 0) v6 ==== 198+0+4194304 (3767831645 0 1676247882) 0x7f90c4000b20 con 0x1cd4b50
  -406> 2013-11-13 16:48:19.082588 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:898 rbd_data.1934e2eb141f2.000000000000037b [read 0~4194304] 5.1f308516 e5250) v4 -- ?+0 0x1cdb120 con 0x1cd4b50
  -405> 2013-11-13 16:48:19.116921 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 160 ==== osd_op_reply(898 rbd_data.1934e2eb141f2.000000000000037b [read 0~4194304] v0'0 uv22651 ondisk = 0) v6 ==== 198+0+4194304 (3499169094 0 1014421300) 0x7f90c4000b20 con 0x1cd4b50
  -404> 2013-11-13 16:48:19.121909 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:899 rbd_data.1934e2eb141f2.000000000000037c [read 0~4194304] 5.497d7474 e5250) v4 -- ?+0 0x1cdb120 con 0x1cd4b50
  -403> 2013-11-13 16:48:19.151973 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 161 ==== osd_op_reply(899 rbd_data.1934e2eb141f2.000000000000037c [read 0~4194304] v0'0 uv802 ondisk = 0) v6 ==== 198+0+4194304 (639475533 0 50015109) 0x7f90c4000b20 con 0x1cd4b50
  -402> 2013-11-13 16:48:19.162084 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:900 rbd_data.1934e2eb141f2.000000000000037d [read 0~4194304] 5.e34f37dc e5250) v4 -- ?+0 0x1cdb120 con 0x1cd2270
  -401> 2013-11-13 16:48:19.228548 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 178 ==== osd_op_reply(900 rbd_data.1934e2eb141f2.000000000000037d [read 0~4194304] v4376'2037 uv2037 ondisk = 0) v4 ==== 138+0+4194304 (3910233364 0 4090808914) 0x7f90ec0009f0 con 0x1cd2270
  -400> 2013-11-13 16:48:19.235597 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:901 rbd_data.1934e2eb141f2.000000000000037e [read 0~4194304] 5.5453557d e5250) v4 -- ?+0 0x1cdb120 con 0x1cd37d0
  -399> 2013-11-13 16:48:19.305353 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 182 ==== osd_op_reply(901 rbd_data.1934e2eb141f2.000000000000037e [read 0~4194304] v4376'11720 uv11720 ondisk = 0) v4 ==== 138+0+4194304 (1588617335 0 3144673100) 0x7f90d4000be0 con 0x1cd37d0
  -398> 2013-11-13 16:48:19.312755 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:902 rbd_data.1934e2eb141f2.000000000000037f [read 0~4194304] 5.f1e087a3 e5250) v4 -- ?+0 0x1cdb120 con 0x1cd37d0
  -397> 2013-11-13 16:48:19.382085 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 183 ==== osd_op_reply(902 rbd_data.1934e2eb141f2.000000000000037f [read 0~4194304] v4376'28825 uv28825 ondisk = 0) v4 ==== 138+0+4194304 (2032934611 0 1082328837) 0x7f90d4000be0 con 0x1cd37d0
  -396> 2013-11-13 16:48:19.389415 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:903 rbd_data.1934e2eb141f2.0000000000000380 [read 0~4194304] 5.fb3d6167 e5250) v4 -- ?+0 0x1cdb120 con 0x1cd4b50
  -395> 2013-11-13 16:48:20.805155 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 162 ==== osd_op_reply(903 rbd_data.1934e2eb141f2.0000000000000380 [read 0~4194304] v0'0 uv9181 ondisk = 0) v6 ==== 198+0+4194304 (3638458679 0 3123547680) 0x7f90c4000b20 con 0x1cd4b50
  -394> 2013-11-13 16:48:20.812426 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:904 rbd_data.1934e2eb141f2.0000000000000381 [read 0~4194304] 5.f8cce761 e5250) v4 -- ?+0 0x1cdb120 con 0x1cd2270
  -393> 2013-11-13 16:48:20.882280 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 179 ==== osd_op_reply(904 rbd_data.1934e2eb141f2.0000000000000381 [read 0~4194304] v4376'19147 uv19147 ondisk = 0) v4 ==== 138+0+4194304 (4079737682 0 1599253064) 0x7f90ec0009f0 con 0x1cd2270
  -392> 2013-11-13 16:48:20.889373 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:905 rbd_data.1934e2eb141f2.0000000000000382 [read 0~4194304] 5.92e4a454 e5250) v4 -- ?+0 0x1cdb120 con 0x1cd2270
  -391> 2013-11-13 16:48:20.956478 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 180 ==== osd_op_reply(905 rbd_data.1934e2eb141f2.0000000000000382 [read 0~4194304] v4376'23677 uv23677 ondisk = 0) v4 ==== 138+0+4194304 (534035134 0 3831142571) 0x7f90ec0009f0 con 0x1cd2270
  -390> 2013-11-13 16:48:20.962978 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:906 rbd_data.1934e2eb141f2.0000000000000383 [read 0~4194304] 5.5dba394b e5250) v4 -- ?+0 0x1cdb120 con 0x1cd4b50
  -389> 2013-11-13 16:48:21.429766 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 163 ==== osd_op_reply(906 rbd_data.1934e2eb141f2.0000000000000383 [read 0~4194304] v0'0 uv52389 ondisk = 0) v6 ==== 198+0+4194304 (3535530041 0 541176447) 0x7f90c4000a20 con 0x1cd4b50
  -388> 2013-11-13 16:48:21.438479 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:907 rbd_data.1934e2eb141f2.0000000000000384 [read 0~4194304] 5.cc094193 e5250) v4 -- ?+0 0x1ced030 con 0x1cd2270
  -387> 2013-11-13 16:48:21.505756 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 181 ==== osd_op_reply(907 rbd_data.1934e2eb141f2.0000000000000384 [read 0~4194304] v4376'12166 uv12166 ondisk = 0) v4 ==== 138+0+4194304 (2291402489 0 2261937872) 0x7f90ec0009f0 con 0x1cd2270
  -386> 2013-11-13 16:48:21.514438 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:908 rbd_data.1934e2eb141f2.0000000000000385 [read 0~4194304] 5.4c76ece2 e5250) v4 -- ?+0 0x1ced030 con 0x1cda050
  -385> 2013-11-13 16:48:21.596980 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 200 ==== osd_op_reply(908 rbd_data.1934e2eb141f2.0000000000000385 [read 0~4194304] v4376'9273 uv9273 ondisk = 0) v4 ==== 138+0+4194304 (3208627646 0 3847304462) 0x7f90bc0009f0 con 0x1cda050
  -384> 2013-11-13 16:48:21.604067 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:909 rbd_data.1934e2eb141f2.0000000000000386 [read 0~4194304] 5.81638b4a e5250) v4 -- ?+0 0x1ced030 con 0x1cd2270
  -383> 2013-11-13 16:48:21.670825 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 182 ==== osd_op_reply(909 rbd_data.1934e2eb141f2.0000000000000386 [read 0~4194304] v4376'10959 uv10959 ondisk = 0) v4 ==== 138+0+4194304 (3741333785 0 3761876560) 0x7f90ec0009f0 con 0x1cd2270
  -382> 2013-11-13 16:48:21.676372 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:910 rbd_data.1934e2eb141f2.0000000000000387 [read 0~4194304] 5.709274db e5250) v4 -- ?+0 0x1ced030 con 0x1cd2270
  -381> 2013-11-13 16:48:21.743857 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 183 ==== osd_op_reply(910 rbd_data.1934e2eb141f2.0000000000000387 [read 0~4194304] v4376'4394568 uv4394568 ondisk = 0) v4 ==== 138+0+4194304 (1494525275 0 610972986) 0x7f90ec0009f0 con 0x1cd2270
  -380> 2013-11-13 16:48:21.751035 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:911 rbd_data.1934e2eb141f2.0000000000000388 [read 0~4194304] 5.79f960a5 e5250) v4 -- ?+0 0x1ced030 con 0x1cd37d0
  -379> 2013-11-13 16:48:21.819979 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 184 ==== osd_op_reply(911 rbd_data.1934e2eb141f2.0000000000000388 [read 0~4194304] v4376'42941 uv42941 ondisk = 0) v4 ==== 138+0+4194304 (432890600 0 1414324632) 0x7f90d4000ac0 con 0x1cd37d0
  -378> 2013-11-13 16:48:21.827245 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:912 rbd_data.1934e2eb141f2.0000000000000389 [read 0~4194304] 5.efa77d54 e5250) v4 -- ?+0 0x1ced030 con 0x1cd37d0
  -377> 2013-11-13 16:48:21.893048 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 185 ==== osd_op_reply(912 rbd_data.1934e2eb141f2.0000000000000389 [read 0~4194304] v4376'37749 uv37749 ondisk = 0) v4 ==== 138+0+4194304 (1747992384 0 312762026) 0x7f90d4001100 con 0x1cd37d0
  -376> 2013-11-13 16:48:21.900097 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:913 rbd_data.1934e2eb141f2.000000000000038a [read 0~4194304] 5.6db770c6 e5250) v4 -- ?+0 0x1ced030 con 0x1cd37d0
  -375> 2013-11-13 16:48:21.967459 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 186 ==== osd_op_reply(913 rbd_data.1934e2eb141f2.000000000000038a [read 0~4194304] v4376'8132 uv8132 ondisk = 0) v4 ==== 138+0+4194304 (3968816410 0 4206017392) 0x7f90d4000b80 con 0x1cd37d0
  -374> 2013-11-13 16:48:21.974433 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:914 rbd_data.1934e2eb141f2.000000000000038b [read 0~4194304] 5.a2a36f5d e5250) v4 -- ?+0 0x1ced030 con 0x1cd2b70
  -373> 2013-11-13 16:48:22.045514 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 182 ==== osd_op_reply(914 rbd_data.1934e2eb141f2.000000000000038b [read 0~4194304] v4376'8477 uv8477 ondisk = 0) v4 ==== 138+0+4194304 (1471228990 0 2311578798) 0x7f90e4000aa0 con 0x1cd2b70
  -372> 2013-11-13 16:48:22.052596 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:915 rbd_data.1934e2eb141f2.000000000000038c [read 0~4194304] 5.e7646891 e5250) v4 -- ?+0 0x1ced030 con 0x1cd4b50
  -371> 2013-11-13 16:48:22.095585 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 164 ==== osd_op_reply(915 rbd_data.1934e2eb141f2.000000000000038c [read 0~4194304] v0'0 uv11993 ondisk = 0) v6 ==== 198+0+4194304 (3088748991 0 263935271) 0x7f90c4000aa0 con 0x1cd4b50
  -370> 2013-11-13 16:48:22.100370 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:916 rbd_data.1934e2eb141f2.000000000000038d [read 0~4194304] 5.b86d0596 e5250) v4 -- ?+0 0x1ced030 con 0x1cd37d0
  -369> 2013-11-13 16:48:22.170724 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 187 ==== osd_op_reply(916 rbd_data.1934e2eb141f2.000000000000038d [read 0~4194304] v4376'28177 uv28177 ondisk = 0) v4 ==== 138+0+4194304 (4277884831 0 4280856827) 0x7f90d4000b80 con 0x1cd37d0
  -368> 2013-11-13 16:48:22.178163 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:917 rbd_data.1934e2eb141f2.000000000000038e [read 0~4194304] 5.3dd35525 e5250) v4 -- ?+0 0x1ced030 con 0x1cda050
  -367> 2013-11-13 16:48:22.274556 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 201 ==== osd_op_reply(917 rbd_data.1934e2eb141f2.000000000000038e [read 0~4194304] v4376'134898 uv134898 ondisk = 0) v4 ==== 138+0+4194304 (2695937783 0 3846545740) 0x7f90bc0009f0 con 0x1cda050
  -366> 2013-11-13 16:48:22.281806 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:918 rbd_data.1934e2eb141f2.000000000000038f [read 0~4194304] 5.1c9bf547 e5250) v4 -- ?+0 0x1ced030 con 0x1cd2b70
  -365> 2013-11-13 16:48:22.352477 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 183 ==== osd_op_reply(918 rbd_data.1934e2eb141f2.000000000000038f [read 0~4194304] v4376'14112 uv14112 ondisk = 0) v4 ==== 138+0+4194304 (4203646982 0 1587324289) 0x7f90e4000aa0 con 0x1cd2b70
  -364> 2013-11-13 16:48:22.359343 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:919 rbd_data.1934e2eb141f2.0000000000000390 [read 0~4194304] 5.e7bfe3e5 e5250) v4 -- ?+0 0x1ced030 con 0x1cda050
  -363> 2013-11-13 16:48:22.491458 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 202 ==== osd_op_reply(919 rbd_data.1934e2eb141f2.0000000000000390 [read 0~4194304] v4376'12453 uv12453 ondisk = 0) v4 ==== 138+0+4194304 (1288870335 0 1735333168) 0x7f90bc0009f0 con 0x1cda050
  -362> 2013-11-13 16:48:22.498295 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:920 rbd_data.1934e2eb141f2.0000000000000391 [read 0~4194304] 5.ca314dc2 e5250) v4 -- ?+0 0x1ced030 con 0x1cda050
  -361> 2013-11-13 16:48:22.577706 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 203 ==== osd_op_reply(920 rbd_data.1934e2eb141f2.0000000000000391 [read 0~4194304] v4376'3471 uv3471 ondisk = 0) v4 ==== 138+0+4194304 (1535885326 0 3988362888) 0x7f90bc0009f0 con 0x1cda050
  -360> 2013-11-13 16:48:22.582967 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:921 rbd_data.1934e2eb141f2.0000000000000392 [read 0~4194304] 5.3d11639b e5250) v4 -- ?+0 0x1ced030 con 0x1cd4b50
  -359> 2013-11-13 16:48:22.619515 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 165 ==== osd_op_reply(921 rbd_data.1934e2eb141f2.0000000000000392 [read 0~4194304] v0'0 uv5164 ondisk = 0) v6 ==== 198+0+4194304 (4003135784 0 2574791577) 0x7f90c4000ae0 con 0x1cd4b50
  -358> 2013-11-13 16:48:22.624033 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:922 rbd_data.1934e2eb141f2.0000000000000393 [read 0~4194304] 5.d9481acf e5250) v4 -- ?+0 0x1cea610 con 0x1cd2b70
  -357> 2013-11-13 16:48:22.690803 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 184 ==== osd_op_reply(922 rbd_data.1934e2eb141f2.0000000000000393 [read 0~4194304] v4376'31668 uv31668 ondisk = 0) v4 ==== 138+0+4194304 (1856884028 0 889164423) 0x7f90e4000aa0 con 0x1cd2b70
  -356> 2013-11-13 16:48:22.698082 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:923 rbd_data.1934e2eb141f2.0000000000000394 [read 0~4194304] 5.21e7a295 e5250) v4 -- ?+0 0x1cea610 con 0x1cd4b50
  -355> 2013-11-13 16:48:22.734108 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 166 ==== osd_op_reply(923 rbd_data.1934e2eb141f2.0000000000000394 [read 0~4194304] v0'0 uv17756 ondisk = 0) v6 ==== 198+0+4194304 (1166740336 0 380981524) 0x7f90c4000b20 con 0x1cd4b50
  -354> 2013-11-13 16:48:22.741549 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:924 rbd_data.1934e2eb141f2.0000000000000395 [read 0~4194304] 5.43c22d9 e5250) v4 -- ?+0 0x1cea610 con 0x1cd2270
  -353> 2013-11-13 16:48:22.811714 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 184 ==== osd_op_reply(924 rbd_data.1934e2eb141f2.0000000000000395 [read 0~4194304] v4376'9755 uv9755 ondisk = 0) v4 ==== 138+0+4194304 (1481525674 0 2116029211) 0x7f90ec0009f0 con 0x1cd2270
  -352> 2013-11-13 16:48:22.818908 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:925 rbd_data.1934e2eb141f2.0000000000000396 [read 0~4194304] 5.c90a618b e5250) v4 -- ?+0 0x1cea610 con 0x1cd2b70
  -351> 2013-11-13 16:48:22.887522 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 185 ==== osd_op_reply(925 rbd_data.1934e2eb141f2.0000000000000396 [read 0~4194304] v4376'333331 uv333331 ondisk = 0) v4 ==== 138+0+4194304 (3395130096 0 3518988164) 0x7f90e4000aa0 con 0x1cd2b70
  -350> 2013-11-13 16:48:22.894311 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:926 rbd_data.1934e2eb141f2.0000000000000397 [read 0~4194304] 5.b56c8cac e5250) v4 -- ?+0 0x1cea610 con 0x1cd2b70
  -349> 2013-11-13 16:48:22.963072 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 186 ==== osd_op_reply(926 rbd_data.1934e2eb141f2.0000000000000397 [read 0~4194304] v4376'426 uv426 ondisk = 0) v4 ==== 138+0+4194304 (2281507457 0 3218322113) 0x7f90e4000aa0 con 0x1cd2b70
  -348> 2013-11-13 16:48:22.970306 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:927 rbd_data.1934e2eb141f2.0000000000000398 [read 0~4194304] 5.c675665b e5250) v4 -- ?+0 0x1cea610 con 0x1cd37d0
  -347> 2013-11-13 16:48:23.040457 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 188 ==== osd_op_reply(927 rbd_data.1934e2eb141f2.0000000000000398 [read 0~4194304] v4376'83231 uv83231 ondisk = 0) v4 ==== 138+0+4194304 (2321206970 0 642743990) 0x7f90d4000b80 con 0x1cd37d0
  -346> 2013-11-13 16:48:23.047637 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:928 rbd_data.1934e2eb141f2.0000000000000399 [read 0~4194304] 5.abc4f356 e5250) v4 -- ?+0 0x1cea610 con 0x1cd4b50
  -345> 2013-11-13 16:48:23.085586 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 167 ==== osd_op_reply(928 rbd_data.1934e2eb141f2.0000000000000399 [read 0~4194304] v0'0 uv772238 ondisk = 0) v6 ==== 198+0+4194304 (2916825971 0 1582177380) 0x7f90c4000b20 con 0x1cd4b50
  -344> 2013-11-13 16:48:23.092722 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:929 rbd_data.1934e2eb141f2.000000000000039a [read 0~4194304] 5.f32691b2 e5250) v4 -- ?+0 0x1cece50 con 0x1cd2270
  -343> 2013-11-13 16:48:23.160268 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 185 ==== osd_op_reply(929 rbd_data.1934e2eb141f2.000000000000039a [read 0~4194304] v4376'8887 uv8887 ondisk = 0) v4 ==== 138+0+4194304 (1113859881 0 228335256) 0x7f90ec0009f0 con 0x1cd2270
  -342> 2013-11-13 16:48:23.167821 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:930 rbd_data.1934e2eb141f2.000000000000039b [read 0~4194304] 5.bb944946 e5250) v4 -- ?+0 0x1cece50 con 0x1cd37d0
  -341> 2013-11-13 16:48:23.236023 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 189 ==== osd_op_reply(930 rbd_data.1934e2eb141f2.000000000000039b [read 0~4194304] v4376'45186 uv45186 ondisk = 0) v4 ==== 138+0+4194304 (401300980 0 2322285617) 0x7f90d4000b80 con 0x1cd37d0
  -340> 2013-11-13 16:48:23.242691 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:931 rbd_data.1934e2eb141f2.000000000000039c [read 0~4194304] 5.9522892c e5250) v4 -- ?+0 0x1cece50 con 0x1cda050
  -339> 2013-11-13 16:48:23.377707 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 204 ==== osd_op_reply(931 rbd_data.1934e2eb141f2.000000000000039c [read 0~4194304] v4376'1225 uv1225 ondisk = 0) v4 ==== 138+0+4194304 (3564352301 0 2838743200) 0x7f90bc0009f0 con 0x1cda050
  -338> 2013-11-13 16:48:23.385036 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:932 rbd_data.1934e2eb141f2.000000000000039d [read 0~4194304] 5.22e7caac e5250) v4 -- ?+0 0x1cece50 con 0x1cd2270
  -337> 2013-11-13 16:48:23.453312 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 186 ==== osd_op_reply(932 rbd_data.1934e2eb141f2.000000000000039d [read 0~4194304] v4376'1643 uv1643 ondisk = 0) v4 ==== 138+0+4194304 (1926208100 0 2375263728) 0x7f90ec0009f0 con 0x1cd2270
  -336> 2013-11-13 16:48:23.460650 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:933 rbd_data.1934e2eb141f2.000000000000039e [read 0~4194304] 5.45eef212 e5250) v4 -- ?+0 0x1cece50 con 0x1cd37d0
  -335> 2013-11-13 16:48:23.526798 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 190 ==== osd_op_reply(933 rbd_data.1934e2eb141f2.000000000000039e [read 0~4194304] v4376'480065 uv480065 ondisk = 0) v4 ==== 138+0+4194304 (1619999326 0 1545890685) 0x7f90d4000b80 con 0x1cd37d0
  -334> 2013-11-13 16:48:23.531678 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:934 rbd_data.1934e2eb141f2.000000000000039f [read 0~4194304] 5.b7fb7693 e5250) v4 -- ?+0 0x1cece50 con 0x1cd37d0
  -333> 2013-11-13 16:48:23.596192 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 191 ==== osd_op_reply(934 rbd_data.1934e2eb141f2.000000000000039f [read 0~4194304] v4376'14611 uv14611 ondisk = 0) v4 ==== 138+0+4194304 (2091484327 0 3630853114) 0x7f90d4000b80 con 0x1cd37d0
  -332> 2013-11-13 16:48:23.603564 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:935 rbd_data.1934e2eb141f2.00000000000003a0 [read 0~4194304] 5.b14d915e e5250) v4 -- ?+0 0x1cece50 con 0x1cd4b50
  -331> 2013-11-13 16:48:23.642487 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 168 ==== osd_op_reply(935 rbd_data.1934e2eb141f2.00000000000003a0 [read 0~4194304] v0'0 uv16813 ondisk = 0) v6 ==== 198+0+4194304 (1053680742 0 3701061939) 0x7f90c4000aa0 con 0x1cd4b50
  -330> 2013-11-13 16:48:23.649695 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:936 rbd_data.1934e2eb141f2.00000000000003a1 [read 0~4194304] 5.9e6de154 e5250) v4 -- ?+0 0x1cece50 con 0x1cd2270
  -329> 2013-11-13 16:48:23.718482 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 187 ==== osd_op_reply(936 rbd_data.1934e2eb141f2.00000000000003a1 [read 0~4194304] v4376'6807 uv6807 ondisk = 0) v4 ==== 138+0+4194304 (616661650 0 3449391028) 0x7f90ec0009f0 con 0x1cd2270
  -328> 2013-11-13 16:48:23.726058 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:937 rbd_data.1934e2eb141f2.00000000000003a2 [read 0~4194304] 5.f7bd28a8 e5250) v4 -- ?+0 0x1cece50 con 0x1cda050
  -327> 2013-11-13 16:48:23.806655 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 205 ==== osd_op_reply(937 rbd_data.1934e2eb141f2.00000000000003a2 [read 0~4194304] v4376'930 uv930 ondisk = 0) v4 ==== 138+0+4194304 (2431411470 0 3977455118) 0x7f90bc0009f0 con 0x1cda050
  -326> 2013-11-13 16:48:23.814322 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:938 rbd_data.1934e2eb141f2.00000000000003a3 [read 0~4194304] 5.1407842f e5250) v4 -- ?+0 0x1cece50 con 0x1cd4b50
  -325> 2013-11-13 16:48:23.854743 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 169 ==== osd_op_reply(938 rbd_data.1934e2eb141f2.00000000000003a3 [read 0~4194304] v0'0 uv6364 ondisk = 0) v6 ==== 198+0+4194304 (1094905553 0 1319325230) 0x7f90c4000aa0 con 0x1cd4b50
  -324> 2013-11-13 16:48:23.859475 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:939 rbd_data.1934e2eb141f2.00000000000003a4 [read 0~4194304] 5.e1f11b0d e5250) v4 -- ?+0 0x1cece50 con 0x1cd4b50
  -323> 2013-11-13 16:48:23.892626 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 170 ==== osd_op_reply(939 rbd_data.1934e2eb141f2.00000000000003a4 [read 0~4194304] v0'0 uv64548 ondisk = 0) v6 ==== 198+0+4194304 (1945975870 0 2687701758) 0x7f90c4000ae0 con 0x1cd4b50
  -322> 2013-11-13 16:48:23.897751 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:940 rbd_data.1934e2eb141f2.00000000000003a5 [read 0~4194304] 5.7e9fb7c9 e5250) v4 -- ?+0 0x1cece50 con 0x1cd4b50
  -321> 2013-11-13 16:48:23.933303 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 171 ==== osd_op_reply(940 rbd_data.1934e2eb141f2.00000000000003a5 [read 0~4194304] v0'0 uv1683 ondisk = 0) v6 ==== 198+0+4194304 (2905165134 0 1869440685) 0x7f90c4000b20 con 0x1cd4b50
  -320> 2013-11-13 16:48:23.941236 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:941 rbd_data.1934e2eb141f2.00000000000003a6 [read 0~4194304] 5.52105999 e5250) v4 -- ?+0 0x1cece50 con 0x1cd4b50
  -319> 2013-11-13 16:48:23.978481 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 172 ==== osd_op_reply(941 rbd_data.1934e2eb141f2.00000000000003a6 [read 0~4194304] v0'0 uv1181 ondisk = 0) v6 ==== 198+0+4194304 (3732467511 0 2156876228) 0x7f90c4000b20 con 0x1cd4b50
  -318> 2013-11-13 16:48:23.986253 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:942 rbd_data.1934e2eb141f2.00000000000003a7 [read 0~4194304] 5.a547e2e8 e5250) v4 -- ?+0 0x1cece50 con 0x1cd2270
  -317> 2013-11-13 16:48:24.053774 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 188 ==== osd_op_reply(942 rbd_data.1934e2eb141f2.00000000000003a7 [read 0~4194304] v4376'20400 uv20400 ondisk = 0) v4 ==== 138+0+4194304 (2202315551 0 1243768196) 0x7f90ec0009f0 con 0x1cd2270
  -316> 2013-11-13 16:48:24.062124 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:943 rbd_data.1934e2eb141f2.00000000000003a8 [read 0~4194304] 5.9df762f9 e5250) v4 -- ?+0 0x1cece50 con 0x1cd2b70
  -315> 2013-11-13 16:48:24.132555 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 187 ==== osd_op_reply(943 rbd_data.1934e2eb141f2.00000000000003a8 [read 0~4194304] v4376'349 uv349 ondisk = 0) v4 ==== 138+0+4194304 (578709019 0 2734511582) 0x7f90e4000aa0 con 0x1cd2b70
  -314> 2013-11-13 16:48:24.141228 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:944 rbd_data.1934e2eb141f2.00000000000003a9 [read 0~4194304] 5.d16227c4 e5250) v4 -- ?+0 0x1cece50 con 0x1cd2b70
  -313> 2013-11-13 16:48:24.210099 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 188 ==== osd_op_reply(944 rbd_data.1934e2eb141f2.00000000000003a9 [read 0~4194304] v4376'20160 uv20160 ondisk = 0) v4 ==== 138+0+4194304 (2349545884 0 953781374) 0x7f90e4000aa0 con 0x1cd2b70
  -312> 2013-11-13 16:48:24.218145 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:945 rbd_data.1934e2eb141f2.00000000000003aa [read 0~4194304] 5.1d05ce1b e5250) v4 -- ?+0 0x1cece50 con 0x1cd2b70
  -311> 2013-11-13 16:48:24.286707 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 189 ==== osd_op_reply(945 rbd_data.1934e2eb141f2.00000000000003aa [read 0~4194304] v4376'10936 uv10936 ondisk = 0) v4 ==== 138+0+4194304 (3291723556 0 638017556) 0x7f90e4000aa0 con 0x1cd2b70
  -310> 2013-11-13 16:48:24.294970 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:946 rbd_data.1934e2eb141f2.00000000000003ab [read 0~4194304] 5.32a8fdd6 e5250) v4 -- ?+0 0x1cece50 con 0x1cda050
  -309> 2013-11-13 16:48:24.374537 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 206 ==== osd_op_reply(946 rbd_data.1934e2eb141f2.00000000000003ab [read 0~4194304] v4376'722364 uv722364 ondisk = 0) v4 ==== 138+0+4194304 (1458225835 0 4060189350) 0x7f90bc0009f0 con 0x1cda050
  -308> 2013-11-13 16:48:24.382277 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:947 rbd_data.1934e2eb141f2.00000000000003ac [read 0~4194304] 5.82c8fcb9 e5250) v4 -- ?+0 0x1cece50 con 0x1cda050
  -307> 2013-11-13 16:48:24.464115 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 207 ==== osd_op_reply(947 rbd_data.1934e2eb141f2.00000000000003ac [read 0~4194304] v4376'4255 uv4255 ondisk = 0) v4 ==== 138+0+4194304 (1871109802 0 206287591) 0x7f90bc0009f0 con 0x1cda050
  -306> 2013-11-13 16:48:24.471850 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:948 rbd_data.1934e2eb141f2.00000000000003ad [read 0~4194304] 5.31b1a6c5 e5250) v4 -- ?+0 0x1cece50 con 0x1cd2b70
  -305> 2013-11-13 16:48:24.540204 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 190 ==== osd_op_reply(948 rbd_data.1934e2eb141f2.00000000000003ad [read 0~4194304] v4376'57738 uv57738 ondisk = 0) v4 ==== 138+0+4194304 (739703741 0 3702993126) 0x7f90e4000aa0 con 0x1cd2b70
  -304> 2013-11-13 16:48:24.548699 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:949 rbd_data.1934e2eb141f2.00000000000003ae [read 0~4194304] 5.ccef3225 e5250) v4 -- ?+0 0x1cece50 con 0x1cd2b70
  -303> 2013-11-13 16:48:24.618387 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 191 ==== osd_op_reply(949 rbd_data.1934e2eb141f2.00000000000003ae [read 0~4194304] v4376'20911 uv20911 ondisk = 0) v4 ==== 138+0+4194304 (1333891316 0 2906004387) 0x7f90e4000aa0 con 0x1cd2b70
  -302> 2013-11-13 16:48:24.626631 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:950 rbd_data.1934e2eb141f2.00000000000003af [read 0~4194304] 5.c7f9d1cd e5250) v4 -- ?+0 0x1cf1290 con 0x1cd4b50
  -301> 2013-11-13 16:48:24.662769 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 173 ==== osd_op_reply(950 rbd_data.1934e2eb141f2.00000000000003af [read 0~4194304] v0'0 uv518 ondisk = 0) v6 ==== 198+0+4194304 (3963328663 0 3292512362) 0x7f90c4000a20 con 0x1cd4b50
  -300> 2013-11-13 16:48:24.667754 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:951 rbd_data.1934e2eb141f2.00000000000003b0 [read 0~4194304] 5.d22d5434 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd2b70
  -299> 2013-11-13 16:48:24.731840 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 192 ==== osd_op_reply(951 rbd_data.1934e2eb141f2.00000000000003b0 [read 0~4194304] v4376'7848 uv7848 ondisk = 0) v4 ==== 138+0+4194304 (2425450346 0 2458664035) 0x7f90e4000aa0 con 0x1cd2b70
  -298> 2013-11-13 16:48:24.739719 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:952 rbd_data.1934e2eb141f2.00000000000003b1 [read 0~4194304] 5.19ffde79 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd2b70
  -297> 2013-11-13 16:48:24.807321 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 193 ==== osd_op_reply(952 rbd_data.1934e2eb141f2.00000000000003b1 [read 0~4194304] v4376'13730 uv13730 ondisk = 0) v4 ==== 138+0+4194304 (1360183627 0 651077784) 0x7f90e4000aa0 con 0x1cd2b70
  -296> 2013-11-13 16:48:24.814624 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:953 rbd_data.1934e2eb141f2.00000000000003b2 [read 0~4194304] 5.25cd68f e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd4b50
  -295> 2013-11-13 16:48:24.849418 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 174 ==== osd_op_reply(953 rbd_data.1934e2eb141f2.00000000000003b2 [read 0~4194304] v0'0 uv18085 ondisk = 0) v6 ==== 198+0+4194304 (2363478633 0 1922542185) 0x7f90c4000a20 con 0x1cd4b50
  -294> 2013-11-13 16:48:24.857684 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:954 rbd_data.1934e2eb141f2.00000000000003b3 [read 0~4194304] 5.65229e36 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd2270
  -293> 2013-11-13 16:48:24.927120 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 189 ==== osd_op_reply(954 rbd_data.1934e2eb141f2.00000000000003b3 [read 0~4194304] v4376'50714 uv50714 ondisk = 0) v4 ==== 138+0+4194304 (3151971387 0 740711562) 0x7f90ec0009f0 con 0x1cd2270
  -292> 2013-11-13 16:48:24.934691 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:955 rbd_data.1934e2eb141f2.00000000000003b4 [read 0~4194304] 5.e3b6de7 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cda050
  -291> 2013-11-13 16:48:25.013455 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 208 ==== osd_op_reply(955 rbd_data.1934e2eb141f2.00000000000003b4 [read 0~4194304] v4376'858 uv858 ondisk = 0) v4 ==== 138+0+4194304 (4197240480 0 3870526351) 0x7f90bc0009f0 con 0x1cda050
  -290> 2013-11-13 16:48:25.021535 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:956 rbd_data.1934e2eb141f2.00000000000003b5 [read 0~4194304] 5.35f8564b e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd37d0
  -289> 2013-11-13 16:48:25.092479 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 192 ==== osd_op_reply(956 rbd_data.1934e2eb141f2.00000000000003b5 [read 0~4194304] v4376'64296 uv64296 ondisk = 0) v4 ==== 138+0+4194304 (3330403147 0 1075988369) 0x7f90d40011e0 con 0x1cd37d0
  -288> 2013-11-13 16:48:25.098244 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:957 rbd_data.1934e2eb141f2.00000000000003b6 [read 0~4194304] 5.b294db15 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd2270
  -287> 2013-11-13 16:48:25.166866 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 190 ==== osd_op_reply(957 rbd_data.1934e2eb141f2.00000000000003b6 [read 0~4194304] v4376'3955 uv3955 ondisk = 0) v4 ==== 138+0+4194304 (3302087652 0 2226743564) 0x7f90ec0009f0 con 0x1cd2270
  -286> 2013-11-13 16:48:25.172446 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:958 rbd_data.1934e2eb141f2.00000000000003b7 [read 0~4194304] 5.144ad913 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd2270
  -285> 2013-11-13 16:48:25.240376 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 191 ==== osd_op_reply(958 rbd_data.1934e2eb141f2.00000000000003b7 [read 0~4194304] v4376'7680 uv7680 ondisk = 0) v4 ==== 138+0+4194304 (1533908371 0 3729685873) 0x7f90ec0009f0 con 0x1cd2270
  -284> 2013-11-13 16:48:25.248462 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:959 rbd_data.1934e2eb141f2.00000000000003b8 [read 0~4194304] 5.2ceacd48 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cda050
  -283> 2013-11-13 16:48:25.330094 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 209 ==== osd_op_reply(959 rbd_data.1934e2eb141f2.00000000000003b8 [read 0~4194304] v4376'3845 uv3845 ondisk = 0) v4 ==== 138+0+4194304 (3155004994 0 1394121416) 0x7f90bc0009f0 con 0x1cda050
  -282> 2013-11-13 16:48:25.338163 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:960 rbd_data.1934e2eb141f2.00000000000003b9 [read 0~4194304] 5.20f07eef e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd2270
  -281> 2013-11-13 16:48:25.406184 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 192 ==== osd_op_reply(960 rbd_data.1934e2eb141f2.00000000000003b9 [read 0~4194304] v4376'20706 uv20706 ondisk = 0) v4 ==== 138+0+4194304 (1561888466 0 1262578878) 0x7f90ec0009f0 con 0x1cd2270
  -280> 2013-11-13 16:48:25.414349 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:961 rbd_data.1934e2eb141f2.00000000000003ba [read 0~4194304] 5.1f21afba e5250) v4 -- ?+0 0x1ceeed0 con 0x1cda050
  -279> 2013-11-13 16:48:25.492693 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 210 ==== osd_op_reply(961 rbd_data.1934e2eb141f2.00000000000003ba [read 0~4194304] v4376'5482 uv5482 ondisk = 0) v4 ==== 138+0+4194304 (1736757006 0 559857522) 0x7f90bc0009f0 con 0x1cda050
  -278> 2013-11-13 16:48:25.497452 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:962 rbd_data.1934e2eb141f2.00000000000003bb [read 0~4194304] 5.b0d13388 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd2b70
  -277> 2013-11-13 16:48:25.566508 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 194 ==== osd_op_reply(962 rbd_data.1934e2eb141f2.00000000000003bb [read 0~4194304] v4376'14447 uv14447 ondisk = 0) v4 ==== 138+0+4194304 (1070585516 0 2295638004) 0x7f90e4000aa0 con 0x1cd2b70
  -276> 2013-11-13 16:48:25.574770 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.4:6800/3857 -- osd_op(client.314912.0:963 rbd_data.1934e2eb141f2.00000000000003bc [read 0~4194304] 5.90f936e0 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cda050
  -275> 2013-11-13 16:48:25.657042 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.0 10.0.0.4:6800/3857 211 ==== osd_op_reply(963 rbd_data.1934e2eb141f2.00000000000003bc [read 0~4194304] v4376'861 uv861 ondisk = 0) v4 ==== 138+0+4194304 (3206826341 0 3017153210) 0x7f90bc0009f0 con 0x1cda050
  -274> 2013-11-13 16:48:25.665341 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.5:6800/8449 -- osd_op(client.314912.0:964 rbd_data.1934e2eb141f2.00000000000003bd [read 0~4194304] 5.580f4565 e5250) v4 -- ?+0 0x1ceeed0 con 0x1cd37d0
  -273> 2013-11-13 16:48:25.736299 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.3 10.0.0.5:6800/8449 193 ==== osd_op_reply(964 rbd_data.1934e2eb141f2.00000000000003bd [read 0~4194304] v4376'889624 uv889624 ondisk = 0) v4 ==== 138+0+4194304 (3584599461 0 3755661316) 0x7f90d40011e0 con 0x1cd37d0
  -272> 2013-11-13 16:48:25.744497 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.7:6800/12719 -- osd_op(client.314912.0:965 rbd_data.1934e2eb141f2.00000000000003be [read 0~4194304] 5.d293477a e5250) v4 -- ?+0 0x1cece50 con 0x1cd4b50
  -271> 2013-11-13 16:48:25.783294 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.9 10.0.0.7:6800/12719 175 ==== osd_op_reply(965 rbd_data.1934e2eb141f2.00000000000003be [read 0~4194304] v0'0 uv5480 ondisk = 0) v6 ==== 198+0+4194304 (2427679694 0 1882884935) 0x7f90c4000a20 con 0x1cd4b50
  -270> 2013-11-13 16:48:25.791785 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.8:6800/24537 -- osd_op(client.314912.0:966 rbd_data.1934e2eb141f2.00000000000003bf [read 0~4194304] 5.164286ae e5250) v4 -- ?+0 0x1cf1290 con 0x1cd2270
  -269> 2013-11-13 16:48:25.859623 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.12 10.0.0.8:6800/24537 193 ==== osd_op_reply(966 rbd_data.1934e2eb141f2.00000000000003bf [read 0~4194304] v4376'10916 uv10916 ondisk = 0) v4 ==== 138+0+4194304 (3556169359 0 318208424) 0x7f90ec0009f0 con 0x1cd2270
  -268> 2013-11-13 16:48:25.867928 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:967 rbd_data.1934e2eb141f2.00000000000003c0 [read 0~4194304] 5.577b7c37 e5250) v4 -- ?+0 0x1cece50 con 0x1cd2b70
  -267> 2013-11-13 16:48:25.937402 7f910067e700  1 -- 10.0.0.7:0/1014692 <== osd.6 10.0.0.6:6800/21239 195 ==== osd_op_reply(967 rbd_data.1934e2eb141f2.00000000000003c0 [read 0~4194304] v4376'10608 uv10608 ondisk = 0) v4 ==== 138+0+4194304 (2292416349 0 1836559768) 0x7f90e4000aa0 con 0x1cd2b70
  -266> 2013-11-13 16:48:25.945533 7f9101ff4780  1 -- 10.0.0.7:0/1014692 --> 10.0.0.6:6800/21239 -- osd_op(client.314912.0:968 rbd_data.1934e2eb141f2.00000000000003c1 [read 0~4194304] 5.5a49b0ec e5250) v4 -- ?+0 0x1cf1290 con 0x1cd2b70
  -265> 2013-11-13 16:48:26.014918 7f910067e700  1 -- 10.0.0.7:
Actions
Copy link
 #6

Updated by Corin Langosch over 10 years ago

Here the full log ('debug objecter = 20' and 'debug objectcacher = 20' on the client) when doing the "rbd export" https://www.dropbox.com/s/sc7luuza9xv3jnu/export.txt

Actions
Copy link
 #7

Updated by Corin Langosch over 10 years ago

I just attached it here too. Sorry.

Actions
Copy link
 #8

Updated by Josh Durgin over 10 years ago

The log shows the clients getting an error (which should be handled better):

ack = -23 (Too many open files in system)

You need to increase the system max open file limit on the hosts running osds.

Actions
Copy link
 #9

Updated by Corin Langosch over 10 years ago

System-Wide limits seem good on all hosts:

root@r-ch104:~# sysctl fs.file-max
fs.file-max = 6548500
root@r-ch104:~# cat /proc/sys/fs/file-max
6548500

The strange this is that the error also occurs for osds that I didn't upgrade yet..?!

Actions
Copy link
 #11

Updated by Ian Colle over 10 years ago

  • Subject changed from running qemu/kvm gusest crashed and do not start again after upgrade dumpling -> emperor to running qemu/kvm guests crashed and do not start again after upgrade dumpling -> emperor
  • Assignee set to Josh Durgin
  • Priority changed from Immediate to Urgent
Actions
Copy link
 #12

Updated by Samuel Just over 10 years ago

  • Assignee changed from Josh Durgin to Samuel Just
  • Priority changed from Urgent to Immediate

(12:56:37 PM) /home: the object_info encoding changed.
(01:07:07 PM) /home: the new "flags" field in object_info_t overlaps with the "lost" field in dumpling
(01:07:14 PM) /home: so a non-zero flags field reads as lost
(01:07:22 PM) /home: and flags is often non-empty because of the dirty bit

Actions
Copy link
 #13

Updated by Samuel Just over 10 years ago

We need to understand why the upgrade suites did not catch this.

Actions
Copy link
 #14

Updated by Ian Colle over 10 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #15

Updated by Greg Farnum over 10 years ago

  • Subject changed from running qemu/kvm guests crashed and do not start again after upgrade dumpling -> emperor to emperor's "dirty" flag is being interpreted as "lost" by Dumpling OSDs

We messed up when changing the encoding for adding flags to the object_info_t. Sam has a patch which looks good to me, and I am reproducing the bug locally (with notes) to generate a small busted store we can use for testing his fixer tool.

Actions
Copy link
 #16

Updated by Greg Farnum over 10 years ago

Yup, it's easy to reproduce. I did it with vstart, notes follow.

Build dumpling branch
create and run cluster: CEPH_NUM_MON=1 CEPH_NUM_MDS=1 CEPH_NUM_OSD=3 ./vstart.sh -n -d -x
build emperor branch
restart one OSD on emperor: kill 22182; ./ceph-osd -i 0 -c ceph.conf
write some data: ./rados -p data bench 60 write --no-cleanup
read the data back: ./rados -p data bench 60 seq
kill the emperor osd: kill 7313
read the data back again: ./rados -p data bench 60 seq

See the failure:
error during benchmark: -23
error 23: (23) Too many open files in system
Actions
Copy link
 #17

Updated by Samuel Just over 10 years ago

(04:25:58 PM) /laptop: yes
(04:26:09 PM) /laptop: you need to create a cluster at dumpling
(04:26:11 PM) /laptop: create some objects
(04:26:15 PM) /laptop: upgrade 1 osd to emperor
(04:26:20 PM) /laptop: overwrite the objects
(04:26:24 PM) /laptop: kill the emperor osd
(04:26:26 PM) /laptop: read the objects
(04:26:36 PM) /laptop: verify that you get back an error on some of the objects
(04:26:40 PM) /laptop: (part 1 of the bug)
(04:26:46 PM) /laptop: then overwrite the objects again
(04:26:56 PM) /laptop: revive the emperor osd and upgrade the rest to emperor
(04:27:03 PM) /laptop: read the objects and verify that you get an error
(04:27:14 PM) : oh ok
(04:27:20 PM) : thanks a lot for the steps, will try it out
(04:27:23 PM) /laptop: yep
(04:27:55 PM) /laptop: wip-6761-emperor has a fix which should prevent the issue if you do the above with wip-6761-emperor instead of emperor
(04:28:04 PM) /laptop: it also has a tool to repair stores in the state caused by the bug
(04:29:07 PM) /laptop: so we'll need to verify upgrades from dumpling to wip-6761-emperor, and from dumpling to emperor (with the above sequence to cause the bug) to wip-6761-emperor. That second case will require the tool to repair the damaged stores

Actions
Copy link
 #18

Updated by Greg Farnum over 10 years ago

wip-6761-emperor tool appears to work:

checkout dumpling; build
run new cluster: CEPH_NUM_MON=1 CEPH_NUM_MDS=1 CEPH_NUM_OSD=3 ./vstart.sh -n -d -x
upgrade to emperor; build
kill and restart osd 0: kill 2058; ./ceph-osd -i 0 -c ceph.conf
put a bunch of objects: for i in `seq 10`; do ./rados -p data put obj$i ceph.conf; done
kill osd 0: kill 21775
try and read the objects: for i in `seq 10`; do ./rados -p data get obj$i -; done | less
notice that some of the reads failed
write the objects again: for i in `seq 10`; do ./rados -p data put obj$i ceph.conf; done
notice that they all succeed, but subsequent reads fail: for i in `seq 10`; do ./rados -p data get obj$i -; done | less
restart the full cluster on emperor: ./stop.sh; CEPH_NUM_MON=1 CEPH_NUM_MDS=1 CEPH_NUM_OSD=3 ./vstart.sh -d -x
get the objects; notice that some reads failed: for i in `seq 10`; do ./rados -p data get obj$i -; done | less
stop the cluster: ./stop.sh
upgrade to wip-6761-emperor; build
run the new tool: ./ceph_filestore_tool --filestore-path dev/osd1 --journal-path dev/osd1.journal --list-lost-objects true
see that it finds lost objects; run the fix: ./ceph_filestore_tool --filestore-path dev/osd1 --journal-path dev/osd1.journal --fix-lost-objects true
turn on the cluster: CEPH_NUM_MON=1 CEPH_NUM_MDS=1 CEPH_NUM_OSD=3 ./vstart.sh -d -x
get the objects and see they all work: for i in `seq 10`; do ./rados -p data get obj$i -; done | less

Reviewing now.

Actions
Copy link
 #19

Updated by Samuel Just over 10 years ago

I've got a repair mechanism for you to try:

Install wip-6761-emperor on all osd machines
stop all osds
for each osd data directory:
ceph_filestore_tool --list-lost-objects=true --filestore-path=<path-to-osd-filestore> --journal-path=<path-to-osd-journal>

That should spit out a complete list of all objects with the lost bit incorrectly set.

for each osd data directory:
ceph_filestore_tool --list-lost-objects=true --filestore-path=<path-to-osd-filestore> --journal-path=<path-to-osd-journal> --fix-lost-objects=true

That should repair the objects listed above.

You should then be able to restart your osds and continue normally. wip-6761-emperor also has a patch which fixes the error in emperor which caused this in the first place. It will be included in the point release when we get there. Once the point release comes out, you can side-grade to the point release.

Actions
Copy link
 #20

Updated by Tamilarasi muthamizhan over 10 years ago

tested wip-6761-emperor on mira056 and mira074 . It works fine!

Actions
Copy link
 #21

Updated by Corin Langosch over 10 years ago

After installing the gitbuilder version and checking all osds reported between 50 - 200 lost objects. Repair went without any problems. The cluster came back up without any problems and I was able to start guests which failed before. Also the rbd export works. I still have to start a lot more guests, but so far everything looks stable!

Thank you, thank you very much!

Actions
Copy link
 #22

Updated by Corin Langosch over 10 years ago

Now some hours later everything is still running perfectly fine. Thanks again!

Actions
Copy link
 #23

Updated by Damien Churchill over 10 years ago

Likewise for me, following the instructions everything is now back up and running without issues, thanks!

Actions
Copy link
 #24

Updated by Samuel Just over 10 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF