Script started on Mon 31 Aug 2015 10:48:29 AM PDT thorvald@sc2-r17-u08:/t$ rbd export-diff AlignNav-db fil librbd/internal.cc: In function 'int librbd::simple_diff_cb(uint64_t, size_t, int, void*)' thread 7f7539f20840 time 2015-08-31 10:48:36.797179 librbd/internal.cc: 2815: FAILED assert(exists) ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b) 1: (()+0x16fabb) [0x7f7537236abb] 2: (()+0xa1268) [0x7f7537168268] 3: (()+0xa1ee6) [0x7f7537168ee6] 4: (()+0xa1701) [0x7f7537168701] 5: (librbd::Image::diff_iterate(char const*, unsigned long, unsigned long, int (*)(unsigned long, unsigned long, int, void*), void*)+0x5b) [0x7f7537113b1b] 6: (main()+0x441c) [0x4c2d6c] 7: (__libc_start_main()+0xf5) [0x7f7532f22ec5] 8: rbd() [0x4c7057] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (Aborted) ** in thread 7f7539f20840 ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b) 1: rbd() [0x59690a] 2: (()+0x10340) [0x7f753467e340] 3: (gsignal()+0x39) [0x7f7532f37cc9] 4: (abort()+0x148) [0x7f7532f3b0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7533842535] 6: (()+0x5e6d6) [0x7f75338406d6] 7: (()+0x5e703) [0x7f7533840703] 8: (()+0x5e922) [0x7f7533840922] 9: (()+0x16fca8) [0x7f7537236ca8] 10: (()+0xa1268) [0x7f7537168268] 11: (()+0xa1ee6) [0x7f7537168ee6] 12: (()+0xa1701) [0x7f7537168701] 13: (librbd::Image::diff_iterate(char const*, unsigned long, unsigned long, int (*)(unsigned long, unsigned long, int, void*), void*)+0x5b) [0x7f7537113b1b] 14: (main()+0x441c) [0x4c2d6c] 15: (__libc_start_main()+0xf5) [0x7f7532f22ec5] 16: rbd() [0x4c7057] 2015-08-31 10:48:36.797843 7f7539f20840 -1 *** Caught signal (Aborted) ** in thread 7f7539f20840 ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b) 1: rbd() [0x59690a] 2: (()+0x10340) [0x7f753467e340] 3: (gsignal()+0x39) [0x7f7532f37cc9] 4: (abort()+0x148) [0x7f7532f3b0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7533842535] 6: (()+0x5e6d6) [0x7f75338406d6] 7: (()+0x5e703) [0x7f7533840703] 8: (()+0x5e922) [0x7f7533840922] 9: (()+0x16fca8) [0x7f7537236ca8] 10: (()+0xa1268) [0x7f7537168268] 11: (()+0xa1ee6) [0x7f7537168ee6] 12: (()+0xa1701) [0x7f7537168701] 13: (librbd::Image::diff_iterate(char const*, unsigned long, unsigned long, int (*)(unsigned long, unsigned long, int, void*), void*)+0x5b) [0x7f7537113b1b] 14: (main()+0x441c) [0x4c2d6c] 15: (__libc_start_main()+0xf5) [0x7f7532f22ec5] 16: rbd() [0x4c7057] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -163> 2015-08-31 10:48:36.620555 7f7539f20840 5 asok(0x32929e0) register_command perfcounters_dump hook 0x3292930 -162> 2015-08-31 10:48:36.620568 7f7539f20840 5 asok(0x32929e0) register_command 1 hook 0x3292930 -161> 2015-08-31 10:48:36.620571 7f7539f20840 5 asok(0x32929e0) register_command perf dump hook 0x3292930 -160> 2015-08-31 10:48:36.620574 7f7539f20840 5 asok(0x32929e0) register_command perfcounters_schema hook 0x3292930 -159> 2015-08-31 10:48:36.620576 7f7539f20840 5 asok(0x32929e0) register_command 2 hook 0x3292930 -158> 2015-08-31 10:48:36.620577 7f7539f20840 5 asok(0x32929e0) register_command perf schema hook 0x3292930 -157> 2015-08-31 10:48:36.620580 7f7539f20840 5 asok(0x32929e0) register_command perf reset hook 0x3292930 -156> 2015-08-31 10:48:36.620584 7f7539f20840 5 asok(0x32929e0) register_command config show hook 0x3292930 -155> 2015-08-31 10:48:36.620587 7f7539f20840 5 asok(0x32929e0) register_command config set hook 0x3292930 -154> 2015-08-31 10:48:36.620612 7f7539f20840 5 asok(0x32929e0) register_command config get hook 0x3292930 -153> 2015-08-31 10:48:36.620614 7f7539f20840 5 asok(0x32929e0) register_command config diff hook 0x3292930 -152> 2015-08-31 10:48:36.620616 7f7539f20840 5 asok(0x32929e0) register_command log flush hook 0x3292930 -151> 2015-08-31 10:48:36.620617 7f7539f20840 5 asok(0x32929e0) register_command log dump hook 0x3292930 -150> 2015-08-31 10:48:36.620619 7f7539f20840 5 asok(0x32929e0) register_command log reopen hook 0x3292930 -149> 2015-08-31 10:48:36.623487 7f7539f20840 10 monclient(hunting): build_initial_monmap -148> 2015-08-31 10:48:36.623555 7f7539f20840 1 librados: starting msgr at :/0 -147> 2015-08-31 10:48:36.623563 7f7539f20840 1 librados: starting objecter -146> 2015-08-31 10:48:36.623589 7f7539f20840 5 asok(0x32929e0) register_command objecter_requests hook 0x32b3dd0 -145> 2015-08-31 10:48:36.623658 7f7539f20840 1 -- :/0 messenger.start -144> 2015-08-31 10:48:36.623688 7f7539f20840 1 librados: setting wanted keys -143> 2015-08-31 10:48:36.623693 7f7539f20840 1 librados: calling monclient init -142> 2015-08-31 10:48:36.623694 7f7539f20840 10 monclient(hunting): init -141> 2015-08-31 10:48:36.623702 7f7539f20840 5 adding auth protocol: cephx -140> 2015-08-31 10:48:36.623709 7f7539f20840 10 monclient(hunting): auth_supported 2 method cephx -139> 2015-08-31 10:48:36.623846 7f7539f20840 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring -138> 2015-08-31 10:48:36.623898 7f7539f20840 10 monclient(hunting): _reopen_session rank -1 name -137> 2015-08-31 10:48:36.623934 7f7539f20840 10 monclient(hunting): picked mon.noname-a con 0x32b53f0 addr 10.225.10.98:6789/0 -136> 2015-08-31 10:48:36.623953 7f7539f20840 10 monclient(hunting): _send_mon_message to mon.noname-a at 10.225.10.98:6789/0 -135> 2015-08-31 10:48:36.623958 7f7539f20840 1 -- :/1016056 --> 10.225.10.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x32b7a70 con 0x32b53f0 -134> 2015-08-31 10:48:36.623968 7f7539f20840 10 monclient(hunting): renew_subs -133> 2015-08-31 10:48:36.623972 7f7539f20840 10 monclient(hunting): authenticate will time out at 2015-08-31 10:53:36.623972 -132> 2015-08-31 10:48:36.624221 7f7539f1e700 1 -- 10.225.17.34:0/1016056 learned my addr 10.225.17.34:0/1016056 -131> 2015-08-31 10:48:36.624532 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 1 ==== mon_map magic: 0 v1 ==== 503+0+0 (2796378027 0 0) 0x7f751c000be0 con 0x32b53f0 -130> 2015-08-31 10:48:36.624592 7f752e42a700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -129> 2015-08-31 10:48:36.624613 7f752e42a700 10 monclient(hunting): got monmap 1, mon.noname-a is now rank -1 -128> 2015-08-31 10:48:36.624623 7f752e42a700 10 monclient(hunting): dump: epoch 1 fsid 386aabfc-9578-4ad4-8372-1b31334c45ce last_changed 0.000000 created 0.000000 0: 10.225.10.98:6789/0 mon.sc2-r10-u24 1: 10.225.11.94:6789/0 mon.sc2-r11-u23 2: 10.225.12.94:6789/0 mon.sc2-r12-u23 -127> 2015-08-31 10:48:36.624663 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3097111796 0 0) 0x7f751c0010f0 con 0x32b53f0 -126> 2015-08-31 10:48:36.624699 7f752e42a700 10 monclient(hunting): my global_id is 7111116 -125> 2015-08-31 10:48:36.624846 7f752e42a700 10 monclient(hunting): _send_mon_message to mon.sc2-r10-u24 at 10.225.10.98:6789/0 -124> 2015-08-31 10:48:36.624853 7f752e42a700 1 -- 10.225.17.34:0/1016056 --> 10.225.10.98:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f75200015c0 con 0x32b53f0 -123> 2015-08-31 10:48:36.625415 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (1181176599 0 0) 0x7f751c0010f0 con 0x32b53f0 -122> 2015-08-31 10:48:36.625489 7f752e42a700 10 monclient(hunting): _send_mon_message to mon.sc2-r10-u24 at 10.225.10.98:6789/0 -121> 2015-08-31 10:48:36.625496 7f752e42a700 1 -- 10.225.17.34:0/1016056 --> 10.225.10.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7f75200037c0 con 0x32b53f0 -120> 2015-08-31 10:48:36.625958 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 393+0+0 (2013097346 0 0) 0x7f751c0008c0 con 0x32b53f0 -119> 2015-08-31 10:48:36.626026 7f752e42a700 1 monclient(hunting): found mon.sc2-r10-u24 -118> 2015-08-31 10:48:36.626031 7f752e42a700 10 monclient: _send_mon_message to mon.sc2-r10-u24 at 10.225.10.98:6789/0 -117> 2015-08-31 10:48:36.626034 7f752e42a700 1 -- 10.225.17.34:0/1016056 --> 10.225.10.98:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x32b7d80 con 0x32b53f0 -116> 2015-08-31 10:48:36.626075 7f7539f20840 5 monclient: authenticate success, global_id 7111116 -115> 2015-08-31 10:48:36.626097 7f7539f20840 10 monclient: renew_subs -114> 2015-08-31 10:48:36.626101 7f7539f20840 10 monclient: _send_mon_message to mon.sc2-r10-u24 at 10.225.10.98:6789/0 -113> 2015-08-31 10:48:36.626107 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.10.98:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x32bc3e0 con 0x32b53f0 -112> 2015-08-31 10:48:36.626141 7f7539f20840 10 monclient: renew_subs -111> 2015-08-31 10:48:36.626146 7f7539f20840 10 monclient: _send_mon_message to mon.sc2-r10-u24 at 10.225.10.98:6789/0 -110> 2015-08-31 10:48:36.626150 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.10.98:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x32bc3e0 con 0x32b53f0 -109> 2015-08-31 10:48:36.626175 7f7539f20840 1 librados: init done -108> 2015-08-31 10:48:36.626229 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 503+0+0 (2796378027 0 0) 0x7f751c001350 con 0x32b53f0 -107> 2015-08-31 10:48:36.626237 7f752e42a700 10 monclient: handle_monmap mon_map magic: 0 v1 -106> 2015-08-31 10:48:36.626243 7f752e42a700 10 monclient: got monmap 1, mon.sc2-r10-u24 is now rank 0 -105> 2015-08-31 10:48:36.626247 7f752e42a700 10 monclient: dump: epoch 1 fsid 386aabfc-9578-4ad4-8372-1b31334c45ce last_changed 0.000000 created 0.000000 0: 10.225.10.98:6789/0 mon.sc2-r10-u24 1: 10.225.11.94:6789/0 mon.sc2-r11-u23 2: 10.225.12.94:6789/0 mon.sc2-r12-u23 -104> 2015-08-31 10:48:36.626260 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (44504396 0 0) 0x7f751c001610 con 0x32b53f0 -103> 2015-08-31 10:48:36.626267 7f752e42a700 10 monclient: handle_subscribe_ack sent 2015-08-31 10:48:36.623971 renew after 2015-08-31 10:51:06.623971 -102> 2015-08-31 10:48:36.627008 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 7 ==== osd_map(101238..101238 src has 99519..101238) v3 ==== 356066+0+0 (3964671413 0 0) 0x7f751c0008c0 con 0x32b53f0 -101> 2015-08-31 10:48:36.628136 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (44504396 0 0) 0x7f751c001400 con 0x32b53f0 -100> 2015-08-31 10:48:36.628147 7f752e42a700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring -99> 2015-08-31 10:48:36.628151 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 9 ==== osd_map(101238..101238 src has 99519..101238) v3 ==== 356066+0+0 (3964671413 0 0) 0x7f751c001780 con 0x32b53f0 -98> 2015-08-31 10:48:36.628199 7f752e42a700 1 -- 10.225.17.34:0/1016056 <== mon.0 10.225.10.98:6789/0 10 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (44504396 0 0) 0x7f751c001bf0 con 0x32b53f0 -97> 2015-08-31 10:48:36.628206 7f752e42a700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring -96> 2015-08-31 10:48:36.628414 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.11.102:6800/31431 -- osd_op(client.7111116.0:1 AlignNav-db.rbd [stat] 2.1f2394bb ack+read+known_if_redirected e101238) v5 -- ?+0 0x32c5010 con 0x32c3c60 -95> 2015-08-31 10:48:36.630596 7f7528a1d700 1 -- 10.225.17.34:0/1016056 <== osd.166 10.225.11.102:6800/31431 1 ==== osd_op_reply(1 AlignNav-db.rbd [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 182+0+0 (812855791 0 0) 0x7f750c000aa0 con 0x32c3c60 -94> 2015-08-31 10:48:36.630696 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.98:6824/14440 -- osd_op(client.7111116.0:2 rbd_id.AlignNav-db [stat] 2.cc0393e5 ack+read+known_if_redirected e101238) v5 -- ?+0 0x32cb710 con 0x32ca430 -93> 2015-08-31 10:48:36.631909 7f752881b700 1 -- 10.225.17.34:0/1016056 <== osd.62 10.225.16.98:6824/14440 1 ==== osd_op_reply(2 rbd_id.AlignNav-db [stat] v0'0 uv290560 ondisk = 0) v6 ==== 185+0+16 (951307066 0 3130641107) 0x7f7504000c20 con 0x32ca430 -92> 2015-08-31 10:48:36.631959 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.98:6824/14440 -- osd_op(client.7111116.0:3 rbd_id.AlignNav-db [call rbd.get_id] 2.cc0393e5 ack+read+known_if_redirected e101238) v5 -- ?+0 0x32cb710 con 0x32ca430 -91> 2015-08-31 10:48:36.632821 7f752881b700 1 -- 10.225.17.34:0/1016056 <== osd.62 10.225.16.98:6824/14440 2 ==== osd_op_reply(3 rbd_id.AlignNav-db [call] v0'0 uv290560 ondisk = 0) v6 ==== 185+0+18 (661511449 0 932158682) 0x7f7504000c20 con 0x32ca430 -90> 2015-08-31 10:48:36.632928 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.14.122:6841/347 -- osd_op(client.7111116.0:4 rbd_header.6bb2042eb141f2 [call rbd.get_size,call rbd.get_object_prefix] 2.c8c60c8e ack+read+known_if_redirected e101238) v5 -- ?+0 0x32cd1a0 con 0x32cfdf0 -89> 2015-08-31 10:48:36.634060 7f7528619700 1 -- 10.225.17.34:0/1016056 <== osd.356 10.225.14.122:6841/347 1 ==== osd_op_reply(4 rbd_header.6bb2042eb141f2 [call,call] v0'0 uv375625 ondisk = 0) v6 ==== 234+0+36 (1990347160 0 2386706239) 0x7f74fc000c60 con 0x32cfdf0 -88> 2015-08-31 10:48:36.634129 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.14.122:6841/347 -- osd_op(client.7111116.0:5 rbd_header.6bb2042eb141f2 [call rbd.get_stripe_unit_count] 2.c8c60c8e ack+read+known_if_redirected e101238) v5 -- ?+0 0x32cd1a0 con 0x32cfdf0 -87> 2015-08-31 10:48:36.634942 7f7528619700 1 -- 10.225.17.34:0/1016056 <== osd.356 10.225.14.122:6841/347 2 ==== osd_op_reply(5 rbd_header.6bb2042eb141f2 [call] v0'0 uv0 ondisk = -8 ((8) Exec format error)) v6 ==== 192+0+0 (314479126 0 0) 0x7f74fc000c60 con 0x32cfdf0 -86> 2015-08-31 10:48:36.635573 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.14.122:6841/347 -- osd_op(client.7111116.0:6 rbd_header.6bb2042eb141f2 [watch watch cookie 53281472] 2.c8c60c8e ondisk+write+known_if_redirected e101238) v5 -- ?+0 0x32d0990 con 0x32cfdf0 -85> 2015-08-31 10:48:36.639173 7f7528619700 1 -- 10.225.17.34:0/1016056 <== osd.356 10.225.14.122:6841/347 3 ==== osd_op_reply(6 rbd_header.6bb2042eb141f2 [watch watch cookie 53281472] v101238'378574 uv375625 ondisk = 0) v6 ==== 192+0+0 (695357705 0 0) 0x7f74fc000c60 con 0x32cfdf0 -84> 2015-08-31 10:48:36.639378 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.14.122:6841/347 -- osd_op(client.7111116.0:7 rbd_header.6bb2042eb141f2 [call rbd.get_size,call rbd.get_features,call rbd.get_snapcontext,call rbd.get_parent,call lock.get_info] 2.c8c60c8e ack+read+known_if_redirected e101238) v5 -- ?+0 0x32d6c50 con 0x32cfdf0 -83> 2015-08-31 10:48:36.640295 7f7528619700 1 -- 10.225.17.34:0/1016056 <== osd.356 10.225.14.122:6841/347 4 ==== osd_op_reply(7 rbd_header.6bb2042eb141f2 [call,call,call,call,call] v0'0 uv375625 ondisk = 0) v6 ==== 360+0+109 (2593115994 0 1644323379) 0x7f74fc000c60 con 0x32cfdf0 -82> 2015-08-31 10:48:36.640451 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.14.122:6841/347 -- osd_op(client.7111116.0:8 rbd_header.6bb2042eb141f2 [call rbd.get_flags,call rbd.get_flags,call rbd.get_flags] 2.c8c60c8e ack+read+known_if_redirected e101238) v5 -- ?+0 0x32d07c0 con 0x32cfdf0 -81> 2015-08-31 10:48:36.641492 7f7528619700 1 -- 10.225.17.34:0/1016056 <== osd.356 10.225.14.122:6841/347 5 ==== osd_op_reply(8 rbd_header.6bb2042eb141f2 [call,call,call] v0'0 uv375625 ondisk = 0) v6 ==== 276+0+24 (2625613286 0 0) 0x7f74fc000c60 con 0x32cfdf0 -80> 2015-08-31 10:48:36.641681 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.14.122:6841/347 -- osd_op(client.7111116.0:9 rbd_header.6bb2042eb141f2 [call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status] 2.c8c60c8e ack+read+known_if_redirected e101238) v5 -- ?+0 0x32d66c0 con 0x32cfdf0 -79> 2015-08-31 10:48:36.642531 7f7528619700 1 -- 10.225.17.34:0/1016056 <== osd.356 10.225.14.122:6841/347 6 ==== osd_op_reply(9 rbd_header.6bb2042eb141f2 [call,call,call,call,call,call,call,call,call,call] v0'0 uv375625 ondisk = 0) v6 ==== 570+0+158 (1277976645 0 2619260201) 0x7f74fc000c60 con 0x32cfdf0 -78> 2015-08-31 10:48:36.642724 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.118:6846/12684 -- osd_op(client.7111116.0:10 rbd_header.d54542ae8944a [call rbd.get_size,call rbd.get_object_prefix] 2.81f1c4ad ack+read+known_if_redirected e101238) v5 -- ?+0 0x32cead0 con 0x32d8730 -77> 2015-08-31 10:48:36.647097 7f7528417700 1 -- 10.225.17.34:0/1016056 <== osd.447 10.225.16.118:6846/12684 1 ==== osd_op_reply(10 rbd_header.d54542ae8944a [call,call] v0'0 uv756154 ondisk = 0) v6 ==== 233+0+35 (3609034154 0 1305094669) 0x7f74f0000c60 con 0x32d8730 -76> 2015-08-31 10:48:36.647166 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.118:6846/12684 -- osd_op(client.7111116.0:11 rbd_header.d54542ae8944a [call rbd.get_stripe_unit_count] 2.81f1c4ad ack+read+known_if_redirected e101238) v5 -- ?+0 0x32ced70 con 0x32d8730 -75> 2015-08-31 10:48:36.649024 7f7528417700 1 -- 10.225.17.34:0/1016056 <== osd.447 10.225.16.118:6846/12684 2 ==== osd_op_reply(11 rbd_header.d54542ae8944a [call] v0'0 uv0 ondisk = -8 ((8) Exec format error)) v6 ==== 191+0+0 (2402334601 0 0) 0x7f74f0000c60 con 0x32d8730 -74> 2015-08-31 10:48:36.649108 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.118:6846/12684 -- osd_op(client.7111116.0:12 rbd_header.d54542ae8944a [call rbd.get_size,call rbd.get_features,call rbd.get_snapcontext,call rbd.get_parent,call lock.get_info] 2.81f1c4ad ack+read+known_if_redirected e101238) v5 -- ?+0 0x32e6010 con 0x32d8730 -73> 2015-08-31 10:48:36.656172 7f7528417700 1 -- 10.225.17.34:0/1016056 <== osd.447 10.225.16.118:6846/12684 3 ==== osd_op_reply(12 rbd_header.d54542ae8944a [call,call,call,call,call] v0'0 uv756154 ondisk = 0) v6 ==== 359+0+152 (2329166894 0 3734159652) 0x7f74f0000c60 con 0x32d8730 -72> 2015-08-31 10:48:36.656285 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.118:6846/12684 -- osd_op(client.7111116.0:13 rbd_header.d54542ae8944a [call rbd.get_flags,call rbd.get_flags,call rbd.get_flags,call rbd.get_flags,call rbd.get_flags,call rbd.get_flags,call rbd.get_flags,call rbd.get_flags,call rbd.get_flags,call rbd.get_flags] 2.81f1c4ad ack+read+known_if_redirected e101238) v5 -- ?+0 0x32e6350 con 0x32d8730 -71> 2015-08-31 10:48:36.669625 7f7528417700 1 -- 10.225.17.34:0/1016056 <== osd.447 10.225.16.118:6846/12684 4 ==== osd_op_reply(13 rbd_header.d54542ae8944a [call,call,call,call,call,call,call,call,call,call] v0'0 uv756154 ondisk = 0) v6 ==== 569+0+80 (953003673 0 0) 0x7f74f0000c60 con 0x32d8730 -70> 2015-08-31 10:48:36.670070 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.118:6846/12684 -- osd_op(client.7111116.0:14 rbd_header.d54542ae8944a [call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status,call rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call rbd.get_parent,call rbd.get_protection_status] 2.81f1c4ad ack+read+known_if_redirected e101238) v5 -- ?+0 0x32eea40 con 0x32d8730 -69> 2015-08-31 10:48:36.750280 7f7528417700 1 -- 10.225.17.34:0/1016056 <== osd.447 10.225.16.118:6846/12684 5 ==== osd_op_reply(14 rbd_header.d54542ae8944a [call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call,call] v0'0 uv756154 ondisk = 0) v6 ==== 2039+0+648 (1951251762 0 4015442582) 0x7f74f0004ba0 con 0x32d8730 -68> 2015-08-31 10:48:36.750555 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.106:6828/31879 -- osd_op(client.7111116.0:15 rbd_data.d54542ae8944a.0000000000000000@snapdir [list-snaps] 2.70b5caf3 ack+read+known_if_redirected e101238) v5 -- ?+0 0x32f0e30 con 0x32cee40 -67> 2015-08-31 10:48:36.752117 7f7528215700 1 -- 10.225.17.34:0/1016056 <== osd.373 10.225.15.106:6828/31879 1 ==== osd_op_reply(15 rbd_data.d54542ae8944a.0000000000000000 [list-snaps] v0'0 uv423192 ondisk = 0) v6 ==== 206+0+352 (1872411861 0 2633943104) 0x7f74e8000d80 con 0x32cee40 -66> 2015-08-31 10:48:36.752270 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.14.110:6804/4368 -- osd_op(client.7111116.0:16 rbd_data.d54542ae8944a.0000000000000001@snapdir [list-snaps] 2.37b2b478 ack+read+known_if_redirected e101238) v5 -- ?+0 0x32f0550 con 0x32f1eb0 -65> 2015-08-31 10:48:36.753195 7f75017fa700 1 -- 10.225.17.34:0/1016056 <== osd.333 10.225.14.110:6804/4368 1 ==== osd_op_reply(16 rbd_data.d54542ae8944a.0000000000000001 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1272271533 0 0) 0x7f74e0000ac0 con 0x32f1eb0 -64> 2015-08-31 10:48:36.753325 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.13.110:6843/5470 -- osd_op(client.7111116.0:17 rbd_data.d54542ae8944a.0000000000000002@snapdir [list-snaps] 2.4ecd9902 ack+read+known_if_redirected e101238) v5 -- ?+0 0x32dbf50 con 0x32d94b0 -63> 2015-08-31 10:48:36.754197 7f75015f8700 1 -- 10.225.17.34:0/1016056 <== osd.249 10.225.13.110:6843/5470 1 ==== osd_op_reply(17 rbd_data.d54542ae8944a.0000000000000002 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (2349933522 0 0) 0x7f74d8000ac0 con 0x32d94b0 -62> 2015-08-31 10:48:36.754322 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.13.114:6840/24206 -- osd_op(client.7111116.0:18 rbd_data.d54542ae8944a.0000000000000003@snapdir [list-snaps] 2.dd83a855 ack+read+known_if_redirected e101238) v5 -- ?+0 0x32da080 con 0x32d9ce0 -61> 2015-08-31 10:48:36.755466 7f75013f6700 1 -- 10.225.17.34:0/1016056 <== osd.275 10.225.13.114:6840/24206 1 ==== osd_op_reply(18 rbd_data.d54542ae8944a.0000000000000003 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1760064444 0 0) 0x7f74d0000ac0 con 0x32d9ce0 -60> 2015-08-31 10:48:36.755590 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.122:6825/9151 -- osd_op(client.7111116.0:19 rbd_data.d54542ae8944a.0000000000000004@snapdir [list-snaps] 2.aed4fc87 ack+read+known_if_redirected e101238) v5 -- ?+0 0x32e5910 con 0x32e4590 -59> 2015-08-31 10:48:36.757178 7f75011f4700 1 -- 10.225.17.34:0/1016056 <== osd.400 10.225.15.122:6825/9151 1 ==== osd_op_reply(19 rbd_data.d54542ae8944a.0000000000000004 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (3379668036 0 0) 0x7f74c8000ac0 con 0x32e4590 -58> 2015-08-31 10:48:36.757298 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.12.94:6804/23703 -- osd_op(client.7111116.0:20 rbd_data.d54542ae8944a.0000000000000005@snapdir [list-snaps] 2.5ed2a9a5 ack+read+known_if_redirected e101238) v5 -- ?+0 0x33047f0 con 0x3303510 -57> 2015-08-31 10:48:36.758298 7f7500ff2700 1 -- 10.225.17.34:0/1016056 <== osd.18 10.225.12.94:6804/23703 1 ==== osd_op_reply(20 rbd_data.d54542ae8944a.0000000000000005 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1972858780 0 0) 0x7f74c0000ac0 con 0x3303510 -56> 2015-08-31 10:48:36.758418 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.11.110:6842/31121 -- osd_op(client.7111116.0:21 rbd_data.d54542ae8944a.0000000000000006@snapdir [list-snaps] 2.6bd74b4d ack+read+known_if_redirected e101238) v5 -- ?+0 0x330af80 con 0x3309b30 -55> 2015-08-31 10:48:36.759896 7f7500df0700 1 -- 10.225.17.34:0/1016056 <== osd.176 10.225.11.110:6842/31121 1 ==== osd_op_reply(21 rbd_data.d54542ae8944a.0000000000000006 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (3174000043 0 0) 0x7f74b8000ac0 con 0x3309b30 -54> 2015-08-31 10:48:36.760018 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.13.106:6804/20469 -- osd_op(client.7111116.0:22 rbd_data.d54542ae8944a.0000000000000007@snapdir [list-snaps] 2.9337b296 ack+read+known_if_redirected e101238) v5 -- ?+0 0x33115c0 con 0x3310240 -53> 2015-08-31 10:48:36.760990 7f7500bee700 1 -- 10.225.17.34:0/1016056 <== osd.282 10.225.13.106:6804/20469 1 ==== osd_op_reply(22 rbd_data.d54542ae8944a.0000000000000007 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (4282231156 0 0) 0x7f74b0000ac0 con 0x3310240 -52> 2015-08-31 10:48:36.761117 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.94:6800/1579 -- osd_op(client.7111116.0:23 rbd_data.d54542ae8944a.0000000000000008@snapdir [list-snaps] 2.958af301 ack+read+known_if_redirected e101238) v5 -- ?+0 0x3317bc0 con 0x3316840 -51> 2015-08-31 10:48:36.762627 7f75009ec700 1 -- 10.225.17.34:0/1016056 <== osd.79 10.225.15.94:6800/1579 1 ==== osd_op_reply(23 rbd_data.d54542ae8944a.0000000000000008 [list-snaps] v0'0 uv337211 ondisk = 0) v6 ==== 206+0+352 (3947433908 0 2963976097) 0x7f74a8000d80 con 0x3316840 -50> 2015-08-31 10:48:36.762758 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.98:6828/18471 -- osd_op(client.7111116.0:24 rbd_data.d54542ae8944a.0000000000000009@snapdir [list-snaps] 2.fed2b31f ack+read+known_if_redirected e101238) v5 -- ?+0 0x331e300 con 0x331d020 -49> 2015-08-31 10:48:36.764950 7f75007ea700 1 -- 10.225.17.34:0/1016056 <== osd.63 10.225.16.98:6828/18471 1 ==== osd_op_reply(24 rbd_data.d54542ae8944a.0000000000000009 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (2759474798 0 0) 0x7f74a0000ac0 con 0x331d020 -48> 2015-08-31 10:48:36.765079 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.11.94:6804/24795 -- osd_op(client.7111116.0:25 rbd_data.d54542ae8944a.000000000000000a@snapdir [list-snaps] 2.1c7adf3e ack+read+known_if_redirected e101238) v5 -- ?+0 0x3324900 con 0x3323580 -47> 2015-08-31 10:48:36.766114 7f75005e8700 1 -- 10.225.17.34:0/1016056 <== osd.8 10.225.11.94:6804/24795 1 ==== osd_op_reply(25 rbd_data.d54542ae8944a.000000000000000a [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (4179455829 0 0) 0x7f7498000ac0 con 0x3323580 -46> 2015-08-31 10:48:36.766246 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.12.118:6808/13791 -- osd_op(client.7111116.0:26 rbd_data.d54542ae8944a.000000000000000b@snapdir [list-snaps] 2.83307b6b ack+read+known_if_redirected e101238) v5 -- ?+0 0x332af70 con 0x3329bf0 -45> 2015-08-31 10:48:36.767198 7f75003e6700 1 -- 10.225.17.34:0/1016056 <== osd.250 10.225.12.118:6808/13791 1 ==== osd_op_reply(26 rbd_data.d54542ae8944a.000000000000000b [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (2786217760 0 0) 0x7f7490000ac0 con 0x3329bf0 -44> 2015-08-31 10:48:36.767336 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.13.102:6842/16146 -- osd_op(client.7111116.0:27 rbd_data.d54542ae8944a.000000000000000c@snapdir [list-snaps] 2.539edc22 ack+read+known_if_redirected e101238) v5 -- ?+0 0x33315e0 con 0x3330260 -43> 2015-08-31 10:48:36.768354 7f75001e4700 1 -- 10.225.17.34:0/1016056 <== osd.251 10.225.13.102:6842/16146 1 ==== osd_op_reply(27 rbd_data.d54542ae8944a.000000000000000c [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1459958753 0 0) 0x7f7488000ac0 con 0x3330260 -42> 2015-08-31 10:48:36.768484 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.102:6841/19115 -- osd_op(client.7111116.0:28 rbd_data.d54542ae8944a.000000000000000d@snapdir [list-snaps] 2.9c283093 ack+read+known_if_redirected e101238) v5 -- ?+0 0x3337c50 con 0x33368d0 -41> 2015-08-31 10:48:36.770037 7f748fefe700 1 -- 10.225.17.34:0/1016056 <== osd.361 10.225.15.102:6841/19115 1 ==== osd_op_reply(28 rbd_data.d54542ae8944a.000000000000000d [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (869793689 0 0) 0x7f747c000ac0 con 0x33368d0 -40> 2015-08-31 10:48:36.770166 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.12.122:6846/29051 -- osd_op(client.7111116.0:29 rbd_data.d54542ae8944a.000000000000000e@snapdir [list-snaps] 2.e76edba2 ack+read+known_if_redirected e101238) v5 -- ?+0 0x333e2c0 con 0x333cf40 -39> 2015-08-31 10:48:36.771080 7f748fcfc700 1 -- 10.225.17.34:0/1016056 <== osd.305 10.225.12.122:6846/29051 1 ==== osd_op_reply(29 rbd_data.d54542ae8944a.000000000000000e [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1012763020 0 0) 0x7f7474000ac0 con 0x333cf40 -38> 2015-08-31 10:48:36.771209 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.10.94:6828/3775 -- osd_op(client.7111116.0:30 rbd_data.d54542ae8944a.000000000000000f@snapdir [list-snaps] 2.32efa3f7 ack+read+known_if_redirected e101238) v5 -- ?+0 0x3344930 con 0x33435b0 -37> 2015-08-31 10:48:36.773285 7f748fafa700 1 -- 10.225.17.34:0/1016056 <== osd.212 10.225.10.94:6828/3775 1 ==== osd_op_reply(30 rbd_data.d54542ae8944a.000000000000000f [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1966240051 0 0) 0x7f746c000ac0 con 0x33435b0 -36> 2015-08-31 10:48:36.773367 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.106:6828/31879 -- osd_op(client.7111116.0:31 rbd_data.d54542ae8944a.0000000000000010@snapdir [list-snaps] 2.d403c06d ack+read+known_if_redirected e101238) v5 -- ?+0 0x3344930 con 0x32cee40 -35> 2015-08-31 10:48:36.773766 7f7528215700 1 -- 10.225.17.34:0/1016056 <== osd.373 10.225.15.106:6828/31879 2 ==== osd_op_reply(31 rbd_data.d54542ae8944a.0000000000000010 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (259675761 0 0) 0x7f74e8000d80 con 0x32cee40 -34> 2015-08-31 10:48:36.773891 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.94:6820/30165 -- osd_op(client.7111116.0:32 rbd_data.d54542ae8944a.0000000000000011@snapdir [list-snaps] 2.67392401 ack+read+known_if_redirected e101238) v5 -- ?+0 0x334b180 con 0x3349e00 -33> 2015-08-31 10:48:36.775907 7f748f8f8700 1 -- 10.225.17.34:0/1016056 <== osd.53 10.225.16.94:6820/30165 1 ==== osd_op_reply(32 rbd_data.d54542ae8944a.0000000000000011 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (3201921814 0 0) 0x7f7464000ac0 con 0x3349e00 -32> 2015-08-31 10:48:36.776037 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.94:6804/24462 -- osd_op(client.7111116.0:33 rbd_data.d54542ae8944a.0000000000000012@snapdir [list-snaps] 2.7ca538ab ack+read+known_if_redirected e101238) v5 -- ?+0 0x33517c0 con 0x3350440 -31> 2015-08-31 10:48:36.777030 7f748f6f6700 1 -- 10.225.17.34:0/1016056 <== osd.86 10.225.15.94:6804/24462 1 ==== osd_op_reply(33 rbd_data.d54542ae8944a.0000000000000012 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (666193108 0 0) 0x7f745c000ac0 con 0x3350440 -30> 2015-08-31 10:48:36.777169 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.106:6848/15736 -- osd_op(client.7111116.0:34 rbd_data.d54542ae8944a.0000000000000013@snapdir [list-snaps] 2.6fc47763 ack+read+known_if_redirected e101238) v5 -- ?+0 0x3357f90 con 0x3356a80 -29> 2015-08-31 10:48:36.779649 7f748f4f4700 1 -- 10.225.17.34:0/1016056 <== osd.423 10.225.16.106:6848/15736 1 ==== osd_op_reply(34 rbd_data.d54542ae8944a.0000000000000013 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (556878703 0 0) 0x7f7454000ac0 con 0x3356a80 -28> 2015-08-31 10:48:36.779779 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.11.94:6841/24799 -- osd_op(client.7111116.0:35 rbd_data.d54542ae8944a.0000000000000014@snapdir [list-snaps] 2.aaeabc41 ack+read+known_if_redirected e101238) v5 -- ?+0 0x335e530 con 0x335d1b0 -27> 2015-08-31 10:48:36.780706 7f748f2f2700 1 -- 10.225.17.34:0/1016056 <== osd.10 10.225.11.94:6841/24799 1 ==== osd_op_reply(35 rbd_data.d54542ae8944a.0000000000000014 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1120777360 0 0) 0x7f744c000ac0 con 0x335d1b0 -26> 2015-08-31 10:48:36.780836 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.11.114:6842/29079 -- osd_op(client.7111116.0:36 rbd_data.d54542ae8944a.0000000000000015@snapdir [list-snaps] 2.7444a68b ack+read+known_if_redirected e101238) v5 -- ?+0 0x3364b30 con 0x3363850 -25> 2015-08-31 10:48:36.781676 7f748f0f0700 1 -- 10.225.17.34:0/1016056 <== osd.199 10.225.11.114:6842/29079 1 ==== osd_op_reply(36 rbd_data.d54542ae8944a.0000000000000015 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (3720208708 0 0) 0x7f7444000ac0 con 0x3363850 -24> 2015-08-31 10:48:36.781804 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.14.122:6831/32731 -- osd_op(client.7111116.0:37 rbd_data.d54542ae8944a.0000000000000016@snapdir [list-snaps] 2.ff76e474 ack+read+known_if_redirected e101238) v5 -- ?+0 0x336b1b0 con 0x3369ed0 -23> 2015-08-31 10:48:36.782677 7f748eeee700 1 -- 10.225.17.34:0/1016056 <== osd.355 10.225.14.122:6831/32731 1 ==== osd_op_reply(37 rbd_data.d54542ae8944a.0000000000000016 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (2537200759 0 0) 0x7f743c000ac0 con 0x3369ed0 -22> 2015-08-31 10:48:36.782808 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.10.94:6803/3766 -- osd_op(client.7111116.0:38 rbd_data.d54542ae8944a.0000000000000017@snapdir [list-snaps] 2.9d1e7ce4 ack+read+known_if_redirected e101238) v5 -- ?+0 0x3371830 con 0x3370550 -21> 2015-08-31 10:48:36.784583 7f748ecec700 1 -- 10.225.17.34:0/1016056 <== osd.209 10.225.10.94:6803/3766 1 ==== osd_op_reply(38 rbd_data.d54542ae8944a.0000000000000017 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (3430010256 0 0) 0x7f7434000ac0 con 0x3370550 -20> 2015-08-31 10:48:36.784716 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.122:6821/9147 -- osd_op(client.7111116.0:39 rbd_data.d54542ae8944a.0000000000000018@snapdir [list-snaps] 2.b9e3d6cf ack+read+known_if_redirected e101238) v5 -- ?+0 0x3377eb0 con 0x3376bd0 -19> 2015-08-31 10:48:36.786391 7f748eaea700 1 -- 10.225.17.34:0/1016056 <== osd.402 10.225.15.122:6821/9147 1 ==== osd_op_reply(39 rbd_data.d54542ae8944a.0000000000000018 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1351301877 0 0) 0x7f742c000ac0 con 0x3376bd0 -18> 2015-08-31 10:48:36.786522 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.11.122:6816/6749 -- osd_op(client.7111116.0:40 rbd_data.d54542ae8944a.0000000000000019@snapdir [list-snaps] 2.35c5b942 ack+read+known_if_redirected e101238) v5 -- ?+0 0x337e530 con 0x337d250 -17> 2015-08-31 10:48:36.787482 7f748e8e8700 1 -- 10.225.17.34:0/1016056 <== osd.218 10.225.11.122:6816/6749 1 ==== osd_op_reply(40 rbd_data.d54542ae8944a.0000000000000019 [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (241031794 0 0) 0x7f7424000ac0 con 0x337d250 -16> 2015-08-31 10:48:36.787566 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.16.98:6828/18471 -- osd_op(client.7111116.0:41 rbd_data.d54542ae8944a.000000000000001a@snapdir [list-snaps] 2.6c3d260d ack+read+known_if_redirected e101238) v5 -- ?+0 0x337e530 con 0x331d020 -15> 2015-08-31 10:48:36.788025 7f75007ea700 1 -- 10.225.17.34:0/1016056 <== osd.63 10.225.16.98:6828/18471 2 ==== osd_op_reply(41 rbd_data.d54542ae8944a.000000000000001a [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1574500955 0 0) 0x7f74a0000ac0 con 0x331d020 -14> 2015-08-31 10:48:36.788156 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.12.110:6808/12914 -- osd_op(client.7111116.0:42 rbd_data.d54542ae8944a.000000000000001b@snapdir [list-snaps] 2.e993014e ack+read+known_if_redirected e101238) v5 -- ?+0 0x3384d90 con 0x3383ab0 -13> 2015-08-31 10:48:36.789463 7f748e6e6700 1 -- 10.225.17.34:0/1016056 <== osd.242 10.225.12.110:6808/12914 1 ==== osd_op_reply(42 rbd_data.d54542ae8944a.000000000000001b [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (2350802013 0 0) 0x7f741c000ac0 con 0x3383ab0 -12> 2015-08-31 10:48:36.789543 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.11.122:6816/6749 -- osd_op(client.7111116.0:43 rbd_data.d54542ae8944a.000000000000001c@snapdir [list-snaps] 2.d56535ba ack+read+known_if_redirected e101238) v5 -- ?+0 0x3384d90 con 0x337d250 -11> 2015-08-31 10:48:36.790275 7f748e8e8700 1 -- 10.225.17.34:0/1016056 <== osd.218 10.225.11.122:6816/6749 2 ==== osd_op_reply(43 rbd_data.d54542ae8944a.000000000000001c [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1977823735 0 0) 0x7f7424000ac0 con 0x337d250 -10> 2015-08-31 10:48:36.790401 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.102:6848/19224 -- osd_op(client.7111116.0:44 rbd_data.d54542ae8944a.000000000000001d@snapdir [list-snaps] 2.3e5f66f9 ack+read+known_if_redirected e101238) v5 -- ?+0 0x338b550 con 0x338a270 -9> 2015-08-31 10:48:36.791532 7f748e4e4700 1 -- 10.225.17.34:0/1016056 <== osd.366 10.225.15.102:6848/19224 1 ==== osd_op_reply(44 rbd_data.d54542ae8944a.000000000000001d [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (357182726 0 0) 0x7f7414000ac0 con 0x338a270 -8> 2015-08-31 10:48:36.791662 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.10.110:6829/445 -- osd_op(client.7111116.0:45 rbd_data.d54542ae8944a.000000000000001e@snapdir [list-snaps] 2.9a02c62e ack+read+known_if_redirected e101238) v5 -- ?+0 0x3391c10 con 0x3390930 -7> 2015-08-31 10:48:36.793089 7f748e2e2700 1 -- 10.225.17.34:0/1016056 <== osd.129 10.225.10.110:6829/445 1 ==== osd_op_reply(45 rbd_data.d54542ae8944a.000000000000001e [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (1100323165 0 0) 0x7f740c000ac0 con 0x3390930 -6> 2015-08-31 10:48:36.793169 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.15.102:6848/19224 -- osd_op(client.7111116.0:46 rbd_data.d54542ae8944a.000000000000001f@snapdir [list-snaps] 2.32bcf30f ack+read+known_if_redirected e101238) v5 -- ?+0 0x3391c10 con 0x338a270 -5> 2015-08-31 10:48:36.794065 7f748e4e4700 1 -- 10.225.17.34:0/1016056 <== osd.366 10.225.15.102:6848/19224 2 ==== osd_op_reply(46 rbd_data.d54542ae8944a.000000000000001f [list-snaps] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 206+0+0 (904285703 0 0) 0x7f7414000ac0 con 0x338a270 -4> 2015-08-31 10:48:36.794196 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.13.118:6827/851 -- osd_op(client.7111116.0:47 rbd_data.d54542ae8944a.0000000000000020@snapdir [list-snaps] 2.87090579 ack+read+known_if_redirected e101238) v5 -- ?+0 0x3398470 con 0x3397190 -3> 2015-08-31 10:48:36.795539 7f748e0e0700 1 -- 10.225.17.34:0/1016056 <== osd.264 10.225.13.118:6827/851 1 ==== osd_op_reply(47 rbd_data.d54542ae8944a.0000000000000020 [list-snaps] v0'0 uv291319 ondisk = 0) v6 ==== 206+0+352 (3274669500 0 3296251259) 0x7f7404000d80 con 0x3397190 -2> 2015-08-31 10:48:36.795678 7f7539f20840 1 -- 10.225.17.34:0/1016056 --> 10.225.11.118:6842/9323 -- osd_op(client.7111116.0:48 rbd_data.d54542ae8944a.0000000000000021@snapdir [list-snaps] 2.bf6b3efa ack+read+known_if_redirected e101238) v5 -- ?+0 0x339ebb0 con 0x339d8d0 -1> 2015-08-31 10:48:36.797120 7f748dede700 1 -- 10.225.17.34:0/1016056 <== osd.187 10.225.11.118:6842/9323 1 ==== osd_op_reply(48 rbd_data.d54542ae8944a.0000000000000021 [list-snaps] v0'0 uv0 ondisk = 0) v6 ==== 206+0+56 (3185468632 0 4227332015) 0x7f73fc000c50 con 0x339d8d0 0> 2015-08-31 10:48:36.797843 7f7539f20840 -1 *** Caught signal (Aborted) ** in thread 7f7539f20840 ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b) 1: rbd() [0x59690a] 2: (()+0x10340) [0x7f753467e340] 3: (gsignal()+0x39) [0x7f7532f37cc9] 4: (abort()+0x148) [0x7f7532f3b0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7533842535] 6: (()+0x5e6d6) [0x7f75338406d6] 7: (()+0x5e703) [0x7f7533840703] 8: (()+0x5e922) [0x7f7533840922] 9: (()+0x16fca8) [0x7f7537236ca8] 10: (()+0xa1268) [0x7f7537168268] 11: (()+0xa1ee6) [0x7f7537168ee6] 12: (()+0xa1701) [0x7f7537168701] 13: (librbd::Image::diff_iterate(char const*, unsigned long, unsigned long, int (*)(unsigned long, unsigned long, int, void*), void*)+0x5b) [0x7f7537113b1b] 14: (main()+0x441c) [0x4c2d6c] 15: (__libc_start_main()+0xf5) [0x7f7532f22ec5] 16: rbd() [0x4c7057] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 500 max_new 1000 log_file --- end dump of recent events --- Aborted (core dumped) thorvald@sc2-r17-u08:/t$ exit Script done on Mon 31 Aug 2015 10:48:53 AM PDT