Bug #7588: OSD Seg fault in string assign ObjectOperation::C_ObjectOperation_copyget::finish() - Ceph - Ceph

Custom queries

Backports: mimic
Backports: missing release
Backports: nautilus
Bluestore
Bug queue
Bug queue - no subprojects
Bug triage
Ceph backlog
Crash queue
Crash triage
Feature Requests
Feedback
My issues
Need Review
Pending backports
Priority queue
Product Backlog Scrub
Project Triage
Test Failures

Actions

Copy link

Bug #7588

closed

OSD Seg fault in string assign ObjectOperation::C_ObjectOperation_copyget::finish()

Added by David Zafman about 10 years ago. Updated over 9 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Samuel Just

Category:

OSD

Target version:

% Done:

Source:

other

Tags:

Backport:

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

ceph-osd.1.log.gz

dzafman-2014-02-28_12:09:58-rados:thrash-wip-7458-testing-basic-plana/111724

*** Caught signal (Segmentation fault) **
 in thread 7f991c37a700
 ceph version 0.77-618-g123ff9e (123ff9e18a27c6517a291aab1e0613b27d70d711)
 1: ceph-osd() [0x97e02a]
 2: (()+0xfcb0) [0x7f992ea20cb0]
 3: (()+0x9ebc3) [0x7f992da59bc3]
 4: (std::string::assign(std::string const&)+0x76) [0x7f992da5aff6]
 5: (ObjectOperation::C_ObjectOperation_copyget::finish(int)+0x19b) [0x8d707b]
 6: (Context::complete(int)+0x9) [0x66e399]
 7: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xb54) [0x70d624]
 8: (Objecter::dispatch(Message*)+0x1c5) [0x716e55]
 9: (OSDService::ObjecterDispatcher::ms_dispatch(Message*)+0x41) [0x614991]
 10: (DispatchQueue::entry()+0x549) [0xb021c9]
 11: (DispatchQueue::DispatchThread::entry()+0xd) [0xa4089d]
 12: (()+0x7e9a) [0x7f992ea18e9a]
 13: (clone()+0x6d) [0x7f992d1dd3fd]

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by David Zafman about 10 years ago

Description updated (diff)

Actions

Copy link

Updated by Loïc Dachary about 10 years ago

Can I help ?

Actions

Copy link

Updated by Ian Colle about 10 years ago

Assignee set to Loïc Dachary

Actions

Copy link

Updated by Loïc Dachary about 10 years ago

Description updated (diff)

Actions

Copy link

Updated by Samuel Just about 10 years ago

(10:44:51 AM) sjusthm: loicd: that one may be tricky to track down, you'll probably want to find a way for the structure into which the copyget operation is copying to have gone away
(10:45:32 AM) sjusthm: See ReplicatedPG::_copy_some for where the op originates
(10:45:53 AM) sjusthm: looks like cop->rval is the target, so the premature death of cop would probably explain it
(10:46:16 AM) sjusthm: oops
(10:46:28 AM) sjusthm: looks like a string, so not rval
(10:46:37 AM) sjusthm: perhaps a different bit of cop though

Actions

Copy link

Updated by Loïc Dachary about 10 years ago

there is just one string : C_ObjectOperation_copyget::out_category

Actions

Copy link

Updated by Loïc Dachary about 10 years ago

start_copy calls copy_some which calls Objecter::read which calls op_submit which calls take_op_budget which may block and drop the lock while doing so.

While the lock is down, another start_copy may call cancel_op which will drop the reference and deallocate the object, rendering the pointers stored in C_ObjectOperation_copyget invalid.

When take_op_budget returns, the operation starts and when it finishes, C_ObjectOperation_copyget is called on an invalid object.

If this is the source of the problem, maybe moving copy_ops[dest] = cop after _copy_some could fix this problem ?

Actions

Copy link

Updated by Sage Weil about 10 years ago

the blocking only happens if the limits are set... which we should probably just disable for the osd since we are already throttling client ops that we are servicing (and the locking here would get much more complex and fragile).

Actions

Copy link

Updated by Loïc Dachary about 10 years ago

Description updated (diff)

Actions

Copy link

#10

Updated by Loïc Dachary about 10 years ago

Category set to OSD
Status changed from New to In Progress

2014-03-01 02:06:08.687687 7f991fb81700  1 -- 10.214.131.24:6814/5231 --> 10.214.131.21:6800/4303 -- osd_op(osd.1.4:327 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 3.d44ec913 RETRY=2 ack+retry+read e29) v4 -- ?+0 0x3c77900 con 0x379a580
...
2014-03-01 02:06:10.778103 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.5 10.214.131.21:6800/4303 21 ==== osd_op_reply(327 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] v0'0 uv120 ondisk = 0) v6 ==== 223+0+3352119 (3017310025 0 3039343544) 0x3092c80 con 0x379a580
2014-03-01 02:06:10.781341 7f991c37a700 -1 *** Caught signal (Segmentation fault) **

Actions

Copy link

#11

Updated by Loïc Dachary about 10 years ago

Crashing thread 7f991c37a700

2014-03-01 02:04:56.959382 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 1 ==== osd_op_reply(1 plana166726-1 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (1924025897 0 0) 0x2b51c80 con 0x2d342c0
2014-03-01 02:04:56.960181 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 2 ==== osd_op_reply(2 plana166726-1 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (933392999 0 0) 0x2b51a00 con 0x2d342c0
2014-03-01 02:04:56.960379 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 3 ==== osd_op_reply(3 plana166726-3 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (2853962643 0 0) 0x2b51780 con 0x2d342c0
2014-03-01 02:04:56.960478 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 4 ==== osd_op_reply(4 plana166726-3 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (4010819549 0 0) 0x2b51280 con 0x2d342c0
2014-03-01 02:04:56.972329 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 5 ==== osd_op_reply(5 plana166726-4 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (254405978 0 0) 0x2b51a00 con 0x2d342c0
2014-03-01 02:04:56.972921 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 6 ==== osd_op_reply(6 plana166726-4 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (1243992340 0 0) 0x2b51c80 con 0x2d342c0
2014-03-01 02:04:57.255445 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 7 ==== osd_op_reply(7 plana166726-8 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (1934821051 0 0) 0x2ff7c80 con 0x2d342c0
2014-03-01 02:04:57.255589 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 8 ==== osd_op_reply(8 plana166726-8 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (912210677 0 0) 0x2ff7780 con 0x2d342c0
2014-03-01 02:04:57.344942 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 9 ==== osd_op_reply(9 plana166726-9 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (1098891299 0 0) 0x2ff7a00 con 0x2d342c0
2014-03-01 02:04:57.345140 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 10 ==== osd_op_reply(10 plana166726-9 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 180+0+0 (74710125 0 0) 0x2b51280 con 0x2d342c0
2014-03-01 02:04:57.556272 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 11 ==== osd_op_reply(11 plana166726-14 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (895035730 0 0) 0x30d9a00 con 0x2d342c0
2014-03-01 02:04:57.556520 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 12 ==== osd_op_reply(12 plana166726-14 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (1884624156 0 0) 0x30d9780 con 0x2d342c0
2014-03-01 02:04:57.702446 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 13 ==== osd_op_reply(13 plana166726-16 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (711175988 0 0) 0x30d9500 con 0x2d342c0
2014-03-01 02:04:57.703285 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 14 ==== osd_op_reply(14 plana166726-16 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (1869583226 0 0) 0x30d9280 con 0x2d342c0
2014-03-01 02:04:57.858423 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 15 ==== osd_op_reply(15 plana166726-19 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (986800301 0 0) 0x30d9000 con 0x2d342c0
2014-03-01 02:04:57.859607 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 16 ==== osd_op_reply(16 plana166726-19 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (2145213667 0 0) 0x30f4c80 con 0x2d342c0
2014-03-01 02:04:57.910883 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 17 ==== osd_op_reply(17 plana166726-21 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (1994807818 0 0) 0x30f4a00 con 0x2d342c0
2014-03-01 02:04:57.911229 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 18 ==== osd_op_reply(18 plana166726-21 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (870999620 0 0) 0x30f4780 con 0x2d342c0
2014-03-01 02:04:58.342792 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 19 ==== osd_op_reply(19 plana166726-35 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (2355955220 0 0) 0x30f4500 con 0x2d342c0
2014-03-01 02:04:58.342975 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 20 ==== osd_op_reply(20 plana166726-35 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (3378592346 0 0) 0x30f4280 con 0x2d342c0
2014-03-01 02:04:58.596438 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 21 ==== osd_op_reply(21 plana166726-41 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (3083637572 0 0) 0x30f4000 con 0x2d342c0
2014-03-01 02:04:58.596539 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 22 ==== osd_op_reply(22 plana166726-41 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (4072713994 0 0) 0x30d9000 con 0x2d342c0
2014-03-01 02:04:58.647432 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 23 ==== osd_op_reply(23 plana166726-43 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (2126479930 0 0) 0x2e89280 con 0x2d342c0
2014-03-01 02:04:58.647580 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 24 ==== osd_op_reply(24 plana166726-43 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (1001635444 0 0) 0x30f4280 con 0x2d342c0
2014-03-01 02:04:58.831568 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 25 ==== osd_op_reply(25 plana166726-49 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (1821373233 0 0) 0x30f4500 con 0x2d342c0
2014-03-01 02:04:58.832264 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 26 ==== osd_op_reply(26 plana166726-49 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (696504191 0 0) 0x30f4780 con 0x2d342c0
2014-03-01 02:04:58.945464 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 27 ==== osd_op_reply(27 plana166726-52 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (540979517 0 0) 0x30f4a00 con 0x2d342c0
2014-03-01 02:04:58.945670 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 28 ==== osd_op_reply(28 plana166726-52 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (1697811827 0 0) 0x30f4c80 con 0x2d342c0
2014-03-01 02:04:58.987415 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 29 ==== osd_op_reply(29 plana166726-54 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (3775203363 0 0) 0x2f15500 con 0x2d342c0
2014-03-01 02:04:58.988079 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 30 ==== osd_op_reply(30 plana166726-54 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (2752066669 0 0) 0x2f15780 con 0x2d342c0
2014-03-01 02:04:59.234729 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 31 ==== osd_op_reply(31 plana166726-59 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (737713726 0 0) 0x2f15280 con 0x2d342c0
2014-03-01 02:04:59.234833 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 32 ==== osd_op_reply(32 plana166726-59 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (1861526128 0 0) 0x2f15000 con 0x2d342c0
2014-03-01 02:04:59.688340 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 33 ==== osd_op_reply(33 plana166726-73 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (2439294750 0 0) 0x2eb6500 con 0x2d342c0
2014-03-01 02:04:59.688595 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 34 ==== osd_op_reply(34 plana166726-73 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (3563623248 0 0) 0x2eb6280 con 0x2d342c0
2014-03-01 02:05:00.035049 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 35 ==== osd_op_reply(35 plana166726-84 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (976823556 0 0) 0x2b51a00 con 0x2d342c0
2014-03-01 02:05:00.035289 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 36 ==== osd_op_reply(36 plana166726-84 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (2134186314 0 0) 0x2b51c80 con 0x2d342c0
2014-03-01 02:05:00.176410 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 37 ==== osd_op_reply(37 plana166726-89 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (962038265 0 0) 0x348bc80 con 0x2d342c0
2014-03-01 02:05:00.176592 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 38 ==== osd_op_reply(38 plana166726-89 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (2086385079 0 0) 0x348b280 con 0x2d342c0
2014-03-01 02:05:00.407005 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 39 ==== osd_op_reply(39 plana166726-94 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (1233101853 0 0) 0x2e89780 con 0x2d342c0
2014-03-01 02:05:00.408155 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 40 ==== osd_op_reply(40 plana166726-94 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (208902227 0 0) 0x30a9280 con 0x2d342c0
2014-03-01 02:05:00.438364 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 41 ==== osd_op_reply(41 plana166726-95 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (2724006611 0 0) 0x30a9a00 con 0x2d342c0
2014-03-01 02:05:00.440528 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 42 ==== osd_op_reply(42 plana166726-95 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 181+0+0 (3880859293 0 0) 0x30f6a00 con 0x2d342c0
2014-03-01 02:05:00.715624 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 43 ==== osd_op_reply(43 plana166726-102 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (965962363 0 0) 0x30f6280 con 0x2d342c0
2014-03-01 02:05:00.715853 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 44 ==== osd_op_reply(44 plana166726-102 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2090817077 0 0) 0x30f6500 con 0x2d342c0
2014-03-01 02:05:00.747306 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 45 ==== osd_op_reply(45 plana166726-103 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2171536080 0 0) 0x2ff7780 con 0x2d342c0
2014-03-01 02:05:00.748180 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 46 ==== osd_op_reply(46 plana166726-103 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3294838430 0 0) 0x30f4280 con 0x2d342c0
2014-03-01 02:05:01.167392 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 47 ==== osd_op_reply(47 plana166726-113 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3539134472 0 0) 0x348b000 con 0x2d342c0
2014-03-01 02:05:01.167438 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 48 ==== osd_op_reply(48 plana166726-113 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2550060102 0 0) 0x348b780 con 0x2d342c0
2014-03-01 02:05:01.312805 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 49 ==== osd_op_reply(49 plana166726-116 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (157849053 0 0) 0x348b280 con 0x2d342c0
2014-03-01 02:05:01.312910 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 50 ==== osd_op_reply(50 plana166726-116 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1281665427 0 0) 0x348bc80 con 0x2d342c0
2014-03-01 02:05:01.520829 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 51 ==== osd_op_reply(51 plana166726-122 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (350136798 0 0) 0x2ff7280 con 0x2d342c0
2014-03-01 02:05:01.521502 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 52 ==== osd_op_reply(52 plana166726-122 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1372755344 0 0) 0x30f6500 con 0x2d342c0
2014-03-01 02:05:01.621232 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 53 ==== osd_op_reply(57 plana166726-123 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2213501420 0 0) 0x30f6280 con 0x2d342c0
2014-03-01 02:05:01.622196 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 54 ==== osd_op_reply(58 plana166726-123 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3336791458 0 0) 0x30f6a00 con 0x2d342c0
2014-03-01 02:05:01.685354 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 55 ==== osd_op_reply(59 plana166726-124 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2924093414 0 0) 0x2e89780 con 0x2d342c0
2014-03-01 02:05:01.686148 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 56 ==== osd_op_reply(60 plana166726-124 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3947242408 0 0) 0x30a9c80 con 0x2d342c0
2014-03-01 02:05:01.741831 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 57 ==== osd_op_reply(53 plana166726-43 [copy-from ver 48] v15'1 uv48 ondisk = 0) v6 ==== 181+0+0 (4068454087 0 0) 0x30f6a00 con 0x2d342c0
2014-03-01 02:05:01.851251 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 58 ==== osd_op_reply(54 plana166726-59 [copy-from ver 66] v15'2 uv66 ondisk = 0) v6 ==== 181+0+0 (69082926 0 0) 0x30f6280 con 0x2d342c0
2014-03-01 02:05:01.854139 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 59 ==== osd_op_reply(55 plana166726-3 [copy-from ver 4] v15'3 uv4 ondisk = 0) v6 ==== 180+0+0 (1223249240 0 0) 0x30f6500 con 0x2d342c0
2014-03-01 02:05:01.854544 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 60 ==== osd_op_reply(56 plana166726-19 [copy-from ver 33] v15'4 uv33 ondisk = 0) v6 ==== 181+0+0 (1071806286 0 0) 0x2ff7280 con 0x2d342c0
2014-03-01 02:05:01.983355 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 61 ==== osd_op_reply(61 plana166726-52 [copy-from ver 56] v15'5 uv56 ondisk = 0) v6 ==== 181+0+0 (4251403393 0 0) 0x2ff7780 con 0x2d342c0
2014-03-01 02:05:02.029877 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 62 ==== osd_op_reply(62 plana166726-133 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (994804227 0 0) 0x348bc80 con 0x2d342c0
2014-03-01 02:05:02.030092 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 63 ==== osd_op_reply(63 plana166726-133 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2118630989 0 0) 0x348b280 con 0x2d342c0
2014-03-01 02:05:02.568158 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 64 ==== osd_op_reply(67 plana166726-147 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (61015251 0 0) 0x348b780 con 0x2d342c0
2014-03-01 02:05:02.570822 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 65 ==== osd_op_reply(68 plana166726-147 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1185888413 0 0) 0x348b000 con 0x2d342c0
2014-03-01 02:05:02.622051 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 66 ==== osd_op_reply(69 plana166726-151 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3862845756 0 0) 0x30f4280 con 0x2d342c0
2014-03-01 02:05:02.622932 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 67 ==== osd_op_reply(70 plana166726-151 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2737988978 0 0) 0x2f15280 con 0x2d342c0
2014-03-01 02:05:02.641609 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 1 ==== osd_op_reply(64 plana166726-119 [assert-version v122,copy-get max 8388608 ver 0] v0'0 uv122 ondisk = 0) v6 ==== 224+0+2224009 (1975598499 0 99600142) 0x3243000 con 0x2e47420
2014-03-01 02:05:02.668040 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 2 ==== osd_op_reply(65 plana166726-39 [assert-version v35,copy-get max 8388608 ver 0] v0'0 uv35 ondisk = 0) v6 ==== 223+0+2746429 (1814759532 0 4133966298) 0x3243780 con 0x2e47420
2014-03-01 02:05:02.683715 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 3 ==== osd_op_reply(66 plana166726-53 [assert-version v49,copy-get max 8388608 ver 0] v0'0 uv49 ondisk = 0) v6 ==== 223+0+1472029 (232138067 0 4243642541) 0x3243500 con 0x2e47420
2014-03-01 02:05:02.810899 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 4 ==== osd_op_reply(75 plana166726-72 [assert-version v78,copy-get max 8388608 ver 0] v0'0 uv78 ondisk = 0) v6 ==== 223+0+2338930 (3849531516 0 1459515309) 0x3196a00 con 0x2e47420
2014-03-01 02:05:02.942991 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 68 ==== osd_op_reply(71 plana166726-9 [copy-from ver 20] v15'6 uv20 ondisk = 0) v6 ==== 180+0+0 (2349705573 0 0) 0x348b000 con 0x2d342c0
2014-03-01 02:05:02.943825 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 69 ==== osd_op_reply(72 plana166726-123 [copy-from ver 114] v15'7 uv114 ondisk = 0) v6 ==== 182+0+0 (75225541 0 0) 0x348b780 con 0x2d342c0
2014-03-01 02:05:02.997502 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 70 ==== osd_op_reply(76 plana166726-156 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1599493504 0 0) 0x348b280 con 0x2d342c0
2014-03-01 02:05:02.997724 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 71 ==== osd_op_reply(77 plana166726-156 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (442132942 0 0) 0x348bc80 con 0x2d342c0
2014-03-01 02:05:03.035161 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 72 ==== osd_op_reply(73 plana166726-41 [copy-from ver 45] v15'8 uv45 ondisk = 0) v6 ==== 181+0+0 (2443699088 0 0) 0x2ff7780 con 0x2d342c0
2014-03-01 02:05:03.037058 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 73 ==== osd_op_reply(74 plana166726-54 [copy-from ver 61] v15'9 uv61 ondisk = 0) v6 ==== 181+0+0 (2559430550 0 0) 0x2ff7280 con 0x2d342c0
2014-03-01 02:05:03.159958 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 74 ==== osd_op_reply(78 plana166726-158 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (4253267684 0 0) 0x30f6500 con 0x2d342c0
2014-03-01 02:05:03.160767 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 75 ==== osd_op_reply(79 plana166726-158 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3096414890 0 0) 0x30f6280 con 0x2d342c0
2014-03-01 02:05:03.174187 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 5 ==== osd_op_reply(80 plana166726-28 [assert-version v21,copy-get max 8388608 ver 0] v0'0 uv21 ondisk = 0) v6 ==== 223+0+1203670 (3500051171 0 3086037950) 0x348b500 con 0x2e47420
2014-03-01 02:05:03.181046 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 6 ==== osd_op_reply(81 plana166726-141 [assert-version v166,copy-get max 8388608 ver 0] v0'0 uv166 ondisk = 0) v6 ==== 224+0+792151 (2326199858 0 2304654153) 0x3322000 con 0x2e47420
2014-03-01 02:05:03.286709 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 7 ==== osd_op_reply(82 plana166726-142 [assert-version v170,copy-get max 8388608 ver 0] v0'0 uv170 ondisk = 0) v6 ==== 224+0+1535957 (2907898514 0 3255729770) 0x2b51780 con 0x2e47420
2014-03-01 02:05:03.311152 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 8 ==== osd_op_reply(83 plana166726-63 [assert-version v67,copy-get max 8388608 ver 0] v0'0 uv67 ondisk = 0) v6 ==== 223+0+2802182 (2052921004 0 3077166307) 0x2b51c80 con 0x2e47420
2014-03-01 02:05:03.695439 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 9 ==== osd_op_reply(84 plana166726-106 [assert-version v104,copy-get max 8388608 ver 0] v0'0 uv104 ondisk = 0) v6 ==== 224+0+3193638 (3371960255 0 3091549231) 0x2eb6280 con 0x2e47420
2014-03-01 02:05:03.752462 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 76 ==== osd_op_reply(85 plana166726-173 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (4168336875 0 0) 0x30f6a00 con 0x2d342c0
2014-03-01 02:05:03.752622 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 77 ==== osd_op_reply(86 plana166726-173 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3179280805 0 0) 0x30a9c80 con 0x2d342c0
2014-03-01 02:05:04.404107 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 10 ==== osd_op_reply(87 plana166726-75 [assert-version v81,copy-get max 8388608 ver 0] v0'0 uv81 ondisk = 0) v6 ==== 223+0+40171 (1364689452 0 2603956677) 0x3196c80 con 0x2e47420
2014-03-01 02:05:06.957676 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 11 ==== osd_op_reply(88 plana166726-137 [assert-version v152,copy-get max 8388608 ver 0] v0'0 uv152 ondisk = 0) v6 ==== 224+0+2347104 (1781384165 0 2869034318) 0x3337c80 con 0x2e47420
2014-03-01 02:05:06.982360 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 12 ==== osd_op_reply(89 plana166726-127 [assert-version v130,copy-get max 8388608 ver 0] v0'0 uv130 ondisk = 0) v6 ==== 224+0+1549175 (1398840844 0 2806554450) 0x3337a00 con 0x2e47420
2014-03-01 02:05:07.529605 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 78 ==== osd_op_reply(92 plana166726-196 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3888611212 0 0) 0x2e89780 con 0x2d342c0
2014-03-01 02:05:07.529762 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 79 ==== osd_op_reply(93 plana166726-196 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2731230146 0 0) 0x3322500 con 0x2d342c0
2014-03-01 02:05:07.672856 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 80 ==== osd_op_reply(100 plana166726-199 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1022873070 0 0) 0x3322000 con 0x2d342c0
2014-03-01 02:05:07.673411 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 81 ==== osd_op_reply(101 plana166726-199 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2046554528 0 0) 0x30d9780 con 0x2d342c0
2014-03-01 02:05:07.736662 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 82 ==== osd_op_reply(102 plana166726-201 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3466913496 0 0) 0x2a8fc80 con 0x2d342c0
2014-03-01 02:05:07.736844 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 83 ==== osd_op_reply(103 plana166726-201 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2343099030 0 0) 0x2e89780 con 0x2d342c0
2014-03-01 02:05:07.793545 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 84 ==== osd_op_reply(94 plana166726-122 [copy-from ver 108] v16'10 uv108 ondisk = 0) v6 ==== 182+0+0 (193412729 0 0) 0x30a9c80 con 0x2d342c0
2014-03-01 02:05:07.803306 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 85 ==== osd_op_reply(104 plana166726-202 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1142147340 0 0) 0x30f6a00 con 0x2d342c0
2014-03-01 02:05:07.804458 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 86 ==== osd_op_reply(105 plana166726-202 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (18871618 0 0) 0x30f6280 con 0x2d342c0
2014-03-01 02:05:07.894896 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 87 ==== osd_op_reply(95 plana166726-113 [copy-from ver 100] v16'11 uv100 ondisk = 0) v6 ==== 182+0+0 (4031713849 0 0) 0x30f6500 con 0x2d342c0
2014-03-01 02:05:07.896165 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 88 ==== osd_op_reply(96 plana166726-21 [copy-from ver 37] v16'12 uv37 ondisk = 0) v6 ==== 181+0+0 (2892397456 0 0) 0x2eb6280 con 0x2d342c0
2014-03-01 02:05:07.896898 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 89 ==== osd_op_reply(97 plana166726-133 [copy-from ver 125] v16'13 uv125 ondisk = 0) v6 ==== 182+0+0 (2513659596 0 0) 0x2b51c80 con 0x2d342c0
2014-03-01 02:05:07.930015 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 90 ==== osd_op_reply(106 plana166726-204 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (418047144 0 0) 0x2b51780 con 0x2d342c0
2014-03-01 02:05:07.930198 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 91 ==== osd_op_reply(107 plana166726-204 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1575409894 0 0) 0x348b500 con 0x2d342c0
2014-03-01 02:05:08.144952 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 92 ==== osd_op_reply(108 plana166726-73 [copy-from ver 71] v16'14 uv71 ondisk = 0) v6 ==== 181+0+0 (3072431112 0 0) 0x3243280 con 0x2d342c0
2014-03-01 02:05:08.205823 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 13 ==== osd_op_reply(90 plana166726-159 [assert-version v199,copy-get max 8388608 ver 0] v0'0 uv199 ondisk = 0) v6 ==== 224+0+914222 (3845951385 0 3936463237) 0x3337780 con 0x2e47420
2014-03-01 02:05:08.224478 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 93 ==== osd_op_reply(112 plana166726-207 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1692090446 0 0) 0x3243a00 con 0x2d342c0
2014-03-01 02:05:08.225804 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 94 ==== osd_op_reply(113 plana166726-207 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (567770112 0 0) 0x3196c80 con 0x2d342c0
2014-03-01 02:05:08.231355 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 14 ==== osd_op_reply(91 plana166726-146 [assert-version v179,copy-get max 8388608 ver 0] v0'0 uv179 ondisk = 0) v6 ==== 224+0+2932428 (3505807453 0 3383342168) 0x3322000 con 0x2e47420
2014-03-01 02:05:08.244709 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 95 ==== osd_op_reply(109 plana166726-4 [copy-from ver 10] v16'15 uv10 ondisk = 0) v6 ==== 180+0+0 (720503678 0 0) 0x3196a00 con 0x2d342c0
2014-03-01 02:05:08.245636 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 96 ==== osd_op_reply(110 plana166726-102 [copy-from ver 91] v16'16 uv91 ondisk = 0) v6 ==== 182+0+0 (2201540663 0 0) 0x3337280 con 0x2d342c0
2014-03-01 02:05:08.245769 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 15 ==== osd_op_reply(98 plana166726-110 [assert-version v114,copy-get max 8388608 ver 0] v0'0 uv114 ondisk = 0) v6 ==== 224+0+1637661 (1969485706 0 1577459536) 0x3322500 con 0x2e47420
2014-03-01 02:05:08.272571 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 97 ==== osd_op_reply(111 plana166726-199 [copy-from ver 161] v16'17 uv161 ondisk = 0) v6 ==== 182+0+0 (2947569692 0 0) 0x3337c80 con 0x2d342c0
2014-03-01 02:05:08.274816 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 98 ==== osd_op_reply(114 plana166726-209 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (4029883730 0 0) 0x30d9500 con 0x2d342c0
2014-03-01 02:05:08.274896 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 99 ==== osd_op_reply(115 plana166726-209 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3040811292 0 0) 0x3322780 con 0x2d342c0
2014-03-01 02:05:08.279987 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 16 ==== osd_op_reply(99 plana166726-18 [assert-version v13,copy-get max 8388608 ver 0] v0'0 uv13 ondisk = 0) v6 ==== 223+0+3874540 (625797611 0 1579601517) 0x30f4c80 con 0x2e47420
2014-03-01 02:05:08.328752 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 100 ==== osd_op_reply(116 plana166726-210 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3961401267 0 0) 0x2e89280 con 0x2d342c0
2014-03-01 02:05:08.330907 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 101 ==== osd_op_reply(117 plana166726-210 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2836534269 0 0) 0x3243280 con 0x2d342c0
2014-03-01 02:05:08.365810 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 102 ==== osd_op_reply(118 plana166726-211 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2725968966 0 0) 0x348b500 con 0x2d342c0
2014-03-01 02:05:08.366961 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 103 ==== osd_op_reply(119 plana166726-211 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3883319304 0 0) 0x2b51780 con 0x2d342c0
2014-03-01 02:05:08.448010 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 104 ==== osd_op_reply(124 plana166726-214 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (873579256 0 0) 0x2b51c80 con 0x2d342c0
2014-03-01 02:05:08.449165 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 105 ==== osd_op_reply(125 plana166726-214 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1897791158 0 0) 0x2b51500 con 0x2d342c0
2014-03-01 02:05:08.463294 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 106 ==== osd_op_reply(126 plana166726-215 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1947813125 0 0) 0x2eb6280 con 0x2d342c0
2014-03-01 02:05:08.463475 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 107 ==== osd_op_reply(127 plana166726-215 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (823490891 0 0) 0x30f6500 con 0x2d342c0
2014-03-01 02:05:08.519356 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 108 ==== osd_op_reply(120 plana166726-103 [copy-from ver 95] v16'18 uv95 ondisk = 0) v6 ==== 182+0+0 (2449368900 0 0) 0x3243a00 con 0x2d342c0
2014-03-01 02:05:08.519749 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 109 ==== osd_op_reply(121 plana166726-94 [copy-from ver 83] v16'19 uv83 ondisk = 0) v6 ==== 181+0+0 (4127339441 0 0) 0x3196780 con 0x2d342c0
2014-03-01 02:05:08.521508 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 110 ==== osd_op_reply(122 plana166726-207 [copy-from ver 184] v16'20 uv184 ondisk = 0) v6 ==== 182+0+0 (1603446543 0 0) 0x2ff7280 con 0x2d342c0
2014-03-01 02:05:08.613813 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 111 ==== osd_op_reply(123 plana166726-89 [copy-from ver 80] v16'21 uv80 ondisk = 0) v6 ==== 181+0+0 (554472843 0 0) 0x3322a00 con 0x2d342c0
2014-03-01 02:05:08.774060 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 112 ==== osd_op_reply(128 plana166726-223 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2361130706 0 0) 0x2f15280 con 0x2d342c0
2014-03-01 02:05:08.774509 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 113 ==== osd_op_reply(129 plana166726-223 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3384263324 0 0) 0x348bc80 con 0x2d342c0
2014-03-01 02:05:08.845783 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 114 ==== osd_op_reply(130 plana166726-226 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1637226703 0 0) 0x3337500 con 0x2d342c0
2014-03-01 02:05:08.846421 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 115 ==== osd_op_reply(131 plana166726-226 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (614081665 0 0) 0x2ab6a00 con 0x2d342c0
2014-03-01 02:05:09.047318 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 116 ==== osd_op_reply(132 plana166726-196 [copy-from ver 158] v16'22 uv158 ondisk = 0) v6 ==== 182+0+0 (3091539574 0 0) 0x2ab6c80 con 0x2d342c0
2014-03-01 02:05:09.048203 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 117 ==== osd_op_reply(133 plana166726-202 [copy-from ver 171] v16'23 uv171 ondisk = 0) v6 ==== 182+0+0 (4160455198 0 0) 0x30a9280 con 0x2d342c0
2014-03-01 02:05:09.170162 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 118 ==== osd_op_reply(134 plana166726-147 [copy-from ver 130] v16'24 uv130 ondisk = 0) v6 ==== 182+0+0 (2209881480 0 0) 0x30a9000 con 0x2d342c0
2014-03-01 02:05:09.176965 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 119 ==== osd_op_reply(135 plana166726-49 [copy-from ver 53] v16'25 uv53 ondisk = 0) v6 ==== 181+0+0 (1576452199 0 0) 0x3089500 con 0x2d342c0
2014-03-01 02:05:09.183911 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 120 ==== osd_op_reply(136 plana166726-232 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2386265897 0 0) 0x3089280 con 0x2d342c0
2014-03-01 02:05:09.184098 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 121 ==== osd_op_reply(137 plana166726-232 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3409427303 0 0) 0x3089780 con 0x2d342c0
2014-03-01 02:05:09.205708 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 122 ==== osd_op_reply(138 plana166726-233 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3875182697 0 0) 0x3089a00 con 0x2d342c0
2014-03-01 02:05:09.206316 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 123 ==== osd_op_reply(139 plana166726-233 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2750850087 0 0) 0x2ab6000 con 0x2d342c0
2014-03-01 02:05:09.247259 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 124 ==== osd_op_reply(140 plana166726-234 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1165386427 0 0) 0x2ab6500 con 0x2d342c0
2014-03-01 02:05:09.492365 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 17 ==== osd_op_reply(146 plana166726-62 [assert-version v62,copy-get max 8388608 ver 0] v0'0 uv62 ondisk = 0) v6 ==== 223+0+1008777 (4109934101 0 2729595614) 0x3196280 con 0x2e47420
2014-03-01 02:05:09.497862 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 18 ==== osd_op_reply(147 plana166726-134 [assert-version v148,copy-get max 8388608 ver 0] v0'0 uv148 ondisk = 0) v6 ==== 224+0+645863 (1822035094 0 3264362420) 0x2f15500 con 0x2e47420
2014-03-01 02:05:09.741868 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 19 ==== osd_op_reply(148 plana166726-192 [assert-version v246,copy-get max 8388608 ver 0] v0'0 uv246 ondisk = 0) v6 ==== 224+0+1556323 (1738035508 0 2920206387) 0x3337a00 con 0x2e47420
2014-03-01 02:05:09.741935 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 20 ==== osd_op_reply(149 plana166726-217 [assert-version v260,copy-get max 8388608 ver 0] v0'0 uv260 ondisk = 0) v6 ==== 224+0+1308331 (629590980 0 1970292937) 0x2ff7280 con 0x2e47420
2014-03-01 02:05:09.990485 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 125 ==== osd_op_reply(141 plana166726-234 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (8019701 0 0) 0x2ab6780 con 0x2d342c0
2014-03-01 02:05:09.990578 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 126 ==== osd_op_reply(142 plana166726-237 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3106516749 0 0) 0x3322780 con 0x2d342c0
2014-03-01 02:05:09.990621 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 127 ==== osd_op_reply(143 plana166726-237 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (4230320963 0 0) 0x3322000 con 0x2d342c0
2014-03-01 02:05:09.990655 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 128 ==== osd_op_reply(144 plana166726-238 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (489336107 0 0) 0x348b500 con 0x2d342c0
2014-03-01 02:05:09.990687 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 129 ==== osd_op_reply(145 plana166726-238 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1478922597 0 0) 0x348ba00 con 0x2d342c0
2014-03-01 02:05:09.990719 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 130 ==== osd_op_reply(150 plana166726-242 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (4225699428 0 0) 0x30f4c80 con 0x2d342c0
2014-03-01 02:05:09.990746 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 131 ==== osd_op_reply(151 plana166726-242 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3201511978 0 0) 0x30f4780 con 0x2d342c0
2014-03-01 02:05:09.990780 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 132 ==== osd_op_reply(152 plana166726-244 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1210712426 0 0) 0x2f15780 con 0x2d342c0
2014-03-01 02:05:09.990812 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 133 ==== osd_op_reply(153 plana166726-244 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (220609828 0 0) 0x2f15a00 con 0x2d342c0
2014-03-01 02:05:09.990847 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 134 ==== osd_op_reply(154 plana166726-245 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (673924236 0 0) 0x30a9780 con 0x2d342c0
2014-03-01 02:05:09.990878 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 135 ==== osd_op_reply(155 plana166726-245 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1831303362 0 0) 0x30a9a00 con 0x2d342c0
2014-03-01 02:05:10.297008 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 136 ==== osd_op_reply(156 plana166726-8 [copy-from ver 15] v16'26 uv15 ondisk = 0) v6 ==== 180+0+0 (3405244980 0 0) 0x30a9500 con 0x2d342c0
2014-03-01 02:05:10.393429 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 21 ==== osd_op_reply(160 plana166726-92 [assert-version v91,copy-get max 8388608 ver 0] v0'0 uv91 ondisk = 0) v6 ==== 223+0+1515589 (1250138003 0 1972831098) 0x3196780 con 0x2e47420
2014-03-01 02:05:10.435757 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 22 ==== osd_op_reply(161 plana166726-203 [assert-version v253,copy-get max 8388608 ver 0] v0'0 uv253 ondisk = 0) v6 ==== 224+0+2581707 (2514238599 0 1720203046) 0x3243a00 con 0x2e47420
2014-03-01 02:05:10.481248 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 137 ==== osd_op_reply(157 plana166726-116 [copy-from ver 103] v16'27 uv103 ondisk = 0) v6 ==== 182+0+0 (14354921 0 0) 0x30a9000 con 0x2d342c0
2014-03-01 02:05:10.483597 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 138 ==== osd_op_reply(158 plana166726-210 [copy-from ver 195] v16'28 uv195 ondisk = 0) v6 ==== 182+0+0 (924482996 0 0) 0x30a9280 con 0x2d342c0
2014-03-01 02:05:10.483636 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 139 ==== osd_op_reply(159 plana166726-226 [copy-from ver 217] v16'29 uv217 ondisk = 0) v6 ==== 182+0+0 (2374564819 0 0) 0x3089500 con 0x2d342c0
2014-03-01 02:05:10.497726 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 23 ==== osd_op_reply(162 plana166726-157 [assert-version v196,copy-get max 8388608 ver 0] v0'0 uv196 ondisk = 0) v6 ==== 224+0+3958835 (3313213254 0 2707442551) 0x30f6500 con 0x2e47420
2014-03-01 02:05:10.509598 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 24 ==== osd_op_reply(163 plana166726-185 [assert-version v230,copy-get max 8388608 ver 0] v0'0 uv230 ondisk = 0) v6 ==== 224+0+1411059 (3623452832 0 700086051) 0x2eb6280 con 0x2e47420
2014-03-01 02:05:10.665938 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 140 ==== osd_op_reply(165 plana166726-260 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (538371960 0 0) 0x3089c80 con 0x2d342c0
2014-03-01 02:05:10.668320 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 141 ==== osd_op_reply(166 plana166726-260 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1696258870 0 0) 0x2ab6c80 con 0x2d342c0
2014-03-01 02:05:10.735329 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 142 ==== osd_op_reply(170 plana166726-261 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (978749647 0 0) 0x2ab6a00 con 0x2d342c0
2014-03-01 02:05:10.736538 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 143 ==== osd_op_reply(171 plana166726-261 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2136650881 0 0) 0x2ab6280 con 0x2d342c0
2014-03-01 02:05:10.785439 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 144 ==== osd_op_reply(172 plana166726-262 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (281449667 0 0) 0x3337500 con 0x2d342c0
2014-03-01 02:05:10.790570 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 145 ==== osd_op_reply(173 plana166726-262 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1439344781 0 0) 0x3337c80 con 0x2d342c0
2014-03-01 02:05:10.794209 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 146 ==== osd_op_reply(174 plana166726-263 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1122189780 0 0) 0x3337280 con 0x2d342c0
2014-03-01 02:05:10.794412 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 147 ==== osd_op_reply(175 plana166726-263 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (133135770 0 0) 0x348bc80 con 0x2d342c0
2014-03-01 02:05:10.807787 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 148 ==== osd_op_reply(164 plana166726-16 [copy-from ver 30] v16'30 uv30 ondisk = 0) v6 ==== 181+0+0 (103636126 0 0) 0x348b280 con 0x2d342c0
2014-03-01 02:05:10.836080 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 149 ==== osd_op_reply(176 plana166726-265 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1962762728 0 0) 0x348b000 con 0x2d342c0
2014-03-01 02:05:10.838044 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 150 ==== osd_op_reply(177 plana166726-265 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (837901734 0 0) 0x2eb6280 con 0x2d342c0
2014-03-01 02:05:10.972741 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 151 ==== osd_op_reply(167 plana166726-238 [copy-from ver 257] v16'31 uv257 ondisk = 0) v6 ==== 182+0+0 (2635300473 0 0) 0x2eb6000 con 0x2d342c0
2014-03-01 02:05:10.973569 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 152 ==== osd_op_reply(168 plana166726-173 [copy-from ver 153] v16'32 uv153 ondisk = 0) v6 ==== 182+0+0 (1555586537 0 0) 0x30f6500 con 0x2d342c0
2014-03-01 02:05:11.012364 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 25 ==== osd_op_reply(178 plana166726-82 [assert-version v87,copy-get max 8388608 ver 0] v0'0 uv87 ondisk = 0) v6 ==== 223+0+3669083 (494619272 0 3230139923) 0x2b51500 con 0x2e47420
2014-03-01 02:05:11.031841 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 153 ==== osd_op_reply(179 plana166726-266 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3280391461 0 0) 0x30a9280 con 0x2d342c0
2014-03-01 02:05:11.033499 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 154 ==== osd_op_reply(180 plana166726-266 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2257252715 0 0) 0x30a9000 con 0x2d342c0
2014-03-01 02:05:11.088400 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 155 ==== osd_op_reply(169 plana166726-1 [copy-from ver 6] v16'33 uv6 ondisk = 0) v6 ==== 180+0+0 (3239023423 0 0) 0x30a9500 con 0x2d342c0
2014-03-01 02:05:11.138602 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 26 ==== osd_op_reply(182 plana166726-252 [assert-version v298,copy-get max 8388608 ver 0] v0'0 uv298 ondisk = 0) v6 ==== 224+0+2906844 (1771531508 0 3437903599) 0x2b51c80 con 0x2e47420
2014-03-01 02:05:11.141980 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 27 ==== osd_op_reply(183 plana166726-42 [assert-version v38,copy-get max 8388608 ver 0] v0'0 uv38 ondisk = 0) v6 ==== 223+0+446646 (3689954181 0 2161629643) 0x30f4000 con 0x2e47420
2014-03-01 02:05:11.146845 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 156 ==== osd_op_reply(185 plana166726-268 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1618358522 0 0) 0x30a9a00 con 0x2d342c0
2014-03-01 02:05:11.156390 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 28 ==== osd_op_reply(184 plana166726-45 [assert-version v45,copy-get max 8388608 ver 0] v0'0 uv45 ondisk = 0) v6 ==== 223+0+1630511 (1307997274 0 2405755019) 0x30f4780 con 0x2e47420
2014-03-01 02:05:11.184060 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 157 ==== osd_op_reply(186 plana166726-268 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (628755636 0 0) 0x30a9780 con 0x2d342c0
2014-03-01 02:05:11.302192 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 158 ==== osd_op_reply(189 plana166726-271 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (888108970 0 0) 0x3243a00 con 0x2d342c0
2014-03-01 02:05:11.302340 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 159 ==== osd_op_reply(190 plana166726-271 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1910721508 0 0) 0x3243500 con 0x2d342c0
2014-03-01 02:05:11.363718 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 160 ==== osd_op_reply(181 plana166726-233 [copy-from ver 229] v16'34 uv229 ondisk = 0) v6 ==== 182+0+0 (2163666131 0 0) 0x3243280 con 0x2d342c0
2014-03-01 02:05:11.475997 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 161 ==== osd_op_reply(191 plana166726-275 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3425308634 0 0) 0x3196780 con 0x2d342c0
2014-03-01 02:05:11.476179 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 162 ==== osd_op_reply(192 plana166726-275 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2300982164 0 0) 0x3196500 con 0x2d342c0
2014-03-01 02:05:11.478456 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 163 ==== osd_op_reply(187 plana166726-262 [copy-from ver 280] v16'35 uv280 ondisk = 0) v6 ==== 182+0+0 (2370771443 0 0) 0x2f15a00 con 0x2d342c0
2014-03-01 02:05:11.766499 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 164 ==== osd_op_reply(194 plana166726-277 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (931106386 0 0) 0x2f15500 con 0x2d342c0
2014-03-01 02:05:11.766724 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 165 ==== osd_op_reply(195 plana166726-277 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1920184860 0 0) 0x2b51a00 con 0x2d342c0
2014-03-01 02:05:11.953482 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 166 ==== osd_op_reply(196 plana166726-279 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2057989864 0 0) 0x30d9500 con 0x2d342c0
2014-03-01 02:05:11.954080 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 167 ==== osd_op_reply(197 plana166726-279 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1067864742 0 0) 0x30d9280 con 0x2d342c0
2014-03-01 02:05:12.347569 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 168 ==== osd_op_reply(188 plana166726-95 [copy-from ver 87] v16'36 uv87 ondisk = 0) v6 ==== 181+0+0 (2450118466 0 0) 0x30d9000 con 0x2d342c0
2014-03-01 02:05:12.352485 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 169 ==== osd_op_reply(193 plana166726-245 [copy-from ver 246] v17'37 uv246 ondisk = 0) v6 ==== 182+0+0 (3276059143 0 0) 0x3232c80 con 0x2d342c0
2014-03-01 02:05:12.618042 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 170 ==== osd_op_reply(200 plana166726-287 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2180204200 0 0) 0x3232a00 con 0x2d342c0
2014-03-01 02:05:12.618295 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 171 ==== osd_op_reply(201 plana166726-287 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3305077478 0 0) 0x3232780 con 0x2d342c0
2014-03-01 02:05:12.748235 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 172 ==== osd_op_reply(198 plana166726-265 [copy-from ver 288] v17'38 uv288 ondisk = 0) v6 ==== 182+0+0 (358423234 0 0) 0x3232500 con 0x2d342c0
2014-03-01 02:05:12.748672 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 173 ==== osd_op_reply(199 plana166726-275 [copy-from ver 314] v17'39 uv314 ondisk = 0) v6 ==== 182+0+0 (2593179115 0 0) 0x3232280 con 0x2d342c0
2014-03-01 02:05:12.944336 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 174 ==== osd_op_reply(204 plana166726-294 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1595161219 0 0) 0x3232000 con 0x2d342c0
2014-03-01 02:05:12.945107 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 175 ==== osd_op_reply(205 plana166726-294 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (437782221 0 0) 0x30a9000 con 0x2d342c0
2014-03-01 02:05:12.972994 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 176 ==== osd_op_reply(202 plana166726-232 [copy-from ver 224] v17'40 uv224 ondisk = 0) v6 ==== 182+0+0 (1897310094 0 0) 0x30a9a00 con 0x2d342c0
2014-03-01 02:05:12.973533 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 177 ==== osd_op_reply(203 plana166726-277 [copy-from ver 319] v17'41 uv319 ondisk = 0) v6 ==== 182+0+0 (2141428691 0 0) 0x30a9500 con 0x2d342c0
2014-03-01 02:05:12.979764 7f991c37a700  1 -- 10.214.131.24:6814/5231 mark_down 0x2d342c0 -- pipe dne
2014-03-01 02:05:12.979819 7f991c37a700  1 -- 10.214.131.24:6814/5231 --> 10.214.131.21:6805/4305 -- osd_op(osd.1.4:206 plana166726-158 [copy-from ver 147] 3.ea8f28e RETRY=1 ondisk+retry+write+ignore_overlay e17) v4 -- ?+0 0x3315240 con 0x2edf580
2014-03-01 02:05:12.979847 7f991c37a700  1 -- 10.214.131.24:6814/5231 --> 10.214.131.21:6805/4305 -- osd_op(osd.1.4:207 plana166726-263 [copy-from ver 283] 3.6d7c7c42 RETRY=1 ondisk+retry+write+ignore_overlay e17) v4 -- ?+0 0x32936c0 con 0x2edf580
2014-03-01 02:05:12.979870 7f991c37a700  1 -- 10.214.131.24:6810/5231 --> 10.214.131.24:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=18}) v2 -- ?+0 0x3246000 con 0x2a3c000
2014-03-01 02:05:13.090034 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 1 ==== osd_op_reply(206 plana166726-158 [copy-from ver 147] v17'42 uv147 ondisk = 0) v6 ==== 182+0+0 (264160167 0 0) 0x2ff7000 con 0x2edf580
2014-03-01 02:05:13.165017 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 2 ==== osd_op_reply(207 plana166726-263 [copy-from ver 283] v17'43 uv283 ondisk = 0) v6 ==== 182+0+0 (3604118713 0 0) 0x348b000 con 0x2edf580
2014-03-01 02:05:13.337100 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 3 ==== osd_op_reply(212 plana166726-303 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3872697959 0 0) 0x30f6500 con 0x2edf580
2014-03-01 02:05:13.337817 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 4 ==== osd_op_reply(213 plana166726-303 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2748879401 0 0) 0x3196c80 con 0x2edf580
2014-03-01 02:05:13.492369 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 5 ==== osd_op_reply(214 plana166726-307 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3208015452 0 0) 0x3196000 con 0x2edf580
2014-03-01 02:05:13.492560 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 6 ==== osd_op_reply(215 plana166726-307 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (4198126098 0 0) 0x3196a00 con 0x2edf580
2014-03-01 02:05:13.641780 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 7 ==== osd_op_reply(208 plana166726-204 [copy-from ver 180] v17'44 uv180 ondisk = 0) v6 ==== 182+0+0 (1280795534 0 0) 0x2b51780 con 0x2edf580
2014-03-01 02:05:13.669196 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 8 ==== osd_op_reply(216 plana166726-309 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3971588944 0 0) 0x2b51280 con 0x2edf580
2014-03-01 02:05:13.669477 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 9 ==== osd_op_reply(217 plana166726-309 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2847252254 0 0) 0x2f15500 con 0x2edf580
2014-03-01 02:05:13.750575 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 10 ==== osd_op_reply(209 plana166726-156 [copy-from ver 144] v17'45 uv144 ondisk = 0) v6 ==== 182+0+0 (2231808829 0 0) 0x2f15a00 con 0x2edf580
2014-03-01 02:05:13.750695 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 11 ==== osd_op_reply(210 plana166726-294 [copy-from ver 343] v17'46 uv343 ondisk = 0) v6 ==== 182+0+0 (4018767651 0 0) 0x348bc80 con 0x2edf580
2014-03-01 02:05:13.750748 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 12 ==== osd_op_reply(211 plana166726-209 [copy-from ver 191] v17'47 uv191 ondisk = 0) v6 ==== 182+0+0 (1311298768 0 0) 0x348b280 con 0x2edf580
2014-03-01 02:05:13.849912 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 13 ==== osd_op_reply(218 plana166726-211 [copy-from ver 200] v17'48 uv200 ondisk = 0) v6 ==== 182+0+0 (3756566369 0 0) 0x2f53000 con 0x2edf580
2014-03-01 02:05:14.817017 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 14 ==== osd_op_reply(219 plana166726-311 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3005732706 0 0) 0x3196780 con 0x2edf580
2014-03-01 02:05:14.817172 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 15 ==== osd_op_reply(220 plana166726-311 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (4130073388 0 0) 0x3337780 con 0x2edf580
2014-03-01 02:05:15.023461 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 16 ==== osd_op_reply(221 plana166726-314 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (591099966 0 0) 0x2ff7c80 con 0x2edf580
2014-03-01 02:05:15.023734 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 17 ==== osd_op_reply(222 plana166726-314 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1714898032 0 0) 0x2f53c80 con 0x2edf580
2014-03-01 02:05:15.386588 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 29 ==== osd_op_reply(223 plana166726-101 [assert-version v99,copy-get max 8388608 ver 0] v0'0 uv99 ondisk = 0) v6 ==== 224+0+1052337 (790045745 0 3987574317) 0x3322780 con 0x2e47420
2014-03-01 02:05:15.397120 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 30 ==== osd_op_reply(224 plana166726-180 [assert-version v215,copy-get max 8388608 ver 0] v0'0 uv215 ondisk = 0) v6 ==== 224+0+319359 (2512392652 0 4282233575) 0x3322000 con 0x2e47420
2014-03-01 02:05:15.416745 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 31 ==== osd_op_reply(225 plana166726-183 [assert-version v211,copy-get max 8388608 ver 0] v0'0 uv211 ondisk = 0) v6 ==== 224+0+969976 (2159484132 0 3806832268) 0x3322280 con 0x2e47420
2014-03-01 02:05:16.327578 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 18 ==== osd_op_reply(226 plana166726-319 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (123390294 0 0) 0x2f53280 con 0x2edf580
2014-03-01 02:05:16.327850 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 19 ==== osd_op_reply(227 plana166726-319 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1112995096 0 0) 0x2f53780 con 0x2edf580
2014-03-01 02:05:16.357186 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 32 ==== osd_op_reply(228 plana166726-251 [assert-version v301,copy-get max 8388608 ver 0] v0'0 uv301 ondisk = 0) v6 ==== 224+0+2162014 (1374744412 0 1003339050) 0x348ba00 con 0x2e47420
2014-03-01 02:05:16.448133 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 33 ==== osd_op_reply(233 plana166726-10 [assert-version v7,copy-get max 8388608 ver 0] v0'0 uv7 ondisk = 0) v6 ==== 223+0+33022 (4139114281 0 1564262716) 0x2f15280 con 0x2e47420
2014-03-01 02:05:16.462176 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 20 ==== osd_op_reply(229 plana166726-260 [copy-from ver 273] v17'49 uv273 ondisk = 0) v6 ==== 182+0+0 (3019701455 0 0) 0x2f53500 con 0x2edf580
2014-03-01 02:05:16.489490 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 34 ==== osd_op_reply(234 plana166726-129 [assert-version v140,copy-get max 8388608 ver 0] v0'0 uv140 ondisk = 0) v6 ==== 224+0+2505586 (2669555376 0 3307337169) 0x2f15780 con 0x2e47420
2014-03-01 02:05:16.511673 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 21 ==== osd_op_reply(235 plana166726-321 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (2588220287 0 0) 0x2f53a00 con 0x2edf580
2014-03-01 02:05:16.514685 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 22 ==== osd_op_reply(236 plana166726-321 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (3746113329 0 0) 0x2f53500 con 0x2edf580
2014-03-01 02:05:16.538759 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 35 ==== osd_op_reply(237 plana166726-11 [assert-version v5,copy-get max 8388608 ver 0] v0'0 uv5 ondisk = 0) v6 ==== 223+0+1294204 (2495087282 0 439694695) 0x348bc80 con 0x2e47420
2014-03-01 02:05:16.542743 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 23 ==== osd_op_reply(238 plana166726-322 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1276231749 0 0) 0x2f53780 con 0x2edf580
2014-03-01 02:05:16.546145 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 24 ==== osd_op_reply(239 plana166726-322 [copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (152960011 0 0) 0x2f53280 con 0x2edf580
2014-03-01 02:05:16.653763 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.4 10.214.131.21:6810/4306 36 ==== osd_op_reply(240 plana166726-208 [assert-version v256,copy-get max 8388608 ver 0] v0'0 uv256 ondisk = 0) v6 ==== 224+0+721983 (420149336 0 2035393159) 0x3196000 con 0x2e47420
2014-03-01 02:05:17.067070 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 25 ==== osd_op_reply(230 plana166726-279 [copy-from ver 325] v17'50 uv325 ondisk = 0) v6 ==== 182+0+0 (2484767664 0 0) 0x2f53c80 con 0x2edf580
2014-03-01 02:05:17.202373 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 26 ==== osd_op_reply(231 plana166726-261 [copy-from ver 277] v17'51 uv277 ondisk = 0) v6 ==== 182+0+0 (59131040 0 0) 0x2f53000 con 0x2edf580
2014-03-01 02:05:17.202421 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 27 ==== osd_op_reply(232 plana166726-311 [copy-from ver 382] v17'52 uv382 ondisk = 0) v6 ==== 182+0+0 (524502439 0 0) 0x2b51280 con 0x2edf580
2014-03-01 02:05:17.284512 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 28 ==== osd_op_reply(241 plana166726-242 [copy-from ver 253] v17'53 uv253 ondisk = 0) v6 ==== 182+0+0 (440653639 0 0) 0x2b51780 con 0x2edf580
2014-03-01 02:05:48.023017 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.3 10.214.131.21:6805/4305 29 ==== osd_op_reply(247 plana166726-457 [list-snaps] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1447496095 0 0) 0x37e7c80 con

Actions

Copy link

#12

Updated by Loïc Dachary about 10 years ago

Logs lines for plana166726-98

2014-03-01 02:06:07.795654 7f9919b75700  1 -- 10.214.131.24:6814/5231 --> 10.214.131.24:6810/5231 -- osd_op(osd.1.4:325 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 4.d44ec913 ack+read e28) v4 -- ?+0 0x3a59900 con 0x68d9dc0
2014-03-01 02:06:07.799048 7f9920b83700  1 -- 10.214.131.24:6810/5231 <== osd.1 10.214.131.24:6814/5231 1 ==== osd_op(osd.1.4:325 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 4.d44ec913 ack+read e28) v4 ==== 203+0+29 (2115722021 0 750716084) 0x2f82480 con 0x30cd000
2014-03-01 02:06:07.799151 7f9920b83700 20 osd.1 28 _dispatch 0x2f82480 osd_op(osd.1.4:325 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 4.d44ec913 ack+read e28) v4
2014-03-01 02:06:07.799284 7f9920b83700 15 osd.1 28 enqueue_op 0x30434b0 prio 63 cost 29 latency 0.000345 osd_op(osd.1.4:325 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 4.d44ec913 ack+read e28) v4
2014-03-01 02:06:07.820210 7f991a376700 10 osd.1 28 dequeue_op 0x30434b0 prio 63 cost 29 latency 0.021271 osd_op(osd.1.4:325 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 4.d44ec913 ack+read e28) v4 pg pg[4.3( v 28'735 lc 15'24 (0'0,28'735] local-les=28 n=63 ec=8 les/c 28/21 27/27/27) [1,0] r=0 lpr=27 pi=8-26/2 luod=28'721 crt=26'720 mlcod 15'24 active+recovering m=61]
2014-03-01 02:06:07.820677 7f991a376700 10 osd.1 pg_epoch: 28 pg[4.3( v 28'735 lc 15'24 (0'0,28'735] local-les=28 n=63 ec=8 les/c 28/21 27/27/27) [1,0] r=0 lpr=27 pi=8-26/2 luod=28'721 crt=26'720 mlcod 15'24 active+recovering m=61] do_op osd_op(osd.1.4:325 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 4.d44ec913 ack+read e28) v4 may_read -> read-ordered flags ack+read
2014-03-01 02:06:07.821359 7f991a376700  1 -- 10.214.131.24:6810/5231 --> 10.214.131.24:6814/5231 -- osd_op_reply(325 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory) redirect: { object , locator{@3} }) v6 -- ?+0 0x2b51780 con 0x30cd000
2014-03-01 02:06:07.821707 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.1 10.214.131.24:6810/5231 1 ==== osd_op_reply(325 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] v0'0 uv0 ack = -2 (No such file or directory) redirect: { object , locator{@3} }) v6 ==== 223+0+0 (2657899360 0 0) 0x3092000 con 0x68d9dc0
2014-03-01 02:06:07.821760 7f991c37a700  1 -- 10.214.131.24:6814/5231 --> 10.214.131.21:6805/4305 -- osd_op(osd.1.4:327 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 3.d44ec913 RETRY=1 ack+retry+read e28) v4 -- ?+0 0x2ae9b40 con 0x2edf580
2014-03-01 02:06:08.687687 7f991fb81700  1 -- 10.214.131.24:6814/5231 --> 10.214.131.21:6800/4303 -- osd_op(osd.1.4:327 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] 3.d44ec913 RETRY=2 ack+retry+read e29) v4 -- ?+0 0x3c77900 con 0x379a580
2014-03-01 02:06:10.778103 7f991c37a700  1 -- 10.214.131.24:6814/5231 <== osd.5 10.214.131.21:6800/4303 21 ==== osd_op_reply(327 plana166726-98 [assert-version v120,copy-get max 8388608 ver 0] v0'0 uv120 ondisk = 0) v6 ==== 223+0+3352119 (3017310025 0 3039343544) 0x3092c80 con 0x379a580

Actions

Copy link

#13