2012-05-21 07:27:28.152697 7ff1a3a00760 10 -- :/0 ready :/0
2012-05-21 07:27:28.152762 7ff1a3a00760  1 -- :/0 messenger.start
2012-05-21 07:27:28.152904 7ff19fde7700 10 -- :/1016781 reaper_entry start
2012-05-21 07:27:28.152924 7ff19fde7700 10 -- :/1016781 reaper
2012-05-21 07:27:28.152927 7ff19fde7700 10 -- :/1016781 reaper done
2012-05-21 07:27:28.153320 7ff1a3a00760 10 -- :/1016781 connect_rank to 10.100.1.1:6789/0, creating pipe and registering
2012-05-21 07:27:28.153365 7ff1a3a00760 10 -- :/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=-1 pgs=0 cs=0 l=0).register_pipe
2012-05-21 07:27:28.153370 7ff1a39f8700 10 -- :/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2012-05-21 07:27:28.153399 7ff1a39f8700 10 -- :/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=-1 pgs=0 cs=0 l=0).connect 0
2012-05-21 07:27:28.153403 7ff1a3a00760  1 -- :/1016781 --> 10.100.1.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x7ff198001990 con 0x7ff198001520
2012-05-21 07:27:28.153411 7ff1a3a00760 20 -- :/1016781 submit_message auth(proto 0 30 bytes epoch 0) v1 remote 10.100.1.1:6789/0
2012-05-21 07:27:28.153419 7ff1a39f8700 10 -- :/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=0 cs=0 l=0).connecting to 10.100.1.1:6789/0
2012-05-21 07:27:28.153980 7ff1a39f8700 20 -- :/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=0 cs=0 l=0).connect read peer addr 10.100.1.1:6789/0 on socket 3
2012-05-21 07:27:28.153992 7ff1a39f8700 20 -- :/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=0 cs=0 l=0).connect peer addr for me is 10.100.1.16:44810/0
2012-05-21 07:27:28.153999 7ff1a39f8700  1 -- 10.100.1.16:0/1016781 learned my addr 10.100.1.16:0/1016781
2012-05-21 07:27:28.154015 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=0 cs=0 l=0).connect sent my addr 10.100.1.16:0/1016781
2012-05-21 07:27:28.154030 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=0 cs=0 l=0).connect sending gseq=1 cseq=0 proto=15
2012-05-21 07:27:28.154039 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
2012-05-21 07:27:28.154261 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 253 proto 15 flags 1
2012-05-21 07:27:28.154272 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).connect success 1, lossy = 1, features 32767
2012-05-21 07:27:28.154283 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> :/0 pipe(0x28092b0 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
2012-05-21 07:27:28.154298 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).connect starting reader
2012-05-21 07:27:28.154316 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x28092b0 dequeued 0
2012-05-21 07:27:28.154344 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.154354 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer encoding 1 0x7ff198001990 auth(proto 0 30 bytes epoch 0) v1
2012-05-21 07:27:28.154379 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sending 1 0x7ff198001990
2012-05-21 07:27:28.154387 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_message 0x7ff198001990
2012-05-21 07:27:28.154368 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.154404 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.154410 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.155081 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got ACK
2012-05-21 07:27:28.155108 7ff19e5e4700 15 reader got ack seq 1
2012-05-21 07:27:28.155110 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.155116 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.155122 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=4 src mon.0 front=751 data=0 off 0
2012-05-21 07:27:28.155132 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 751 from dispatch throttler 0/104857600
2012-05-21 07:27:28.155150 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 751
2012-05-21 07:27:28.155157 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.155167 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 751 + 0 + 0 byte message
2012-05-21 07:27:28.155203 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 1 0x7ff198003ab0 mon_map v1
2012-05-21 07:27:28.155220 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.155243 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 1
2012-05-21 07:27:28.155245 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.155275 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.155282 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.155278 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x7ff198003ab0
2012-05-21 07:27:28.155288 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=18 src mon.0 front=33 data=0 off 0
2012-05-21 07:27:28.155298 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 33 from dispatch throttler 751/104857600
2012-05-21 07:27:28.155297 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 1 ==== mon_map v1 ==== 751+0+0 (4218198165 0 0) 0x7ff198003ab0 con 0x7ff198001520
2012-05-21 07:27:28.155300 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.155308 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.155309 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 33
2012-05-21 07:27:28.155316 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.155322 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 33 + 0 + 0 byte message
2012-05-21 07:27:28.155361 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 751 to dispatch throttler 784/104857600
2012-05-21 07:27:28.155369 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x7ff198003ab0
2012-05-21 07:27:28.155351 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 2 0x7ff198005000 auth_reply(proto 2 0 Success) v1
2012-05-21 07:27:28.155393 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.155407 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x7ff198005000
2012-05-21 07:27:28.155406 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.155419 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.155416 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3033707448 0 0) 0x7ff198005000 con 0x7ff198001520
2012-05-21 07:27:28.155432 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 2
2012-05-21 07:27:28.155438 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.155443 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.155622 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7ff198005b20 con 0x7ff198001520
2012-05-21 07:27:28.155630 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 submit_message auth(proto 2 32 bytes epoch 0) v1 remote 10.100.1.1:6789/0
2012-05-21 07:27:28.155640 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 33 to dispatch throttler 33/104857600
2012-05-21 07:27:28.155644 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x7ff198005000
2012-05-21 07:27:28.155643 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.155654 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer encoding 2 0x7ff198005b20 auth(proto 2 32 bytes epoch 0) v1
2012-05-21 07:27:28.155665 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sending 2 0x7ff198005b20
2012-05-21 07:27:28.155671 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_message 0x7ff198005b20
2012-05-21 07:27:28.155686 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.155691 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.156242 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got ACK
2012-05-21 07:27:28.156254 7ff19e5e4700 15 reader got ack seq 2
2012-05-21 07:27:28.156255 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.156261 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.156266 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=18 src mon.0 front=206 data=0 off 0
2012-05-21 07:27:28.156272 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 206 from dispatch throttler 0/104857600
2012-05-21 07:27:28.156282 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 206
2012-05-21 07:27:28.156288 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.156293 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 206 + 0 + 0 byte message
2012-05-21 07:27:28.156310 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 3 0x280c450 auth_reply(proto 2 0 Success) v1
2012-05-21 07:27:28.156317 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.156332 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.156337 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.156356 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 3
2012-05-21 07:27:28.156360 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x280c450
2012-05-21 07:27:28.156378 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.156375 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (635237937 0 0) 0x280c450 con 0x7ff198001520
2012-05-21 07:27:28.156385 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.156467 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7ff198006ee0 con 0x7ff198001520
2012-05-21 07:27:28.156474 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 submit_message auth(proto 2 165 bytes epoch 0) v1 remote 10.100.1.1:6789/0
2012-05-21 07:27:28.156483 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 206 to dispatch throttler 206/104857600
2012-05-21 07:27:28.156488 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x280c450
2012-05-21 07:27:28.156486 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.156494 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer encoding 3 0x7ff198006ee0 auth(proto 2 165 bytes epoch 0) v1
2012-05-21 07:27:28.156509 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sending 3 0x7ff198006ee0
2012-05-21 07:27:28.156515 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_message 0x7ff198006ee0
2012-05-21 07:27:28.156536 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.156541 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.157201 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got ACK
2012-05-21 07:27:28.157211 7ff19e5e4700 15 reader got ack seq 3
2012-05-21 07:27:28.157212 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.157217 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.157223 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=18 src mon.0 front=409 data=0 off 0
2012-05-21 07:27:28.157229 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 409 from dispatch throttler 0/104857600
2012-05-21 07:27:28.157236 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 409
2012-05-21 07:27:28.157242 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.157246 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 409 + 0 + 0 byte message
2012-05-21 07:27:28.157260 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 4 0x280e910 auth_reply(proto 2 0 Success) v1
2012-05-21 07:27:28.157275 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.157291 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.157295 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.157315 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 4
2012-05-21 07:27:28.157321 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x280e910
2012-05-21 07:27:28.157327 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.157327 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 409+0+0 (1595411229 0 0) 0x280e910 con 0x7ff198001520
2012-05-21 07:27:28.157332 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.157376 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7ff198003010 con 0x7ff198001520
2012-05-21 07:27:28.157388 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 submit_message mon_subscribe({monmap=0+}) v2 remote 10.100.1.1:6789/0
2012-05-21 07:27:28.157405 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.157407 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 409 to dispatch throttler 409/104857600
2012-05-21 07:27:28.157412 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x280e910
2012-05-21 07:27:28.157413 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer encoding 4 0x7ff198003010 mon_subscribe({monmap=0+}) v2
2012-05-21 07:27:28.157424 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sending 4 0x7ff198003010
2012-05-21 07:27:28.157429 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_message 0x7ff198003010
2012-05-21 07:27:28.157448 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.157455 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.157501 7ff1a3a00760 10 client.11820.objecter maybe_request_map subscribing (onetime) to next osd map
2012-05-21 07:27:28.157527 7ff1a3a00760  1 -- 10.100.1.16:0/1016781 --> 10.100.1.1:6789/0 -- mon_subscribe({monmap=9+,osdmap=0}) v2 -- ?+0 0x7ff198008460 con 0x7ff198001520
2012-05-21 07:27:28.157538 7ff1a3a00760 20 -- 10.100.1.16:0/1016781 submit_message mon_subscribe({monmap=9+,osdmap=0}) v2 remote 10.100.1.1:6789/0
2012-05-21 07:27:28.157549 7ff1a3a00760  1 -- 10.100.1.16:0/1016781 --> 10.100.1.1:6789/0 -- mon_subscribe({monmap=9+,osdmap=0}) v2 -- ?+0 0x7ff198008900 con 0x7ff198001520
2012-05-21 07:27:28.157551 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.157560 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer encoding 5 0x7ff198008460 mon_subscribe({monmap=9+,osdmap=0}) v2
2012-05-21 07:27:28.157563 7ff1a3a00760 20 -- 10.100.1.16:0/1016781 submit_message mon_subscribe({monmap=9+,osdmap=0}) v2 remote 10.100.1.1:6789/0
2012-05-21 07:27:28.157569 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sending 5 0x7ff198008460
2012-05-21 07:27:28.157575 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_message 0x7ff198008460
2012-05-21 07:27:28.157586 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.157592 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer encoding 6 0x7ff198008900 mon_subscribe({monmap=9+,osdmap=0}) v2
2012-05-21 07:27:28.157607 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sending 6 0x7ff198008900
2012-05-21 07:27:28.157612 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_message 0x7ff198008900
2012-05-21 07:27:28.157623 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.157628 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.158025 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got ACK
2012-05-21 07:27:28.158040 7ff19e5e4700 15 reader got ack seq 4
2012-05-21 07:27:28.158041 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.158047 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got ACK
2012-05-21 07:27:28.158053 7ff19e5e4700 15 reader got ack seq 5
2012-05-21 07:27:28.158054 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.158062 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.158068 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=4 src mon.0 front=751 data=0 off 0
2012-05-21 07:27:28.158074 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 751 from dispatch throttler 0/104857600
2012-05-21 07:27:28.158084 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 751
2012-05-21 07:27:28.158091 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.158095 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 751 + 0 + 0 byte message
2012-05-21 07:27:28.158113 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 5 0x2811920 mon_map v1
2012-05-21 07:27:28.158119 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.158135 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.158138 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x2811920
2012-05-21 07:27:28.158141 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got ACK
2012-05-21 07:27:28.158152 7ff19e5e4700 15 reader got ack seq 6
2012-05-21 07:27:28.158153 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.158156 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 5 ==== mon_map v1 ==== 751+0+0 (4218198165 0 0) 0x2811920 con 0x7ff198001520
2012-05-21 07:27:28.158159 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.158159 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.158168 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=16 src mon.0 front=20 data=0 off 0
2012-05-21 07:27:28.158175 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 5
2012-05-21 07:27:28.158174 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 20 from dispatch throttler 751/104857600
2012-05-21 07:27:28.158181 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 751 to dispatch throttler 771/104857600
2012-05-21 07:27:28.158187 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x2811920
2012-05-21 07:27:28.158192 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.158196 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 20
2012-05-21 07:27:28.158198 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.158204 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.158212 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 20 + 0 + 0 byte message
2012-05-21 07:27:28.158228 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 6 0x7ff198009150 mon_subscribe_ack(300s) v1
2012-05-21 07:27:28.158239 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.158250 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x7ff198009150
2012-05-21 07:27:28.158252 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.158258 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (496062740 0 0) 0x7ff198009150 con 0x7ff198001520
2012-05-21 07:27:28.158260 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.158267 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 6
2012-05-21 07:27:28.158270 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 20 to dispatch throttler 20/104857600
2012-05-21 07:27:28.158272 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.158275 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x7ff198009150
2012-05-21 07:27:28.158278 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.158288 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.158311 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=41 src mon.0 front=11213 data=0 off 0
2012-05-21 07:27:28.158327 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 11213 from dispatch throttler 0/104857600
2012-05-21 07:27:28.158597 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 11213
2012-05-21 07:27:28.158604 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.158609 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 11213 + 0 + 0 byte message
2012-05-21 07:27:28.158638 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 7 0x7ff19800ccc0 osd_map(6883..6883 src has 6383..6883) v3
2012-05-21 07:27:28.158651 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.158661 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.158662 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x7ff19800ccc0
2012-05-21 07:27:28.158672 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.158670 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 7 ==== osd_map(6883..6883 src has 6383..6883) v3 ==== 11213+0+0 (2197969998 0 0) 0x7ff19800ccc0 con 0x7ff198001520
2012-05-21 07:27:28.158671 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.158678 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 7
2012-05-21 07:27:28.158681 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=16 src mon.0 front=20 data=0 off 0
2012-05-21 07:27:28.158684 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.158687 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 20 from dispatch throttler 11213/104857600
2012-05-21 07:27:28.158689 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.158696 7ff1a05e8700  3 client.11820.objecter handle_osd_map got epochs [6883,6883] > 0
2012-05-21 07:27:28.158695 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 20
2012-05-21 07:27:28.158702 7ff1a05e8700  3 client.11820.objecter handle_osd_map decoding full epoch 6883
2012-05-21 07:27:28.158705 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.158710 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 20 + 0 + 0 byte message
2012-05-21 07:27:28.158718 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 8 0x7ff19800dff0 mon_subscribe_ack(300s) v1
2012-05-21 07:27:28.158726 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.158744 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.158746 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.158753 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 8
2012-05-21 07:27:28.158753 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.158759 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.158760 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=41 src mon.0 front=11213 data=0 off 0
2012-05-21 07:27:28.158764 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.158766 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 11213 from dispatch throttler 11233/104857600
2012-05-21 07:27:28.158829 7ff1a05e8700 20 client.11820.objecter dump_active .. 0 homeless
2012-05-21 07:27:28.158852 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 11213 to dispatch throttler 22446/104857600
2012-05-21 07:27:28.158857 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x7ff19800ccc0
2012-05-21 07:27:28.158861 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x7ff19800dff0
2012-05-21 07:27:28.158864 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (496062740 0 0) 0x7ff19800dff0 con 0x7ff198001520
2012-05-21 07:27:28.158872 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 20 to dispatch throttler 11233/104857600
2012-05-21 07:27:28.158877 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x7ff19800dff0
2012-05-21 07:27:28.158965 7ff1a3a00760 20 librbd: open_image: ictx =  0x7ff19800a450 name =  'kvm1148' snap_name = ''
2012-05-21 07:27:28.158978 7ff1a3a00760 20 librbd: ictx_refresh 0x7ff19800a450
2012-05-21 07:27:28.159025 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 11213
2012-05-21 07:27:28.159037 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.159041 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 11213 + 0 + 0 byte message
2012-05-21 07:27:28.159038 7ff1a3a00760 10 client.11820.objecter recalc_op_target tid 1 pgid 2.7d848fd9 acting [2,15]
2012-05-21 07:27:28.159056 7ff1a3a00760 10 -- 10.100.1.16:0/1016781 connect_rank to 10.100.1.11:6802/30752, creating pipe and registering
2012-05-21 07:27:28.159063 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 9 0x2816a20 osd_map(6883..6883 src has 6383..6883) v3
2012-05-21 07:27:28.159074 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.159093 7ff1a3a00760 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=-1 pgs=0 cs=0 l=0).register_pipe
2012-05-21 07:27:28.159105 7ff1a3a00760 20 client.11820.objecter  note: not requesting commit
2012-05-21 07:27:28.159092 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.159106 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x2816a20
2012-05-21 07:27:28.159110 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:28.159116 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=16 src mon.0 front=20 data=0 off 0
2012-05-21 07:27:28.159114 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 9 ==== osd_map(6883..6883 src has 6383..6883) v3 ==== 11213+0+0 (2197969998 0 0) 0x2816a20 con 0x7ff198001520
2012-05-21 07:27:28.159113 7ff1a3a00760 10 client.11820.objecter op_submit oid kvm1148.rbd @2 [read 0~4096] tid 1 osd.2
2012-05-21 07:27:28.159126 7ff1a3a00760 15 client.11820.objecter send_op 1 to osd.2
2012-05-21 07:27:28.159121 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 20 from dispatch throttler 11213/104857600
2012-05-21 07:27:28.159125 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.159133 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 20
2012-05-21 07:27:28.159135 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 9
2012-05-21 07:27:28.159139 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:28.159135 7ff1a3a00760  1 -- 10.100.1.16:0/1016781 --> 10.100.1.11:6802/30752 -- osd_op(client.11820.0:1 kvm1148.rbd [read 0~4096] 2.7d848fd9) v4 -- ?+0 0x7ff19800bfd0 con 0x7ff19800b5c0
2012-05-21 07:27:28.159144 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 20 + 0 + 0 byte message
2012-05-21 07:27:28.159144 7ff1a3a00760 20 -- 10.100.1.16:0/1016781 submit_message osd_op(client.11820.0:1 kvm1148.rbd [read 0~4096] 2.7d848fd9) v4 remote 10.100.1.11:6802/30752
2012-05-21 07:27:28.159146 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.159151 7ff1a3a00760  5 client.11820.objecter 1 unacked, 0 uncommitted
2012-05-21 07:27:28.159151 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.159162 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2012-05-21 07:27:28.159167 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 10 0x28178c0 mon_subscribe_ack(300s) v1
2012-05-21 07:27:28.159174 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=-1 pgs=0 cs=0 l=0).connect 0
2012-05-21 07:27:28.159169 7ff1a05e8700  3 client.11820.objecter handle_osd_map ignoring epochs [6883,6883] <= 6883
2012-05-21 07:27:28.159177 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:28.159182 7ff1a05e8700 20 client.11820.objecter dump_active .. 0 homeless
2012-05-21 07:27:28.159184 7ff1a05e8700 20 client.11820.objecter 1	2.7d848fd9	osd.2	kvm1148.rbd	[read 0~4096]
2012-05-21 07:27:28.159184 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.159188 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).connecting to 10.100.1.11:6802/30752
2012-05-21 07:27:28.159200 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.159205 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 11213 to dispatch throttler 11233/104857600
2012-05-21 07:27:28.159210 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 10
2012-05-21 07:27:28.159215 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x2816a20
2012-05-21 07:27:28.159218 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.159226 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.159238 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x28178c0
2012-05-21 07:27:28.159242 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 10 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (496062740 0 0) 0x28178c0 con 0x7ff198001520
2012-05-21 07:27:28.159251 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 20 to dispatch throttler 20/104857600
2012-05-21 07:27:28.159255 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x28178c0
2012-05-21 07:27:28.159808 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).connect read peer addr 10.100.1.11:6802/30752 on socket 5
2012-05-21 07:27:28.159820 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).connect peer addr for me is 10.100.1.16:54011/0
2012-05-21 07:27:28.159836 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).connect sent my addr 10.100.1.16:0/1016781
2012-05-21 07:27:28.159884 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).connect.authorizer_len=174 protocol=2
2012-05-21 07:27:28.159891 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).connect sending gseq=2 cseq=0 proto=24
2012-05-21 07:27:28.159900 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
2012-05-21 07:27:28.160274 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 1566 proto 24 flags 1
2012-05-21 07:27:28.160289 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=0 cs=0 l=0).reply.authorizer_len=36
2012-05-21 07:27:28.160318 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).connect success 1, lossy = 1, features 32767
2012-05-21 07:27:28.160328 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> :/0 pipe(0x28092b0 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
2012-05-21 07:27:28.160348 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).connect starting reader
2012-05-21 07:27:28.160358 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x28092b0 dequeued 0
2012-05-21 07:27:28.160400 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.160413 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer encoding 1 0x7ff19800bfd0 osd_op(client.11820.0:1 kvm1148.rbd [read 0~4096] 2.7d848fd9) v4
2012-05-21 07:27:28.160417 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...
2012-05-21 07:27:28.160434 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sending 1 0x7ff19800bfd0
2012-05-21 07:27:28.160443 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).write_message 0x7ff19800bfd0
2012-05-21 07:27:28.160455 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:28.160465 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sleeping
2012-05-21 07:27:28.365080 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader got ACK
2012-05-21 07:27:28.365108 7ff19d3e0700 15 reader got ack seq 1
2012-05-21 07:27:28.365110 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...
2012-05-21 07:27:31.153372 7ff19f5e6700 20 -- 10.100.1.16:0/1016781 send_keepalive con 0x7ff198001520, have pipe.
2012-05-21 07:27:31.153422 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:31.153443 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_keepalive
2012-05-21 07:27:31.153471 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:31.153477 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:33.157589 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:27:38.157700 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:27:41.153498 7ff19f5e6700 20 -- 10.100.1.16:0/1016781 send_keepalive con 0x7ff198001520, have pipe.
2012-05-21 07:27:41.153531 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:41.153549 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_keepalive
2012-05-21 07:27:41.153566 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:41.153571 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:43.157800 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:27:43.157811 7ff19e4e3700  2 client.11820.objecter  tid 1 on osd.2 is laggy
2012-05-21 07:27:43.157819 7ff19e4e3700 10 client.11820.objecter maybe_request_map subscribing (onetime) to next osd map
2012-05-21 07:27:43.157828 7ff19e4e3700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.1:6789/0 -- mon_subscribe({monmap=9+,osdmap=6884}) v2 -- ?+0 0x2814ae0 con 0x7ff198001520
2012-05-21 07:27:43.157855 7ff19e4e3700 20 -- 10.100.1.16:0/1016781 submit_message mon_subscribe({monmap=9+,osdmap=6884}) v2 remote 10.100.1.1:6789/0
2012-05-21 07:27:43.157883 7ff19e4e3700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.11:6802/30752 -- ping v1 -- ?+0 0x2814fc0 con 0x7ff19800b5c0
2012-05-21 07:27:43.157889 7ff19e4e3700 20 -- 10.100.1.16:0/1016781 submit_message ping v1 remote 10.100.1.11:6802/30752
2012-05-21 07:27:43.157906 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:43.157900 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:43.157923 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer encoding 2 0x2814fc0 ping v1
2012-05-21 07:27:43.157927 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer encoding 7 0x2814ae0 mon_subscribe({monmap=9+,osdmap=6884}) v2
2012-05-21 07:27:43.157939 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sending 2 0x2814fc0
2012-05-21 07:27:43.157945 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).write_message 0x2814fc0
2012-05-21 07:27:43.157948 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sending 7 0x2814ae0
2012-05-21 07:27:43.157956 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_message 0x2814ae0
2012-05-21 07:27:43.157963 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:43.158371 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sleeping
2012-05-21 07:27:43.158372 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:43.158380 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:43.158917 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got ACK
2012-05-21 07:27:43.158927 7ff19e5e4700 15 reader got ack seq 7
2012-05-21 07:27:43.158928 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:43.158934 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got MSG
2012-05-21 07:27:43.158941 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got envelope type=16 src mon.0 front=20 data=0 off 0
2012-05-21 07:27:43.158947 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader wants 20 from dispatch throttler 0/104857600
2012-05-21 07:27:43.158956 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got front 20
2012-05-21 07:27:43.158963 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).aborted = 0
2012-05-21 07:27:43.158967 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got 20 + 0 + 0 byte message
2012-05-21 07:27:43.158978 7ff19e5e4700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader got message 11 0x2818f80 mon_subscribe_ack(300s) v1
2012-05-21 07:27:43.158986 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).queue_received queuing pipe
2012-05-21 07:27:43.159008 7ff19e5e4700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).reader reading tag...
2012-05-21 07:27:43.159011 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 dispatch_entry pipe 0x7ff1980012b0 dequeued 0x2818f80
2012-05-21 07:27:43.159029 7ff1a05e8700  1 -- 10.100.1.16:0/1016781 <== mon.0 10.100.1.1:6789/0 11 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (496062740 0 0) 0x2818f80 con 0x7ff198001520
2012-05-21 07:27:43.159033 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:43.159043 7ff1a05e8700 10 -- 10.100.1.16:0/1016781 dispatch_throttle_release 20 to dispatch throttler 20/104857600
2012-05-21 07:27:43.159050 7ff1a05e8700 20 -- 10.100.1.16:0/1016781 done calling dispatch on 0x2818f80
2012-05-21 07:27:43.159050 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_ack 11
2012-05-21 07:27:43.159064 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:43.159069 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:43.369110 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader got ACK
2012-05-21 07:27:43.369154 7ff19d3e0700 15 reader got ack seq 2
2012-05-21 07:27:43.369156 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...
2012-05-21 07:27:48.157987 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:27:48.158003 7ff19e4e3700  2 client.11820.objecter  tid 1 on osd.2 is laggy
2012-05-21 07:27:48.158007 7ff19e4e3700 10 client.11820.objecter maybe_request_map subscribing (onetime) to next osd map
2012-05-21 07:27:48.158012 7ff19e4e3700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.11:6802/30752 -- ping v1 -- ?+0 0x28199a0 con 0x7ff19800b5c0
2012-05-21 07:27:48.158021 7ff19e4e3700 20 -- 10.100.1.16:0/1016781 submit_message ping v1 remote 10.100.1.11:6802/30752
2012-05-21 07:27:48.158044 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:48.158068 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer encoding 3 0x28199a0 ping v1
2012-05-21 07:27:48.158074 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sending 3 0x28199a0
2012-05-21 07:27:48.158085 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).write_message 0x28199a0
2012-05-21 07:27:48.158103 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:48.158112 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sleeping
2012-05-21 07:27:48.369155 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader got ACK
2012-05-21 07:27:48.369182 7ff19d3e0700 15 reader got ack seq 3
2012-05-21 07:27:48.369184 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...
2012-05-21 07:27:51.153610 7ff19f5e6700 20 -- 10.100.1.16:0/1016781 send_keepalive con 0x7ff198001520, have pipe.
2012-05-21 07:27:51.153650 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:51.153668 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_keepalive
2012-05-21 07:27:51.153691 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:51.153700 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:27:53.158120 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:27:53.158136 7ff19e4e3700  2 client.11820.objecter  tid 1 on osd.2 is laggy
2012-05-21 07:27:53.158140 7ff19e4e3700 10 client.11820.objecter maybe_request_map subscribing (onetime) to next osd map
2012-05-21 07:27:53.158145 7ff19e4e3700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.11:6802/30752 -- ping v1 -- ?+0 0x281a2b0 con 0x7ff19800b5c0
2012-05-21 07:27:53.158153 7ff19e4e3700 20 -- 10.100.1.16:0/1016781 submit_message ping v1 remote 10.100.1.11:6802/30752
2012-05-21 07:27:53.158171 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:53.158197 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer encoding 4 0x281a2b0 ping v1
2012-05-21 07:27:53.158208 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sending 4 0x281a2b0
2012-05-21 07:27:53.158213 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).write_message 0x281a2b0
2012-05-21 07:27:53.158236 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:53.158245 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sleeping
2012-05-21 07:27:53.365137 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader got ACK
2012-05-21 07:27:53.365161 7ff19d3e0700 15 reader got ack seq 4
2012-05-21 07:27:53.365162 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...
2012-05-21 07:27:58.158249 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:27:58.158265 7ff19e4e3700  2 client.11820.objecter  tid 1 on osd.2 is laggy
2012-05-21 07:27:58.158268 7ff19e4e3700 10 client.11820.objecter maybe_request_map subscribing (onetime) to next osd map
2012-05-21 07:27:58.158273 7ff19e4e3700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.11:6802/30752 -- ping v1 -- ?+0 0x7ff190003d80 con 0x7ff19800b5c0
2012-05-21 07:27:58.158281 7ff19e4e3700 20 -- 10.100.1.16:0/1016781 submit_message ping v1 remote 10.100.1.11:6802/30752
2012-05-21 07:27:58.158303 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:58.158325 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer encoding 5 0x7ff190003d80 ping v1
2012-05-21 07:27:58.158332 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sending 5 0x7ff190003d80
2012-05-21 07:27:58.158336 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).write_message 0x7ff190003d80
2012-05-21 07:27:58.158358 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:27:58.158368 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sleeping
2012-05-21 07:27:58.365138 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader got ACK
2012-05-21 07:27:58.365163 7ff19d3e0700 15 reader got ack seq 5
2012-05-21 07:27:58.365164 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...
2012-05-21 07:28:01.153723 7ff19f5e6700 20 -- 10.100.1.16:0/1016781 send_keepalive con 0x7ff198001520, have pipe.
2012-05-21 07:28:01.153762 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:01.153779 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_keepalive
2012-05-21 07:28:01.153803 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:01.153812 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:28:03.158374 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:28:03.158390 7ff19e4e3700  2 client.11820.objecter  tid 1 on osd.2 is laggy
2012-05-21 07:28:03.158393 7ff19e4e3700 10 client.11820.objecter maybe_request_map subscribing (onetime) to next osd map
2012-05-21 07:28:03.158398 7ff19e4e3700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.11:6802/30752 -- ping v1 -- ?+0 0x7ff190004ab0 con 0x7ff19800b5c0
2012-05-21 07:28:03.158406 7ff19e4e3700 20 -- 10.100.1.16:0/1016781 submit_message ping v1 remote 10.100.1.11:6802/30752
2012-05-21 07:28:03.158422 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:03.158448 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer encoding 6 0x7ff190004ab0 ping v1
2012-05-21 07:28:03.158458 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sending 6 0x7ff190004ab0
2012-05-21 07:28:03.158463 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).write_message 0x7ff190004ab0
2012-05-21 07:28:03.158539 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:03.158550 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sleeping
2012-05-21 07:28:03.365087 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader got ACK
2012-05-21 07:28:03.365111 7ff19d3e0700 15 reader got ack seq 6
2012-05-21 07:28:03.365113 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...
2012-05-21 07:28:08.158500 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:28:08.158515 7ff19e4e3700  2 client.11820.objecter  tid 1 on osd.2 is laggy
2012-05-21 07:28:08.158519 7ff19e4e3700 10 client.11820.objecter maybe_request_map subscribing (onetime) to next osd map
2012-05-21 07:28:08.158523 7ff19e4e3700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.11:6802/30752 -- ping v1 -- ?+0 0x7ff190005760 con 0x7ff19800b5c0
2012-05-21 07:28:08.158531 7ff19e4e3700 20 -- 10.100.1.16:0/1016781 submit_message ping v1 remote 10.100.1.11:6802/30752
2012-05-21 07:28:08.158546 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:08.158570 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer encoding 7 0x7ff190005760 ping v1
2012-05-21 07:28:08.158575 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sending 7 0x7ff190005760
2012-05-21 07:28:08.158585 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).write_message 0x7ff190005760
2012-05-21 07:28:08.158609 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:08.158618 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sleeping
2012-05-21 07:28:08.365090 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader got ACK
2012-05-21 07:28:08.365114 7ff19d3e0700 15 reader got ack seq 7
2012-05-21 07:28:08.365115 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...
2012-05-21 07:28:11.153839 7ff19f5e6700 20 -- 10.100.1.16:0/1016781 send_keepalive con 0x7ff198001520, have pipe.
2012-05-21 07:28:11.153879 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:11.153894 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).write_keepalive
2012-05-21 07:28:11.153918 7ff1a39f8700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:11.153928 7ff1a39f8700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.1:6789/0 pipe(0x7ff1980012b0 sd=3 pgs=253 cs=1 l=1).writer sleeping
2012-05-21 07:28:13.158629 7ff19e4e3700 10 client.11820.objecter tick
2012-05-21 07:28:13.158644 7ff19e4e3700  2 client.11820.objecter  tid 1 on osd.2 is laggy
2012-05-21 07:28:13.158648 7ff19e4e3700 10 client.11820.objecter maybe_request_map subscribing (onetime) to next osd map
2012-05-21 07:28:13.158652 7ff19e4e3700  1 -- 10.100.1.16:0/1016781 --> 10.100.1.11:6802/30752 -- ping v1 -- ?+0 0x7ff190006410 con 0x7ff19800b5c0
2012-05-21 07:28:13.158660 7ff19e4e3700 20 -- 10.100.1.16:0/1016781 submit_message ping v1 remote 10.100.1.11:6802/30752
2012-05-21 07:28:13.158677 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:13.158704 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer encoding 8 0x7ff190006410 ping v1
2012-05-21 07:28:13.158710 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sending 8 0x7ff190006410
2012-05-21 07:28:13.158751 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).write_message 0x7ff190006410
2012-05-21 07:28:13.158780 7ff19d4e1700 10 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer: state = 2 policy.server=0
2012-05-21 07:28:13.158806 7ff19d4e1700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).writer sleeping
2012-05-21 07:28:13.365087 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader got ACK
2012-05-21 07:28:13.365112 7ff19d3e0700 15 reader got ack seq 8
2012-05-21 07:28:13.365113 7ff19d3e0700 20 -- 10.100.1.16:0/1016781 >> 10.100.1.11:6802/30752 pipe(0x7ff19800b350 sd=5 pgs=1566 cs=1 l=1).reader reading tag...