Project

General

Profile

Bug #5674 ยป radosgw.log

Faidon Liambotis, 07/19/2013 08:38 AM

 
2013-07-19 07:06:18.700054 7f1baff4a780 0 ceph version 0.66-699-gc0e77c9 (c0e77c91b6c39998ef4e19a726db87b66850cf2c), process radosgw, pid 3923
2013-07-19 07:10:18.643383 7fd6c21bc780 0 ceph version 0.66-699-gc0e77c9 (c0e77c91b6c39998ef4e19a726db87b66850cf2c), process radosgw, pid 4057
2013-07-19 07:10:18.643420 7fd6c21bc780 -1 WARNING: libcurl doesn't support curl_multi_wait()
2013-07-19 07:10:18.643426 7fd6c21bc780 -1 WARNING: cross zone / region transfer performance may be affected
2013-07-19 07:10:18.647850 7fd6c21bc780 10 -- :/0 ready :/0
2013-07-19 07:10:18.647909 7fd6c21bc780 1 -- :/0 messenger.start
2013-07-19 07:10:18.648123 7fd6b851e700 10 -- :/1004059 reaper_entry start
2013-07-19 07:10:18.648419 7fd6b851e700 10 -- :/1004059 reaper
2013-07-19 07:10:18.648423 7fd6b851e700 10 -- :/1004059 reaper done
2013-07-19 07:10:18.648767 7fd6c21bc780 10 -- :/1004059 connect_rank to 10.64.0.167:6789/0, creating pipe and registering
2013-07-19 07:10:18.648822 7fd6c21bc780 10 -- :/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=-1 :0 s=1 pgs=0 cs=0 l=1 c=0xd386e0).register_pipe
2013-07-19 07:10:18.648890 7fd6c21bc780 10 -- :/1004059 get_connection mon.0 10.64.0.167:6789/0 new 0xd38480
2013-07-19 07:10:18.648884 7fd6c21b4700 10 -- :/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=-1 :0 s=1 pgs=0 cs=0 l=1 c=0xd386e0).writer: state = connecting policy.server=0
2013-07-19 07:10:18.648929 7fd6c21b4700 10 -- :/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=-1 :0 s=1 pgs=0 cs=0 l=1 c=0xd386e0).connect 0
2013-07-19 07:10:18.648960 7fd6c21b4700 10 -- :/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0xd386e0).connecting to 10.64.0.167:6789/0
2013-07-19 07:10:18.648995 7fd6c21bc780 1 -- :/1004059 --> 10.64.0.167:6789/0 -- auth(proto 0 32 bytes epoch 0) v1 -- ?+0 0xd38c40 con 0xd386e0
2013-07-19 07:10:18.649010 7fd6c21bc780 20 -- :/1004059 submit_message auth(proto 0 32 bytes epoch 0) v1 remote, 10.64.0.167:6789/0, have pipe.
2013-07-19 07:10:18.649623 7fd6c21b4700 20 -- :/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=1 pgs=0 cs=0 l=1 c=0xd386e0).connect read peer addr 10.64.0.167:6789/0 on socket 9
2013-07-19 07:10:18.649655 7fd6c21b4700 20 -- :/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=1 pgs=0 cs=0 l=1 c=0xd386e0).connect peer addr for me is 10.64.32.92:16279/0
2013-07-19 07:10:18.649668 7fd6c21b4700 1 -- 10.64.32.92:0/1004059 learned my addr 10.64.32.92:0/1004059
2013-07-19 07:10:18.649691 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=1 pgs=0 cs=0 l=1 c=0xd386e0).connect sent my addr 10.64.32.92:0/1004059
2013-07-19 07:10:18.649711 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=1 pgs=0 cs=0 l=1 c=0xd386e0).connect sending gseq=1 cseq=0 proto=15
2013-07-19 07:10:18.649732 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=1 pgs=0 cs=0 l=1 c=0xd386e0).connect wrote (self +) cseq, waiting for reply
2013-07-19 07:10:18.649869 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=1 pgs=0 cs=0 l=1 c=0xd386e0).connect got reply tag 1 connect_seq 1 global_seq 25 proto 15 flags 1 features 34359738367
2013-07-19 07:10:18.649912 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).connect success 1, lossy = 1, features 34359738367
2013-07-19 07:10:18.649938 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).connect starting reader
2013-07-19 07:10:18.650008 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.650044 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer encoding 1 features 34359738367 0xd38c40 auth(proto 0 32 bytes epoch 0) v1
2013-07-19 07:10:18.650063 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer no session security
2013-07-19 07:10:18.650071 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sending 1 0xd38c40
2013-07-19 07:10:18.650115 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.650127 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.650083 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.650654 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ACK
2013-07-19 07:10:18.650681 7fd6b7d1d700 15 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ack seq 1
2013-07-19 07:10:18.650691 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.650700 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.650708 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=4 src mon.0 front=497 data=0 off 0
2013-07-19 07:10:18.650732 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 497 from dispatch throttler 0/104857600
2013-07-19 07:10:18.650748 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 497
2013-07-19 07:10:18.650759 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.650766 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 497 + 0 + 0 byte message
2013-07-19 07:10:18.650792 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.650804 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 1 0x7fd69c001890 mon_map v1
2013-07-19 07:10:18.650870 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c001890 prio 196
2013-07-19 07:10:18.650931 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.650993 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.650988 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.650949 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 1 ==== mon_map v1 ==== 497+0+0 (1640470026 0 0) 0x7fd69c001890 con 0xd386e0
2013-07-19 07:10:18.651008 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 1
2013-07-19 07:10:18.651011 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=18 src mon.0 front=33 data=0 off 0
2013-07-19 07:10:18.651020 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 33 from dispatch throttler 497/104857600
2013-07-19 07:10:18.651025 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.651030 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 33
2013-07-19 07:10:18.651035 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.651040 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.651047 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 33 + 0 + 0 byte message
2013-07-19 07:10:18.651062 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.651078 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 497 to dispatch throttler 530/104857600
2013-07-19 07:10:18.651084 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c001890
2013-07-19 07:10:18.651071 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 2 0x7fd69c002930 auth_reply(proto 2 0 Success) v1
2013-07-19 07:10:18.651105 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c002930 prio 196
2013-07-19 07:10:18.651114 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.651124 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.651144 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 2
2013-07-19 07:10:18.651135 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (2607927786 0 0) 0x7fd69c002930 con 0xd386e0
2013-07-19 07:10:18.651158 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.651168 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.651436 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 --> 10.64.0.167:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7fd6a0001900 con 0xd386e0
2013-07-19 07:10:18.651463 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 submit_message auth(proto 2 32 bytes epoch 0) v1 remote, 10.64.0.167:6789/0, have pipe.
2013-07-19 07:10:18.651474 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 33 to dispatch throttler 33/104857600
2013-07-19 07:10:18.651480 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c002930
2013-07-19 07:10:18.651533 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.651558 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer encoding 2 features 34359738367 0x7fd6a0001900 auth(proto 2 32 bytes epoch 0) v1
2013-07-19 07:10:18.651583 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer no session security
2013-07-19 07:10:18.651594 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sending 2 0x7fd6a0001900
2013-07-19 07:10:18.651617 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.651625 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.652255 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ACK
2013-07-19 07:10:18.652281 7fd6b7d1d700 15 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ack seq 2
2013-07-19 07:10:18.652292 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.652302 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.652317 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=18 src mon.0 front=222 data=0 off 0
2013-07-19 07:10:18.652326 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 222 from dispatch throttler 0/104857600
2013-07-19 07:10:18.652338 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 222
2013-07-19 07:10:18.652348 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.652354 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 222 + 0 + 0 byte message
2013-07-19 07:10:18.652364 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.652374 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 3 0x7fd69c003870 auth_reply(proto 2 0 Success) v1
2013-07-19 07:10:18.652385 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c003870 prio 196
2013-07-19 07:10:18.652393 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.652397 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.652413 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 3
2013-07-19 07:10:18.652427 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.652437 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.652454 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 222+0+0 (4236280918 0 0) 0x7fd69c003870 con 0xd386e0
2013-07-19 07:10:18.652570 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 --> 10.64.0.167:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0 0x7fd6a0002000 con 0xd386e0
2013-07-19 07:10:18.652578 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 submit_message auth(proto 2 181 bytes epoch 0) v1 remote, 10.64.0.167:6789/0, have pipe.
2013-07-19 07:10:18.652594 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 222 to dispatch throttler 222/104857600
2013-07-19 07:10:18.652599 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c003870
2013-07-19 07:10:18.652625 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.652641 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer encoding 3 features 34359738367 0x7fd6a0002000 auth(proto 2 181 bytes epoch 0) v1
2013-07-19 07:10:18.652657 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer no session security
2013-07-19 07:10:18.652668 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sending 3 0x7fd6a0002000
2013-07-19 07:10:18.652691 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.652700 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.653309 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ACK
2013-07-19 07:10:18.653335 7fd6b7d1d700 15 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ack seq 3
2013-07-19 07:10:18.653351 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.653359 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.653368 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=18 src mon.0 front=409 data=0 off 0
2013-07-19 07:10:18.653377 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 409 from dispatch throttler 0/104857600
2013-07-19 07:10:18.653388 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 409
2013-07-19 07:10:18.653398 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.653405 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 409 + 0 + 0 byte message
2013-07-19 07:10:18.653414 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.653424 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 4 0x7fd69c004bb0 auth_reply(proto 2 0 Success) v1
2013-07-19 07:10:18.653479 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c004bb0 prio 196
2013-07-19 07:10:18.653489 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.653516 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.653534 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 4
2013-07-19 07:10:18.653526 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 409+0+0 (2081683292 0 0) 0x7fd69c004bb0 con 0xd386e0
2013-07-19 07:10:18.653546 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.653553 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.653587 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 --> 10.64.0.167:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0xd3d090 con 0xd386e0
2013-07-19 07:10:18.653596 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 submit_message mon_subscribe({monmap=0+}) v2 remote, 10.64.0.167:6789/0, have pipe.
2013-07-19 07:10:18.653657 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 409 to dispatch throttler 409/104857600
2013-07-19 07:10:18.653697 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c004bb0
2013-07-19 07:10:18.653663 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.653726 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer encoding 4 features 34359738367 0xd3d090 mon_subscribe({monmap=0+}) v2
2013-07-19 07:10:18.653766 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer no session security
2013-07-19 07:10:18.653783 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sending 4 0xd3d090
2013-07-19 07:10:18.653804 7fd6c21bc780 1 -- 10.64.32.92:0/1004059 --> 10.64.0.167:6789/0 -- mon_subscribe({monmap=20+,osdmap=0}) v2 -- ?+0 0xd39b70 con 0xd386e0
2013-07-19 07:10:18.653811 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.653820 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.653845 7fd6c21bc780 20 -- 10.64.32.92:0/1004059 submit_message mon_subscribe({monmap=20+,osdmap=0}) v2 remote, 10.64.0.167:6789/0, have pipe.
2013-07-19 07:10:18.653911 7fd6c21bc780 1 -- 10.64.32.92:0/1004059 --> 10.64.0.167:6789/0 -- mon_subscribe({monmap=20+,osdmap=0}) v2 -- ?+0 0xd39f60 con 0xd386e0
2013-07-19 07:10:18.653911 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.653930 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer encoding 5 features 34359738367 0xd39b70 mon_subscribe({monmap=20+,osdmap=0}) v2
2013-07-19 07:10:18.653947 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer no session security
2013-07-19 07:10:18.653960 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sending 5 0xd39b70
2013-07-19 07:10:18.653980 7fd6c21bc780 20 -- 10.64.32.92:0/1004059 submit_message mon_subscribe({monmap=20+,osdmap=0}) v2 remote, 10.64.0.167:6789/0, have pipe.
2013-07-19 07:10:18.654010 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.654029 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer encoding 6 features 34359738367 0xd39f60 mon_subscribe({monmap=20+,osdmap=0}) v2
2013-07-19 07:10:18.654085 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer no session security
2013-07-19 07:10:18.654096 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sending 6 0xd39f60
2013-07-19 07:10:18.654115 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.654123 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.654233 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ACK
2013-07-19 07:10:18.654238 7fd6b6c1a700 2 RGWDataChangesLog::ChangesRenewThread: start
2013-07-19 07:10:18.654288 7fd6b7d1d700 15 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ack seq 4
2013-07-19 07:10:18.654300 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.654312 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ACK
2013-07-19 07:10:18.654331 7fd6b7d1d700 15 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ack seq 5
2013-07-19 07:10:18.654339 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.654346 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.654354 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=4 src mon.0 front=497 data=0 off 0
2013-07-19 07:10:18.654363 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 497 from dispatch throttler 0/104857600
2013-07-19 07:10:18.654374 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 497
2013-07-19 07:10:18.654388 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.654395 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 497 + 0 + 0 byte message
2013-07-19 07:10:18.654405 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.654415 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 5 0x7fd69c006330 mon_map v1
2013-07-19 07:10:18.654423 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c006330 prio 196
2013-07-19 07:10:18.654445 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.654489 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.654453 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.654497 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 5 ==== mon_map v1 ==== 497+0+0 (1640470026 0 0) 0x7fd69c006330 con 0xd386e0
2013-07-19 07:10:18.654498 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=16 src mon.0 front=20 data=0 off 0
2013-07-19 07:10:18.654517 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 5
2013-07-19 07:10:18.654524 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 20 from dispatch throttler 497/104857600
2013-07-19 07:10:18.654535 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 497 to dispatch throttler 517/104857600
2013-07-19 07:10:18.654540 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c006330
2013-07-19 07:10:18.654537 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 20
2013-07-19 07:10:18.654537 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.654548 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.654548 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.654556 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 20 + 0 + 0 byte message
2013-07-19 07:10:18.654571 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.654581 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 6 0x7fd69c006f70 mon_subscribe_ack(300s) v1
2013-07-19 07:10:18.654589 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c006f70 prio 196
2013-07-19 07:10:18.654596 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.654605 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ACK
2013-07-19 07:10:18.654601 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.654636 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 6
2013-07-19 07:10:18.654642 7fd6b7d1d700 15 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ack seq 6
2013-07-19 07:10:18.654646 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2485604908 0 0) 0x7fd69c006f70 con 0xd386e0
2013-07-19 07:10:18.654654 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.654666 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 20 to dispatch throttler 20/104857600
2013-07-19 07:10:18.654660 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.654671 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c006f70
2013-07-19 07:10:18.654673 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.654894 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.654917 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=41 src mon.0 front=98995 data=0 off 0
2013-07-19 07:10:18.654929 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 98995 from dispatch throttler 0/104857600
2013-07-19 07:10:18.656952 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 98995
2013-07-19 07:10:18.656976 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.656988 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 98995 + 0 + 0 byte message
2013-07-19 07:10:18.657088 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.657102 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 7 0x7fd69c020140 osd_map(195833..195833 src has 195089..195833) v3
2013-07-19 07:10:18.657131 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c020140 prio 196
2013-07-19 07:10:18.657193 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.657208 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 7 ==== osd_map(195833..195833 src has 195089..195833) v3 ==== 98995+0+0 (1044879129 0 0) 0x7fd69c020140 con 0xd386e0
2013-07-19 07:10:18.657223 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.657232 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=16 src mon.0 front=20 data=0 off 0
2013-07-19 07:10:18.657227 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.657241 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 20 from dispatch throttler 98995/104857600
2013-07-19 07:10:18.657243 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 7
2013-07-19 07:10:18.657252 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 20
2013-07-19 07:10:18.657256 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.657265 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.657261 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.657296 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 20 + 0 + 0 byte message
2013-07-19 07:10:18.657361 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.657371 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 8 0x7fd69c021110 mon_subscribe_ack(300s) v1
2013-07-19 07:10:18.657379 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c021110 prio 196
2013-07-19 07:10:18.657385 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.657396 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.657407 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=41 src mon.0 front=98995 data=0 off 0
2013-07-19 07:10:18.657405 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.657429 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 98995 from dispatch throttler 99015/104857600
2013-07-19 07:10:18.657436 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 8
2013-07-19 07:10:18.657468 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.657481 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.657892 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 98995 to dispatch throttler 198010/104857600
2013-07-19 07:10:18.657939 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c020140
2013-07-19 07:10:18.657948 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2485604908 0 0) 0x7fd69c021110 con 0xd386e0
2013-07-19 07:10:18.657959 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 20 to dispatch throttler 99015/104857600
2013-07-19 07:10:18.657963 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c021110
2013-07-19 07:10:18.657983 7fd6c21bc780 1 -- 10.64.32.92:0/1004059 --> 10.64.0.167:6789/0 -- pool_op(create pool 0 auid 0 tid 1 name v0) v4 -- ?+0 0xd3c280 con 0xd386e0
2013-07-19 07:10:18.657995 7fd6c21bc780 20 -- 10.64.32.92:0/1004059 submit_message pool_op(create pool 0 auid 0 tid 1 name v0) v4 remote, 10.64.0.167:6789/0, have pipe.
2013-07-19 07:10:18.658060 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.658084 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer encoding 7 features 34359738367 0xd3c280 pool_op(create pool 0 auid 0 tid 1 name v0) v4
2013-07-19 07:10:18.658118 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer no session security
2013-07-19 07:10:18.658132 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sending 7 0xd3c280
2013-07-19 07:10:18.658163 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.658172 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.658966 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 98995
2013-07-19 07:10:18.658991 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.658999 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 98995 + 0 + 0 byte message
2013-07-19 07:10:18.659092 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.659105 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 9 0x7fd69c007c10 osd_map(195833..195833 src has 195089..195833) v3
2013-07-19 07:10:18.659115 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c007c10 prio 196
2013-07-19 07:10:18.659127 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.659178 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.659185 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 9 ==== osd_map(195833..195833 src has 195089..195833) v3 ==== 98995+0+0 (1044879129 0 0) 0x7fd69c007c10 con 0xd386e0
2013-07-19 07:10:18.659202 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 98995 to dispatch throttler 98995/104857600
2013-07-19 07:10:18.659210 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c007c10
2013-07-19 07:10:18.659136 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.659217 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 9
2013-07-19 07:10:18.659207 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=16 src mon.0 front=20 data=0 off 0
2013-07-19 07:10:18.659242 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 20 from dispatch throttler 0/104857600
2013-07-19 07:10:18.659242 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.659265 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 20
2013-07-19 07:10:18.659269 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.659275 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.659283 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 20 + 0 + 0 byte message
2013-07-19 07:10:18.659291 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.659299 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 10 0x7fd69c008860 mon_subscribe_ack(300s) v1
2013-07-19 07:10:18.659309 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c008860 prio 196
2013-07-19 07:10:18.659340 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.659349 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 10 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2485604908 0 0) 0x7fd69c008860 con 0xd386e0
2013-07-19 07:10:18.659370 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ACK
2013-07-19 07:10:18.659374 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.659389 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 20 to dispatch throttler 20/104857600
2013-07-19 07:10:18.659396 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c008860
2013-07-19 07:10:18.659394 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 10
2013-07-19 07:10:18.659407 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.659419 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.659451 7fd6b7d1d700 15 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got ack seq 7
2013-07-19 07:10:18.659470 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.659483 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got MSG
2013-07-19 07:10:18.659495 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got envelope type=48 src mon.0 front=43 data=0 off 0
2013-07-19 07:10:18.659509 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader wants 43 from dispatch throttler 0/104857600
2013-07-19 07:10:18.659522 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got front 43
2013-07-19 07:10:18.659535 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).aborted = 0
2013-07-19 07:10:18.659542 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got 43 + 0 + 0 byte message
2013-07-19 07:10:18.659552 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).No session security set
2013-07-19 07:10:18.659562 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader got message 11 0x7fd69c009ab0 pool_op_reply(tid 1 (1) Operation not permitted v195833) v1
2013-07-19 07:10:18.659591 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 queue 0x7fd69c009ab0 prio 196
2013-07-19 07:10:18.659642 7fd6b7d1d700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).reader reading tag...
2013-07-19 07:10:18.659650 7fd6b8d1f700 1 -- 10.64.32.92:0/1004059 <== mon.0 10.64.0.167:6789/0 11 ==== pool_op_reply(tid 1 (1) Operation not permitted v195833) v1 ==== 43+0+0 (902465819 0 0) 0x7fd69c009ab0 con 0xd386e0
2013-07-19 07:10:18.659672 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.659690 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).write_ack 11
2013-07-19 07:10:18.659702 7fd6b8d1f700 10 -- 10.64.32.92:0/1004059 dispatch_throttle_release 43 to dispatch throttler 43/104857600
2013-07-19 07:10:18.659702 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer: state = open policy.server=0
2013-07-19 07:10:18.659710 7fd6b8d1f700 20 -- 10.64.32.92:0/1004059 done calling dispatch on 0x7fd69c009ab0
2013-07-19 07:10:18.659713 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).writer sleeping
2013-07-19 07:10:18.659947 7fd6c21bc780 1 -- 10.64.32.92:0/1004059 mark_down 0xd386e0 -- 0xd38480
2013-07-19 07:10:18.659966 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).unregister_pipe
2013-07-19 07:10:18.659979 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=2 pgs=25 cs=1 l=1 c=0xd386e0).stop
2013-07-19 07:10:18.660023 7fd6c21b4700 20 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=4 pgs=25 cs=1 l=1 c=0xd386e0).writer finishing
2013-07-19 07:10:18.660040 7fd6c21b4700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=4 pgs=25 cs=1 l=1 c=0xd386e0).writer done
2013-07-19 07:10:18.660064 7fd6b7d1d700 2 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=4 pgs=25 cs=1 l=1 c=0xd386e0).reader couldn't read tag, Success
2013-07-19 07:10:18.660089 7fd6b7d1d700 2 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=4 pgs=25 cs=1 l=1 c=0xd386e0).fault 0: Success
2013-07-19 07:10:18.660101 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=4 pgs=25 cs=1 l=1 c=0xd386e0).fault already closed|closing
2013-07-19 07:10:18.660111 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 queue_reap 0xd38480
2013-07-19 07:10:18.660120 7fd6b7d1d700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=4 pgs=25 cs=1 l=1 c=0xd386e0).reader done
2013-07-19 07:10:18.660195 7fd6b851e700 10 -- 10.64.32.92:0/1004059 reaper
2013-07-19 07:10:18.660212 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 shutdown 10.64.32.92:0/1004059
2013-07-19 07:10:18.660216 7fd6b851e700 10 -- 10.64.32.92:0/1004059 reaper reaping pipe 0xd38480 10.64.0.167:6789/0
2013-07-19 07:10:18.660222 7fd6c21bc780 1 -- 10.64.32.92:0/1004059 mark_down_all
2013-07-19 07:10:18.660223 7fd6b851e700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=4 pgs=25 cs=1 l=1 c=0xd386e0).discard_queue
2013-07-19 07:10:18.660230 7fd6b851e700 10 -- 10.64.32.92:0/1004059 >> 10.64.0.167:6789/0 pipe(0xd38480 sd=9 :16279 s=4 pgs=25 cs=1 l=1 c=0xd386e0).unregister_pipe - not registered
2013-07-19 07:10:18.660245 7fd6b851e700 10 -- 10.64.32.92:0/1004059 reaper reaped pipe 0xd38480 10.64.0.167:6789/0
2013-07-19 07:10:18.660256 7fd6b851e700 10 -- 10.64.32.92:0/1004059 reaper deleted pipe 0xd38480
2013-07-19 07:10:18.660259 7fd6b851e700 10 -- 10.64.32.92:0/1004059 reaper done
2013-07-19 07:10:18.660276 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 wait: waiting for dispatch queue
2013-07-19 07:10:18.660346 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 wait: dispatch queue is stopped
2013-07-19 07:10:18.660357 7fd6c21bc780 20 -- 10.64.32.92:0/1004059 wait: stopping reaper thread
2013-07-19 07:10:18.660420 7fd6b851e700 10 -- 10.64.32.92:0/1004059 reaper_entry done
2013-07-19 07:10:18.660458 7fd6c21bc780 20 -- 10.64.32.92:0/1004059 wait: stopped reaper thread
2013-07-19 07:10:18.660473 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 wait: closing pipes
2013-07-19 07:10:18.660478 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 reaper
2013-07-19 07:10:18.660482 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 reaper done
2013-07-19 07:10:18.660486 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 wait: waiting for pipes to close
2013-07-19 07:10:18.660490 7fd6c21bc780 10 -- 10.64.32.92:0/1004059 wait: done.
2013-07-19 07:10:18.660494 7fd6c21bc780 1 -- 10.64.32.92:0/1004059 shutdown complete.
2013-07-19 07:10:18.660665 7fd6c21bc780 -1 Couldn't init storage provider (RADOS)
    (1-1/1)