2017-12-24 18:53:32.895017 7f01932ec700 -1 received signal: Terminated from PID: 1 task name: /sbin/init UID: 0 2017-12-24 18:53:32.895039 7f01932ec700 -1 osd.0 1080 *** Got signal Terminated *** 2017-12-24 18:53:32.895229 7f01932ec700 0 osd.0 1080 prepare_to_stop telling mon we are shutting down 2017-12-24 18:53:32.949841 7f01acba2700 0 osd.0 1080 got_stop_ack starting shutdown 2017-12-24 18:53:32.949873 7f01932ec700 0 osd.0 1080 prepare_to_stop starting shutdown 2017-12-24 18:53:32.949881 7f01932ec700 -1 osd.0 1080 shutdown 2017-12-24 18:53:32.950758 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 64 mask is 1 2017-12-24 18:53:32.950764 7f01acba2700 2 osd.0 1080 ms_handle_reset con 0x55825b1e1000 session 0x558245c63c00 2017-12-24 18:53:32.950771 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x55812f5404b0 2017-12-24 18:53:32.950775 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_time_event id=135260 2017-12-24 18:53:32.950778 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event started fd=64 mask=3 original mask is 1 2017-12-24 18:53:32.950832 7f01b6317700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.950848 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.950862 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_bulk peer close file descriptor 65 2017-12-24 18:53:32.950869 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.950876 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.950874 7f01b5b16700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.950884 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.950886 7f01b5b16700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.950890 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.950896 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1).read_bulk peer close file descriptor 57 2017-12-24 18:53:32.950894 7f01b6317700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1)._stop 2017-12-24 18:53:32.950901 7f01b5b16700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.950902 7f01b6317700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/3011660791 conn(0x55823663d000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.950905 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.950907 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x5581d26fce70 pending 1 2017-12-24 18:53:32.950908 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.950915 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 65 mask is 1 2017-12-24 18:53:32.950915 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.950920 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x5581d26fce70 2017-12-24 18:53:32.950922 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_time_event id=390543 2017-12-24 18:53:32.950922 7f01acba2700 2 osd.0 1080 ms_handle_reset con 0x55823663d000 session 0x5581bba72c80 2017-12-24 18:53:32.950922 7f01b5b16700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1)._stop 2017-12-24 18:53:32.950925 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event started fd=65 mask=3 original mask is 1 2017-12-24 18:53:32.950926 7f01b5b16700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.5:0/3675858229 conn(0x55813c557800 :6801 s=STATE_OPEN pgs=2360 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.950933 7f01b5b16700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:32.950936 7f01b5b16700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x558118166310 pending 1 2017-12-24 18:53:32.950939 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x5581b6941800 pending 1 2017-12-24 18:53:32.950942 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 57 mask is 1 2017-12-24 18:53:32.950944 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x5581b6941800 2017-12-24 18:53:32.950947 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_time_event id=188650 2017-12-24 18:53:32.950950 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event started fd=57 mask=3 original mask is 1 2017-12-24 18:53:32.950926 7f01acba2700 30 osd.0 pg_epoch: 1080 pg[1.a8( v 1080'364308 (1076'362741,1080'364308] local-lis/les=1054/1055 n=1491 ec=18/18 lis/c 1054/1054 les/c/f 1055/1055/0 1054/1054/1052) [0,2,1] r=0 lpr=1054 crt=1080'364308 lcod 1080'364307 mlcod 1080'364307 active+clean] lock 2017-12-24 18:53:32.950952 7f01acba2700 10 osd.0 pg_epoch: 1080 pg[1.a8( v 1080'364308 (1076'362741,1080'364308] local-lis/les=1054/1055 n=1491 ec=18/18 lis/c 1054/1054 les/c/f 1055/1055/0 1054/1054/1052) [0,2,1] r=0 lpr=1054 crt=1080'364308 lcod 1080'364307 mlcod 1080'364307 active+clean] -- Watch(139663746548208,client.2208801) disconnect (con was 0x55823663d000) 2017-12-24 18:53:32.950977 7f01acba2700 2 osd.0 1080 ms_handle_reset con 0x55813c557800 session 0x558225006c80 2017-12-24 18:53:32.951034 7f01b6317700 20 EpollDriver.del_event del event fd=65 cur_mask=1 delmask=3 to 7 2017-12-24 18:53:32.951036 7f01b6b18700 20 EpollDriver.del_event del event fd=64 cur_mask=1 delmask=3 to 4 2017-12-24 18:53:32.951038 7f01b5b16700 20 EpollDriver.del_event del event fd=57 cur_mask=1 delmask=3 to 10 2017-12-24 18:53:32.951040 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event end fd=65 mask=3 original mask is 0 2017-12-24 18:53:32.951043 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event end fd=64 mask=3 original mask is 0 2017-12-24 18:53:32.951045 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event end fd=57 mask=3 original mask is 0 2017-12-24 18:53:32.951067 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:32.951069 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:32.951070 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:32.951072 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:32.951078 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:32.951076 7f01b5b16700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.951081 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 0 usec 0 2017-12-24 18:53:32.951081 7f01b5b16700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.951086 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:32.951088 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x558118166310 2017-12-24 18:53:32.951087 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_bulk peer close file descriptor 69 2017-12-24 18:53:32.951091 7f01b5b16700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.951090 7f01b6b18700 1 -- 10.103.0.1:6801/6033 reap_dead start 2017-12-24 18:53:32.951094 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.951097 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x55813c557800 2017-12-24 18:53:32.951098 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.951101 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.951104 7f01b5b16700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1)._stop 2017-12-24 18:53:32.951107 7f01b5b16700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.3:6800/234216682 conn(0x5581f5e72800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.951119 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x55814c73e800 2017-12-24 18:53:32.951132 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x55823663d000 2017-12-24 18:53:32.951144 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x55825b1e1000 2017-12-24 18:53:32.951160 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x5582b8a73000 2017-12-24 18:53:32.951181 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:32.951183 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:32.951184 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x5581b6941800 pending 1 2017-12-24 18:53:32.951191 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 69 mask is 1 2017-12-24 18:53:32.951193 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x5581b6941800 2017-12-24 18:53:32.951195 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_time_event id=188662 2017-12-24 18:53:32.951197 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event started fd=69 mask=3 original mask is 1 2017-12-24 18:53:32.951196 7f01acba2700 2 osd.0 1080 ms_handle_reset con 0x5581f5e72800 session 0x5581bba74f80 2017-12-24 18:53:32.951199 7f01b5b16700 20 EpollDriver.del_event del event fd=69 cur_mask=1 delmask=3 to 10 2017-12-24 18:53:32.951203 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event end fd=69 mask=3 original mask is 0 2017-12-24 18:53:32.951221 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:32.951222 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:32.951227 7f01b5b16700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.951233 7f01b5b16700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.951239 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1).read_bulk peer close file descriptor 38 2017-12-24 18:53:32.951243 7f01b5b16700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.951247 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.951250 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.951254 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.951257 7f01b5b16700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1)._stop 2017-12-24 18:53:32.951260 7f01b5b16700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.3:0/2266272612 conn(0x558242d4a800 :6801 s=STATE_OPEN pgs=2066 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.951264 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x5581b6941800 pending 1 2017-12-24 18:53:32.951270 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 38 mask is 1 2017-12-24 18:53:32.951272 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x5581b6941800 2017-12-24 18:53:32.951273 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_time_event id=188655 2017-12-24 18:53:32.951275 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event started fd=38 mask=3 original mask is 1 2017-12-24 18:53:32.951275 7f01acba2700 2 osd.0 1080 ms_handle_reset con 0x558242d4a800 session 0x55827664ca80 2017-12-24 18:53:32.951278 7f01b5b16700 20 EpollDriver.del_event del event fd=38 cur_mask=1 delmask=3 to 10 2017-12-24 18:53:32.951281 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event end fd=38 mask=3 original mask is 0 2017-12-24 18:53:32.951275 7f01b6317700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.951284 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.951290 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_bulk peer close file descriptor 37 2017-12-24 18:53:32.951293 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.951295 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:32.951297 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:32.951297 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.951300 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.951304 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.951308 7f01b6317700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1)._stop 2017-12-24 18:53:32.951312 7f01b6317700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.2:0/1591466891 conn(0x55823ce5a800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.951316 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x5581d26fce70 pending 1 2017-12-24 18:53:32.951321 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 37 mask is 1 2017-12-24 18:53:32.951323 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x5581d26fce70 2017-12-24 18:53:32.951326 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_time_event id=390540 2017-12-24 18:53:32.951327 7f01acba2700 2 osd.0 1080 ms_handle_reset con 0x55823ce5a800 session 0x5582b4155480 2017-12-24 18:53:32.951331 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event started fd=37 mask=3 original mask is 1 2017-12-24 18:53:32.951333 7f01b6317700 20 EpollDriver.del_event del event fd=37 cur_mask=1 delmask=3 to 7 2017-12-24 18:53:32.951336 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event end fd=37 mask=3 original mask is 0 2017-12-24 18:53:32.951331 7f01acba2700 30 osd.0 pg_epoch: 1080 pg[1.ec( v 1080'297083 (1076'295525,1080'297083] local-lis/les=1054/1055 n=1602 ec=18/18 lis/c 1054/1054 les/c/f 1055/1055/0 1054/1054/1052) [0,3,1] r=0 lpr=1054 crt=1080'297083 lcod 1079'297082 mlcod 1079'297082 active+clean] lock 2017-12-24 18:53:32.951341 7f01acba2700 10 osd.0 pg_epoch: 1080 pg[1.ec( v 1080'297083 (1076'295525,1080'297083] local-lis/les=1054/1055 n=1602 ec=18/18 lis/c 1054/1054 les/c/f 1055/1055/0 1054/1054/1052) [0,3,1] r=0 lpr=1054 crt=1080'297083 lcod 1079'297082 mlcod 1079'297082 active+clean] -- Watch(140288127723936,client.1653169) disconnect (con was 0x55823ce5a800) 2017-12-24 18:53:32.951357 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:32.951359 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:32.951457 7f01b6317700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.951475 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.951487 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1).read_bulk peer close file descriptor 63 2017-12-24 18:53:32.951494 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.951501 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.951508 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.951514 7f01b6317700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.951519 7f01b6317700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1)._stop 2017-12-24 18:53:32.951527 7f01b6317700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.6:0/712136626 conn(0x55811a8b9000 :6801 s=STATE_OPEN pgs=6 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.951535 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x5581d26fce70 pending 1 2017-12-24 18:53:32.951544 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 63 mask is 1 2017-12-24 18:53:32.951547 7f01acba2700 2 osd.0 1080 ms_handle_reset con 0x55811a8b9000 session 0x55815096c500 2017-12-24 18:53:32.951548 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x5581d26fce70 2017-12-24 18:53:32.951552 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_time_event id=390542 2017-12-24 18:53:32.951556 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event started fd=63 mask=3 original mask is 1 2017-12-24 18:53:32.951560 7f01b6317700 20 EpollDriver.del_event del event fd=63 cur_mask=1 delmask=3 to 7 2017-12-24 18:53:32.951564 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event end fd=63 mask=3 original mask is 0 2017-12-24 18:53:32.951594 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:32.951599 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:32.952677 7f01b5b16700 20 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_OPEN pgs=4 cs=1 l=0).process prev state is STATE_OPEN 2017-12-24 18:53:32.952688 7f01b5b16700 25 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_OPEN pgs=4 cs=1 l=0).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.952689 7f01b6317700 20 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.952695 7f01b5b16700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_OPEN pgs=4 cs=1 l=0).read_bulk peer close file descriptor 54 2017-12-24 18:53:32.952700 7f01b5b16700 25 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_OPEN pgs=4 cs=1 l=0).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.952700 7f01b6317700 25 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.952703 7f01b5b16700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_OPEN pgs=4 cs=1 l=0).read_until read failed 2017-12-24 18:53:32.952703 7f01b6b18700 20 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.952707 7f01b5b16700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_OPEN pgs=4 cs=1 l=0).process read tag failed 2017-12-24 18:53:32.952707 7f01b6317700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1).read_bulk peer close file descriptor 52 2017-12-24 18:53:32.952711 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_time_event id=188651 2017-12-24 18:53:32.952711 7f01b6b18700 25 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.952711 7f01b6317700 25 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.952714 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event started fd=54 mask=3 original mask is 1 2017-12-24 18:53:32.952716 7f01b5b16700 20 EpollDriver.del_event del event fd=54 cur_mask=1 delmask=3 to 10 2017-12-24 18:53:32.952715 7f01b6317700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.952719 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event end fd=54 mask=3 original mask is 0 2017-12-24 18:53:32.952718 7f01b6b18700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1).read_bulk peer close file descriptor 53 2017-12-24 18:53:32.952719 7f01b6317700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.952722 7f01b6b18700 25 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.952724 7f01b6317700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.952726 7f01b6b18700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.952728 7f01b6317700 2 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1)._stop 2017-12-24 18:53:32.952729 7f01b6b18700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.952733 7f01b6317700 10 -- 10.103.0.1:6802/6033 >> 10.103.0.2:0/5930 conn(0x55813bee6000 :6802 s=STATE_OPEN pgs=3 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.952734 7f01b6b18700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.952737 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x5581d26fce70 pending 1 2017-12-24 18:53:32.952738 7f01b6b18700 2 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1)._stop 2017-12-24 18:53:32.952737 7f01b5b16700 10 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_OPEN pgs=4 cs=1 l=0).fault with nothing to send, going to standby 2017-12-24 18:53:32.952743 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 54 mask is 1 2017-12-24 18:53:32.952743 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 52 mask is 1 2017-12-24 18:53:32.952742 7f01b6b18700 10 -- 10.104.0.1:6801/6033 >> 10.104.0.2:0/5930 conn(0x55813bee7800 :6801 s=STATE_OPEN pgs=3 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.952745 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:32.952745 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x5581d26fce70 2017-12-24 18:53:32.952746 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:32.952746 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x55811a4b12f0 pending 1 2017-12-24 18:53:32.952747 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_time_event id=390539 2017-12-24 18:53:32.952749 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event started fd=52 mask=3 original mask is 1 2017-12-24 18:53:32.952752 7f01b6317700 20 EpollDriver.del_event del event fd=52 cur_mask=1 delmask=3 to 7 2017-12-24 18:53:32.952752 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 53 mask is 1 2017-12-24 18:53:32.952754 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x55811a4b12f0 2017-12-24 18:53:32.952755 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event end fd=52 mask=3 original mask is 0 2017-12-24 18:53:32.952756 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_time_event id=135250 2017-12-24 18:53:32.952759 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event started fd=53 mask=3 original mask is 1 2017-12-24 18:53:32.952761 7f01b6b18700 20 EpollDriver.del_event del event fd=53 cur_mask=1 delmask=3 to 4 2017-12-24 18:53:32.952764 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event end fd=53 mask=3 original mask is 0 2017-12-24 18:53:32.952774 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:32.952777 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:32.952783 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:32.952785 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:32.953083 7f01b6317700 20 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_OPEN pgs=9 cs=1 l=0).process prev state is STATE_OPEN 2017-12-24 18:53:32.953098 7f01b6317700 25 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_OPEN pgs=9 cs=1 l=0).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.953109 7f01b6317700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_OPEN pgs=9 cs=1 l=0).read_bulk peer close file descriptor 40 2017-12-24 18:53:32.953115 7f01b6317700 25 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_OPEN pgs=9 cs=1 l=0).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.953120 7f01b6317700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_OPEN pgs=9 cs=1 l=0).read_until read failed 2017-12-24 18:53:32.953125 7f01b6317700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_OPEN pgs=9 cs=1 l=0).process read tag failed 2017-12-24 18:53:32.953130 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_time_event id=390546 2017-12-24 18:53:32.953134 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event started fd=40 mask=3 original mask is 1 2017-12-24 18:53:32.953129 7f01b5b16700 20 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.953140 7f01b5b16700 25 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.953146 7f01b5b16700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1).read_bulk peer close file descriptor 39 2017-12-24 18:53:32.953150 7f01b5b16700 25 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.953137 7f01b6317700 20 EpollDriver.del_event del event fd=40 cur_mask=1 delmask=3 to 7 2017-12-24 18:53:32.953153 7f01b5b16700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.953156 7f01b5b16700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.953158 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event end fd=40 mask=3 original mask is 0 2017-12-24 18:53:32.953159 7f01b5b16700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.953163 7f01b5b16700 2 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1)._stop 2017-12-24 18:53:32.953167 7f01b5b16700 10 -- 10.103.0.1:6802/6033 >> 10.103.0.4:0/5071 conn(0x55813571b000 :6802 s=STATE_OPEN pgs=5 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.953171 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x5581b6941800 pending 1 2017-12-24 18:53:32.953177 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 39 mask is 1 2017-12-24 18:53:32.953179 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x5581b6941800 2017-12-24 18:53:32.953180 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_time_event id=188657 2017-12-24 18:53:32.953183 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event started fd=39 mask=3 original mask is 1 2017-12-24 18:53:32.953186 7f01b5b16700 20 EpollDriver.del_event del event fd=39 cur_mask=1 delmask=3 to 10 2017-12-24 18:53:32.953189 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event end fd=39 mask=3 original mask is 0 2017-12-24 18:53:32.953191 7f01b6317700 10 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_OPEN pgs=9 cs=1 l=0).fault with nothing to send, going to standby 2017-12-24 18:53:32.953201 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 40 mask is 1 2017-12-24 18:53:32.953213 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:32.953215 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:32.953219 7f01b5b16700 20 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.953226 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:32.953230 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:32.953226 7f01b6b18700 20 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.953224 7f01b5b16700 25 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.953237 7f01b6b18700 25 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.953241 7f01b5b16700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_bulk peer close file descriptor 46 2017-12-24 18:53:32.953243 7f01b6b18700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_bulk peer close file descriptor 48 2017-12-24 18:53:32.953245 7f01b5b16700 25 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.953246 7f01b6b18700 25 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.953248 7f01b5b16700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.953249 7f01b6b18700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.953251 7f01b5b16700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.953252 7f01b6b18700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.953254 7f01b5b16700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.953255 7f01b6b18700 1 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.953257 7f01b5b16700 2 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1)._stop 2017-12-24 18:53:32.953258 7f01b6b18700 2 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1)._stop 2017-12-24 18:53:32.953260 7f01b5b16700 10 -- 10.104.0.1:6801/6033 >> 10.104.0.4:0/5071 conn(0x5581356b6000 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.953265 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x5581b6941800 pending 1 2017-12-24 18:53:32.953263 7f01b6b18700 10 -- 10.104.0.1:6801/6033 >> 10.104.0.3:0/5420 conn(0x5581356ba800 :6801 s=STATE_OPEN pgs=4 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.953270 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x55811a4b12f0 pending 1 2017-12-24 18:53:32.953271 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 46 mask is 1 2017-12-24 18:53:32.953272 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 48 mask is 1 2017-12-24 18:53:32.953273 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x5581b6941800 2017-12-24 18:53:32.953274 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x55811a4b12f0 2017-12-24 18:53:32.953275 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_time_event id=188659 2017-12-24 18:53:32.953276 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_time_event id=135254 2017-12-24 18:53:32.953277 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event started fd=46 mask=3 original mask is 1 2017-12-24 18:53:32.953278 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event started fd=48 mask=3 original mask is 1 2017-12-24 18:53:32.953280 7f01b6b18700 20 EpollDriver.del_event del event fd=48 cur_mask=1 delmask=3 to 4 2017-12-24 18:53:32.953280 7f01b5b16700 20 EpollDriver.del_event del event fd=46 cur_mask=1 delmask=3 to 10 2017-12-24 18:53:32.953282 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event end fd=48 mask=3 original mask is 0 2017-12-24 18:53:32.953283 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event end fd=46 mask=3 original mask is 0 2017-12-24 18:53:32.953298 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:32.953300 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:32.953303 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:32.953305 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:32.953507 7f01b6b18700 20 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_OPEN pgs=6 cs=1 l=0).process prev state is STATE_OPEN 2017-12-24 18:53:32.953515 7f01b6b18700 25 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_OPEN pgs=6 cs=1 l=0).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.953521 7f01b6b18700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_OPEN pgs=6 cs=1 l=0).read_bulk peer close file descriptor 45 2017-12-24 18:53:32.953524 7f01b6b18700 25 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_OPEN pgs=6 cs=1 l=0).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.953527 7f01b6b18700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_OPEN pgs=6 cs=1 l=0).read_until read failed 2017-12-24 18:53:32.953530 7f01b6b18700 1 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_OPEN pgs=6 cs=1 l=0).process read tag failed 2017-12-24 18:53:32.953533 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_time_event id=135255 2017-12-24 18:53:32.953535 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event started fd=45 mask=3 original mask is 1 2017-12-24 18:53:32.953536 7f01b6b18700 20 EpollDriver.del_event del event fd=45 cur_mask=1 delmask=3 to 4 2017-12-24 18:53:32.953538 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event end fd=45 mask=3 original mask is 0 2017-12-24 18:53:32.953522 7f01b6317700 20 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:32.953544 7f01b6317700 25 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:32.953553 7f01b6317700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1).read_bulk peer close file descriptor 47 2017-12-24 18:53:32.953554 7f01b6b18700 10 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_OPEN pgs=6 cs=1 l=0).fault with nothing to send, going to standby 2017-12-24 18:53:32.953559 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 45 mask is 1 2017-12-24 18:53:32.953561 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:32.953562 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:32.953561 7f01b6317700 25 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:32.953567 7f01b6317700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1).read_until read failed 2017-12-24 18:53:32.953571 7f01b6317700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1).process read tag failed 2017-12-24 18:53:32.953577 7f01b6317700 1 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:32.953581 7f01b6317700 2 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1)._stop 2017-12-24 18:53:32.953594 7f01b6317700 10 -- 10.103.0.1:6802/6033 >> 10.103.0.3:0/5420 conn(0x5581356b9000 :6802 s=STATE_OPEN pgs=4 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.953602 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x5581d26fce70 pending 1 2017-12-24 18:53:32.953613 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 47 mask is 1 2017-12-24 18:53:32.953615 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x5581d26fce70 2017-12-24 18:53:32.953618 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_time_event id=390547 2017-12-24 18:53:32.953620 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event started fd=47 mask=3 original mask is 1 2017-12-24 18:53:32.953624 7f01b6317700 20 EpollDriver.del_event del event fd=47 cur_mask=1 delmask=3 to 7 2017-12-24 18:53:32.953628 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event end fd=47 mask=3 original mask is 0 2017-12-24 18:53:32.953660 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:32.953664 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:32.972649 7f01932ec700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6803/137693 conn(0x55812d107000 :-1 s=STATE_OPEN pgs=2 cs=1 l=1).mark_down 2017-12-24 18:53:32.972675 7f01932ec700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6803/137693 conn(0x55812d107000 :-1 s=STATE_OPEN pgs=2 cs=1 l=1)._stop 2017-12-24 18:53:32.972683 7f01932ec700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6803/137693 conn(0x55812d107000 :-1 s=STATE_OPEN pgs=2 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:32.972691 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:32.972702 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x558118166310 pending 1 2017-12-24 18:53:32.972706 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:32.972705 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:32.972711 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x558118166310 2017-12-24 18:53:32.972712 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x558133d05f20 pending 1 2017-12-24 18:53:32.972713 7f01b6b18700 1 -- 10.103.0.1:6801/6033 reap_dead start 2017-12-24 18:53:32.972717 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x55811a8b9000 2017-12-24 18:53:32.972720 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:32.972725 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x558133d05f20 2017-12-24 18:53:32.972727 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_time_event id=188672 2017-12-24 18:53:32.972731 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event started fd=35 mask=3 original mask is 1 2017-12-24 18:53:32.972733 7f01b5b16700 20 EpollDriver.del_event del event fd=35 cur_mask=1 delmask=3 to 10 2017-12-24 18:53:32.972736 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event end fd=35 mask=3 original mask is 0 2017-12-24 18:53:32.972741 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x55812d107000 2017-12-24 18:53:32.972744 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x5581f5e72800 2017-12-24 18:53:32.972756 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x55823ce5a800 2017-12-24 18:53:32.972771 7f01b6b18700 5 -- 10.103.0.1:6801/6033 reap_dead delete 0x558242d4a800 2017-12-24 18:53:32.972773 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:32.972775 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:32.972797 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:32.972800 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:32.973969 7f01932ec700 20 osd.0 1080 kicking pg 1.16 2017-12-24 18:53:32.973975 7f01a0306700 20 osd.0 op_wq(1) _process empty q, waiting 2017-12-24 18:53:32.973972 7f019f304700 20 osd.0 op_wq(3) _process empty q, waiting 2017-12-24 18:53:32.973979 7f019eb03700 20 osd.0 op_wq(4) _process empty q, waiting 2017-12-24 18:53:32.973983 7f019c2fe700 20 osd.0 op_wq(1) _process empty q, waiting 2017-12-24 18:53:32.973985 7f019a2fa700 20 osd.0 op_wq(5) _process empty q, waiting 2017-12-24 18:53:32.973985 7f019aafb700 20 osd.0 op_wq(4) _process empty q, waiting 2017-12-24 18:53:32.973972 7f019caff700 20 osd.0 op_wq(0) _process empty q, waiting 2017-12-24 18:53:32.973990 7f019db01700 20 osd.0 op_wq(6) _process empty q, waiting 2017-12-24 18:53:32.973992 7f019b2fc700 20 osd.0 op_wq(3) _process empty q, waiting 2017-12-24 18:53:32.973995 7f019bafd700 20 osd.0 op_wq(2) _process empty q, waiting 2017-12-24 18:53:32.973999 7f01a0b07700 20 osd.0 op_wq(0) _process empty q, waiting 2017-12-24 18:53:32.974001 7f019e302700 20 osd.0 op_wq(5) _process empty q, waiting 2017-12-24 18:53:32.973999 7f0199af9700 20 osd.0 op_wq(6) _process empty q, waiting 2017-12-24 18:53:32.973982 7f01932ec700 30 osd.0 pg_epoch: 1080 pg[1.16( v 1080'245195 (1076'243612,1080'245195] local-lis/les=1052/1053 n=1656 ec=18/18 lis/c 1052/1052 les/c/f 1053/1053/0 1052/1052/1041) [3,0,2] r=1 lpr=1052 luod=0'0 crt=1080'245195 lcod 1080'245194 active] lock 2017-12-24 18:53:32.974003 7f019d300700 20 osd.0 op_wq(7) _process empty q, waiting 2017-12-24 18:53:32.974003 7f019fb05700 20 osd.0 op_wq(2) _process empty q, waiting ........................................................................... the same things ................................. 2017-12-24 18:53:35.739705 7f01932ec700 30 osd.0 pg_epoch: 1080 pg[4.1a( v 1076'25623 (1076'24096,1076'25623] local-lis/les=1052/1053 n=33 ec=65/65 lis/c 1052/1052 les/c/f 1053/1053/0 1052/1052/1041) [3,0,2] r=1 lpr=1052 luod=0'0 crt=1076'25623 lcod 1076'25622 active] lock 2017-12-24 18:53:35.745425 7f01932ec700 20 osd.0 1080 kicking pg 1.48 2017-12-24 18:53:35.745436 7f01932ec700 30 osd.0 pg_epoch: 1080 pg[1.48( v 1076'403079 (1076'401485,1076'403079] local-lis/les=1054/1055 n=1553 ec=18/18 lis/c 1054/1054 les/c/f 1055/1055/0 1054/1054/1054) [1,0,2] r=1 lpr=1054 luod=0'0 crt=1076'403079 lcod 1076'403078 active] lock 2017-12-24 18:53:35.750499 7f01932ec700 10 osd.0 1080 syncing store 2017-12-24 18:53:35.750509 7f01932ec700 10 osd.0 1080 flushing journal 2017-12-24 18:53:35.750641 7f01932ec700 1 bluestore(/var/lib/ceph/osd/ceph-0) umount 2017-12-24 18:53:36.548759 7f01b5b16700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:36.548788 7f01b5b16700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:36.548799 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_bulk peer close file descriptor 58 2017-12-24 18:53:36.548803 7f01b5b16700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got -1 2017-12-24 18:53:36.548807 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).read_until read failed 2017-12-24 18:53:36.548810 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).process read tag failed 2017-12-24 18:53:36.548814 7f01b5b16700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).fault on lossy channel, failing 2017-12-24 18:53:36.548818 7f01b5b16700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1)._stop 2017-12-24 18:53:36.548823 7f01b5b16700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.1:0/1098863877 conn(0x55825d5c1800 :6801 s=STATE_OPEN pgs=5 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:36.548829 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x558118169290 pending 1 2017-12-24 18:53:36.548837 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 58 mask is 1 2017-12-24 18:53:36.548839 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x558118169290 2017-12-24 18:53:36.548842 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_time_event id=188673 2017-12-24 18:53:36.548845 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event started fd=58 mask=3 original mask is 1 2017-12-24 18:53:36.548850 7f01b5b16700 20 EpollDriver.del_event del event fd=58 cur_mask=1 delmask=3 to 10 2017-12-24 18:53:36.548854 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).delete_file_event delete event end fd=58 mask=3 original mask is 0 2017-12-24 18:53:36.548851 7f01acba2700 2 osd.0 1080 ms_handle_reset con 0x55825d5c1800 session 0x558234545680 2017-12-24 18:53:36.548915 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:36.548921 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:42.651442 7f01a330c700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).send_keepalive 2017-12-24 18:53:42.651488 7f01a330c700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:42.651510 7f01a330c700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x558207c2a4f0 pending 1 2017-12-24 18:53:42.651528 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:42.651534 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x558207c2a4f0 2017-12-24 18:53:42.651537 7f01b6317700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).handle_write 2017-12-24 18:53:42.651548 7f01b6317700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1)._append_keepalive_or_ack 2017-12-24 18:53:42.651641 7f01b6317700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1)._try_send sent bytes 9 remaining bytes 0 2017-12-24 18:53:42.651670 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:42.651673 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:42.651678 7f01b6317700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:42.651682 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:42.651691 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got 9 2017-12-24 18:53:42.651695 7f01b6317700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN_KEEPALIVE2_ACK pgs=74 cs=1 l=1).process prev state is STATE_OPEN 2017-12-24 18:53:42.651699 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN_KEEPALIVE2_ACK pgs=74 cs=1 l=1).read_until len is 8 state_offset is 0 2017-12-24 18:53:42.651704 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN_KEEPALIVE2_ACK pgs=74 cs=1 l=1).read_until got 8 in buffer left is 0 buffer still has 0 2017-12-24 18:53:42.651709 7f01b6317700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN_KEEPALIVE2_ACK pgs=74 cs=1 l=1).process got KEEPALIVE_ACK 2017-12-24 18:53:42.651712 7f01b6317700 20 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).process prev state is STATE_OPEN_KEEPALIVE2_ACK 2017-12-24 18:53:42.651715 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).read_until len is 1 state_offset is 0 2017-12-24 18:53:42.651719 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got 0 2017-12-24 18:53:42.651723 7f01b6317700 25 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).read_until need len 1 remaining 1 bytes 2017-12-24 18:53:42.651728 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 70 mask is 1 2017-12-24 18:53:42.651730 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:42.651732 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:51.602978 7f01932ec700 1 stupidalloc shutdown 2017-12-24 18:53:51.648317 7f01932ec700 1 freelist shutdown 2017-12-24 18:53:51.648724 7f01932ec700 4 rocksdb: [/build/ceph-12.2.1/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work 2017-12-24 18:53:51.665005 7f01932ec700 4 rocksdb: [/build/ceph-12.2.1/src/rocksdb/db/db_impl.cc:343] Shutdown complete 2017-12-24 18:53:51.902440 7f01932ec700 1 bluefs umount 2017-12-24 18:53:51.902769 7f01932ec700 1 stupidalloc shutdown 2017-12-24 18:53:51.902850 7f01932ec700 1 bdev(0x55811816cd00 /var/lib/ceph/osd/ceph-0/block) close 2017-12-24 18:53:52.205574 7f01932ec700 1 bdev(0x55811816bb00 /var/lib/ceph/osd/ceph-0/block) close 2017-12-24 18:53:52.318516 7f01932ec700 10 osd.0 1080 Store synced 2017-12-24 18:53:52.318538 7f01932ec700 1 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).mark_down 2017-12-24 18:53:52.318550 7f01932ec700 2 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1)._stop 2017-12-24 18:53:52.318556 7f01932ec700 10 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6789/0 conn(0x558273349800 :-1 s=STATE_OPEN pgs=74 cs=1 l=1).discard_out_queue started 2017-12-24 18:53:52.318565 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.318574 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x558118166940 pending 1 2017-12-24 18:53:52.318592 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.318601 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x558118166940 2017-12-24 18:53:52.318604 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_time_event id=390544 2017-12-24 18:53:52.318608 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event started fd=70 mask=3 original mask is 1 2017-12-24 18:53:52.318612 7f01b6317700 20 EpollDriver.del_event del event fd=70 cur_mask=1 delmask=3 to 7 2017-12-24 18:53:52.318617 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).delete_file_event delete event end fd=70 mask=3 original mask is 0 2017-12-24 18:53:52.318667 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.318672 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.345488 7f01932ec700 10 -- 10.103.0.1:6801/6033 shutdown 10.103.0.1:6801/6033 2017-12-24 18:53:52.345512 7f01932ec700 10 Processor -- stop 2017-12-24 18:53:52.345516 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.345537 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9e00 pending 1 2017-12-24 18:53:52.345555 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.345564 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9e00 2017-12-24 18:53:52.345581 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event started fd=13 mask=1 original mask is 1 2017-12-24 18:53:52.345593 7f01b6b18700 20 EpollDriver.del_event del event fd=13 cur_mask=1 delmask=1 to 4 2017-12-24 18:53:52.345598 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event end fd=13 mask=1 original mask is 0 2017-12-24 18:53:52.345619 7f01932ec700 1 -- 10.103.0.1:6801/6033 shutdown_connections 2017-12-24 18:53:52.345626 7f01932ec700 5 -- 10.103.0.1:6801/6033 shutdown_connections mark down 10.103.0.1:0/1098863877 0x55825d5c1800 2017-12-24 18:53:52.345633 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.345635 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.345636 7f01932ec700 5 -- 10.103.0.1:6801/6033 shutdown_connections mark down 10.103.0.1:6789/0 0x558273349800 2017-12-24 18:53:52.345640 7f01932ec700 5 -- 10.103.0.1:6801/6033 shutdown_connections delete 0x55825d5c1800 2017-12-24 18:53:52.345684 7f01932ec700 5 -- 10.103.0.1:6801/6033 shutdown_connections delete 0x558273349800 2017-12-24 18:53:52.345695 7f01932ec700 30 stack drain started. 2017-12-24 18:53:52.345699 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.345704 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.345706 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.345713 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.345714 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.345716 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.345718 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.345721 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.345722 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.345727 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.345726 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.345736 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.345749 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.345752 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.345755 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.345763 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.345773 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.345776 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.345777 7f01932ec700 30 stack drain end. 2017-12-24 18:53:52.345782 7f01932ec700 10 -- 10.104.0.1:6800/6033 shutdown 10.104.0.1:6800/6033 2017-12-24 18:53:52.345788 7f01932ec700 10 Processor -- stop 2017-12-24 18:53:52.345790 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.345796 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9e00 pending 1 2017-12-24 18:53:52.345802 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.345807 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9e00 2017-12-24 18:53:52.345808 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event started fd=14 mask=1 original mask is 1 2017-12-24 18:53:52.345810 7f01b6b18700 20 EpollDriver.del_event del event fd=14 cur_mask=1 delmask=1 to 4 2017-12-24 18:53:52.345814 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event end fd=14 mask=1 original mask is 0 2017-12-24 18:53:52.345824 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.345826 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.345828 7f01932ec700 1 -- 10.104.0.1:6800/6033 shutdown_connections 2017-12-24 18:53:52.345835 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections accepting_conn 0x5581356b0800 2017-12-24 18:53:52.345839 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections accepting_conn 0x55813b76d000 2017-12-24 18:53:52.345844 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections mark down 10.104.0.2:6800/5930 0x55815ae63000 2017-12-24 18:53:52.345849 7f01932ec700 2 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_STANDBY pgs=4 cs=1 l=0)._stop 2017-12-24 18:53:52.345855 7f01932ec700 10 -- 10.104.0.1:6800/6033 >> 10.104.0.2:6800/5930 conn(0x55815ae63000 :-1 s=STATE_STANDBY pgs=4 cs=1 l=0).discard_out_queue started 2017-12-24 18:53:52.345870 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.345875 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x558249c4e7b0 pending 1 2017-12-24 18:53:52.345881 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.345885 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x558249c4e7b0 2017-12-24 18:53:52.345884 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections mark down 10.104.0.3:6800/5420 0x55813b754000 2017-12-24 18:53:52.345889 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.345892 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.345891 7f01932ec700 2 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_STANDBY pgs=6 cs=1 l=0)._stop 2017-12-24 18:53:52.345897 7f01932ec700 10 -- 10.104.0.1:6800/6033 >> 10.104.0.3:6800/5420 conn(0x55813b754000 :-1 s=STATE_STANDBY pgs=6 cs=1 l=0).discard_out_queue started 2017-12-24 18:53:52.345903 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.345902 7f01abba0700 2 osd.0 0 ms_handle_reset con 0x55815ae63000 session 0x55816177c280 2017-12-24 18:53:52.345908 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x5581181666b0 pending 1 2017-12-24 18:53:52.345911 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.345915 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x5581181666b0 2017-12-24 18:53:52.345917 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.345918 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.345928 7f01abba0700 2 osd.0 0 ms_handle_reset con 0x55813b754000 session 0x55813572ef00 2017-12-24 18:53:52.345916 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections mark down 10.104.0.4:6800/5071 0x5581356c4000 2017-12-24 18:53:52.345932 7f01932ec700 2 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_STANDBY pgs=9 cs=1 l=0)._stop 2017-12-24 18:53:52.345937 7f01932ec700 10 -- 10.104.0.1:6800/6033 >> 10.104.0.4:6800/5071 conn(0x5581356c4000 :-1 s=STATE_STANDBY pgs=9 cs=1 l=0).discard_out_queue started 2017-12-24 18:53:52.345941 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.345946 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x558118166380 pending 1 2017-12-24 18:53:52.345949 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x5581c55ced70 pending 2 2017-12-24 18:53:52.345952 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.345953 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections delete 0x5581356b0800 2017-12-24 18:53:52.345958 7f01abba0700 2 osd.0 0 ms_handle_reset con 0x5581356c4000 session 0x558135643680 2017-12-24 18:53:52.345960 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x558118166380 2017-12-24 18:53:52.345962 7f01b6317700 1 -- 10.104.0.1:6800/6033 reap_dead start 2017-12-24 18:53:52.345949 7f01bbe38e00 10 -- 10.103.0.1:6801/6033 wait: waiting for dispatch queue 2017-12-24 18:53:52.346663 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections delete 0x5581356c4000 2017-12-24 18:53:52.346680 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections delete 0x55813b754000 2017-12-24 18:53:52.346921 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections delete 0x55813b76d000 2017-12-24 18:53:52.347131 7f01932ec700 5 -- 10.104.0.1:6800/6033 shutdown_connections delete 0x55815ae63000 2017-12-24 18:53:52.347510 7f01932ec700 30 stack drain started. 2017-12-24 18:53:52.347516 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x5581c55ced70 2017-12-24 18:53:52.347522 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.347533 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347536 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.347542 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347544 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.347543 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.347550 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347551 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347555 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.347557 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.347560 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.347563 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 0 usec 0 2017-12-24 18:53:52.347560 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.347566 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347569 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.347570 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.347570 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.347573 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347579 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.347580 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.347585 7f01932ec700 30 stack drain end. 2017-12-24 18:53:52.347590 7f01932ec700 10 -- 10.103.0.1:0/6033 shutdown 10.103.0.1:0/6033 2017-12-24 18:53:52.347599 7f01932ec700 10 Processor -- stop 2017-12-24 18:53:52.347600 7f01932ec700 1 -- 10.103.0.1:0/6033 shutdown_connections 2017-12-24 18:53:52.347607 7f01932ec700 5 -- 10.103.0.1:0/6033 shutdown_connections mark down 10.103.0.4:6801/5071 0x5581356ac000 2017-12-24 18:53:52.347615 7f01932ec700 5 -- 10.103.0.1:0/6033 shutdown_connections mark down 10.103.0.2:6802/5930 0x5581356af000 2017-12-24 18:53:52.347621 7f01932ec700 5 -- 10.103.0.1:0/6033 shutdown_connections mark down 10.103.0.3:6802/5420 0x5581356c7000 2017-12-24 18:53:52.347625 7f01932ec700 5 -- 10.103.0.1:0/6033 shutdown_connections delete 0x5581356ac000 2017-12-24 18:53:52.347636 7f01932ec700 5 -- 10.103.0.1:0/6033 shutdown_connections delete 0x5581356af000 2017-12-24 18:53:52.347646 7f01932ec700 5 -- 10.103.0.1:0/6033 shutdown_connections delete 0x5581356c7000 2017-12-24 18:53:52.347654 7f01932ec700 30 stack drain started. 2017-12-24 18:53:52.347656 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.347662 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347664 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.347668 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347670 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.347671 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.347674 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347674 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.347680 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347680 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.347679 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347682 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.347684 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347685 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.347685 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.347687 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.347689 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.347690 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.347696 7f01932ec700 30 stack drain end. 2017-12-24 18:53:52.347699 7f01932ec700 10 -- 10.104.0.1:0/6033 shutdown 10.104.0.1:0/6033 2017-12-24 18:53:52.347703 7f01932ec700 10 Processor -- stop 2017-12-24 18:53:52.347704 7f01932ec700 1 -- 10.104.0.1:0/6033 shutdown_connections 2017-12-24 18:53:52.347707 7f01932ec700 5 -- 10.104.0.1:0/6033 shutdown_connections mark down 10.104.0.2:6801/5930 0x5581356ad800 2017-12-24 18:53:52.347711 7f01932ec700 5 -- 10.104.0.1:0/6033 shutdown_connections mark down 10.104.0.4:6801/5071 0x5581356c8800 2017-12-24 18:53:52.347717 7f01932ec700 5 -- 10.104.0.1:0/6033 shutdown_connections mark down 10.104.0.3:6801/5420 0x5581356c5800 2017-12-24 18:53:52.347725 7f01932ec700 5 -- 10.104.0.1:0/6033 shutdown_connections delete 0x5581356ad800 2017-12-24 18:53:52.347747 7f01932ec700 5 -- 10.104.0.1:0/6033 shutdown_connections delete 0x5581356c5800 2017-12-24 18:53:52.347821 7f01932ec700 5 -- 10.104.0.1:0/6033 shutdown_connections delete 0x5581356c8800 2017-12-24 18:53:52.347950 7f01932ec700 30 stack drain started. 2017-12-24 18:53:52.347956 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.347963 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347966 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.347972 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347970 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.347975 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.347977 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347980 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.347980 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.347984 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.347981 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.347990 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347992 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.347990 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.347995 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.347995 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.348000 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.348002 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.348005 7f01932ec700 30 stack drain end. 2017-12-24 18:53:52.348010 7f01932ec700 10 -- - shutdown - 2017-12-24 18:53:52.348011 7f01932ec700 10 Processor -- stop 2017-12-24 18:53:52.348013 7f01932ec700 1 -- - shutdown_connections 2017-12-24 18:53:52.348016 7f01932ec700 30 stack drain started. 2017-12-24 18:53:52.348019 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.348023 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348031 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.348035 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348038 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.348040 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.348050 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.348054 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348058 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.348060 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.348067 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.348068 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348063 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348071 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348072 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.348073 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.348076 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.348079 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.348081 7f01932ec700 30 stack drain end. 2017-12-24 18:53:52.348085 7f01932ec700 10 -- 10.103.0.1:6802/6033 shutdown 10.103.0.1:6802/6033 2017-12-24 18:53:52.348091 7f01932ec700 10 Processor -- stop 2017-12-24 18:53:52.348093 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.348098 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9e00 pending 1 2017-12-24 18:53:52.348105 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.348107 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9e00 2017-12-24 18:53:52.348109 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event started fd=16 mask=1 original mask is 1 2017-12-24 18:53:52.348112 7f01b6b18700 20 EpollDriver.del_event del event fd=16 cur_mask=1 delmask=1 to 4 2017-12-24 18:53:52.348114 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event end fd=16 mask=1 original mask is 0 2017-12-24 18:53:52.348131 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.348133 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.348134 7f01932ec700 1 -- 10.103.0.1:6802/6033 shutdown_connections 2017-12-24 18:53:52.348153 7f01932ec700 5 -- 10.103.0.1:6802/6033 shutdown_connections mark down 10.103.0.2:0/5930 0x55813bee6000 2017-12-24 18:53:52.348160 7f01932ec700 5 -- 10.103.0.1:6802/6033 shutdown_connections mark down 10.103.0.3:0/5420 0x5581356b9000 2017-12-24 18:53:52.348165 7f01932ec700 5 -- 10.103.0.1:6802/6033 shutdown_connections mark down 10.103.0.4:0/5071 0x55813571b000 2017-12-24 18:53:52.348170 7f01932ec700 5 -- 10.103.0.1:6802/6033 shutdown_connections delete 0x5581356b9000 2017-12-24 18:53:52.348182 7f01932ec700 5 -- 10.103.0.1:6802/6033 shutdown_connections delete 0x55813571b000 2017-12-24 18:53:52.348195 7f01932ec700 5 -- 10.103.0.1:6802/6033 shutdown_connections delete 0x55813bee6000 2017-12-24 18:53:52.348206 7f01932ec700 30 stack drain started. 2017-12-24 18:53:52.348209 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.348215 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348217 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.348220 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.348226 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.348222 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348228 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348231 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348234 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.348234 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.348234 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.348237 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.348238 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.348241 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348248 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.348252 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348256 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.348258 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.348260 7f01932ec700 30 stack drain end. 2017-12-24 18:53:52.348264 7f01932ec700 10 -- 10.104.0.1:6801/6033 shutdown 10.104.0.1:6801/6033 2017-12-24 18:53:52.348269 7f01932ec700 10 Processor -- stop 2017-12-24 18:53:52.348270 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.348275 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9e00 pending 1 2017-12-24 18:53:52.348280 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.348284 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9e00 2017-12-24 18:53:52.348286 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event started fd=15 mask=1 original mask is 1 2017-12-24 18:53:52.348287 7f01b6b18700 20 EpollDriver.del_event del event fd=15 cur_mask=1 delmask=1 to 4 2017-12-24 18:53:52.348291 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).delete_file_event delete event end fd=15 mask=1 original mask is 0 2017-12-24 18:53:52.348301 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.348302 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.348304 7f01932ec700 1 -- 10.104.0.1:6801/6033 shutdown_connections 2017-12-24 18:53:52.348311 7f01932ec700 5 -- 10.104.0.1:6801/6033 shutdown_connections mark down 10.104.0.2:0/5930 0x55813bee7800 2017-12-24 18:53:52.348315 7f01932ec700 5 -- 10.104.0.1:6801/6033 shutdown_connections mark down 10.104.0.3:0/5420 0x5581356ba800 2017-12-24 18:53:52.348320 7f01932ec700 5 -- 10.104.0.1:6801/6033 shutdown_connections mark down 10.104.0.4:0/5071 0x5581356b6000 2017-12-24 18:53:52.348324 7f01932ec700 5 -- 10.104.0.1:6801/6033 shutdown_connections delete 0x5581356b6000 2017-12-24 18:53:52.348333 7f01932ec700 5 -- 10.104.0.1:6801/6033 shutdown_connections delete 0x5581356ba800 2017-12-24 18:53:52.348343 7f01932ec700 5 -- 10.104.0.1:6801/6033 shutdown_connections delete 0x55813bee7800 2017-12-24 18:53:52.348352 7f01932ec700 30 stack drain started. 2017-12-24 18:53:52.348355 7f01932ec700 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.348360 7f01932ec700 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348362 7f01932ec700 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.348365 7f01932ec700 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348367 7f01932ec700 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.348371 7f01932ec700 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7f01932e9d30 pending 1 2017-12-24 18:53:52.348371 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.348375 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348376 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.348376 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.348380 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348380 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.348382 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.348383 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.348396 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.348402 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7f01932e9d30 2017-12-24 18:53:52.348408 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.348411 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.348412 7f01932ec700 30 stack drain end. 2017-12-24 18:53:52.355669 7f01bbe38e00 10 -- 10.103.0.1:6801/6033 wait: dispatch queue is stopped 2017-12-24 18:53:52.355682 7f01bbe38e00 1 -- 10.103.0.1:6801/6033 shutdown_connections 2017-12-24 18:53:52.355686 7f01bbe38e00 30 stack drain started. 2017-12-24 18:53:52.359060 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.359073 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.359076 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.359080 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.359087 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.359083 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.359090 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.359091 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.359093 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.359090 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.359097 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.359097 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.359100 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.359102 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.359101 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.359110 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.359117 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.359119 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.359121 7f01bbe38e00 30 stack drain end. 2017-12-24 18:53:52.360165 7f01bbe38e00 10 -- 10.103.0.1:6801/6033 wait: done. 2017-12-24 18:53:52.360176 7f01bbe38e00 1 -- 10.103.0.1:6801/6033 wait complete. 2017-12-24 18:53:52.360186 7f01bbe38e00 10 -- 10.103.0.1:0/6033 wait: waiting for dispatch queue 2017-12-24 18:53:52.361296 7f01bbe38e00 10 -- 10.103.0.1:0/6033 wait: dispatch queue is stopped 2017-12-24 18:53:52.361306 7f01bbe38e00 1 -- 10.103.0.1:0/6033 shutdown_connections 2017-12-24 18:53:52.361310 7f01bbe38e00 30 stack drain started. 2017-12-24 18:53:52.361313 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.361320 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361322 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.361325 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361327 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.361329 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.361334 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361334 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361332 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.361336 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.361337 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.361340 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361337 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.361342 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.361342 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361344 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.361347 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.361348 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.361349 7f01bbe38e00 30 stack drain end. 2017-12-24 18:53:52.361351 7f01bbe38e00 10 -- 10.103.0.1:0/6033 wait: done. 2017-12-24 18:53:52.361353 7f01bbe38e00 1 -- 10.103.0.1:0/6033 wait complete. 2017-12-24 18:53:52.361361 7f01bbe38e00 10 -- 10.104.0.1:0/6033 wait: waiting for dispatch queue 2017-12-24 18:53:52.361576 7f01bbe38e00 10 -- 10.104.0.1:0/6033 wait: dispatch queue is stopped 2017-12-24 18:53:52.361584 7f01bbe38e00 1 -- 10.104.0.1:0/6033 shutdown_connections 2017-12-24 18:53:52.361587 7f01bbe38e00 30 stack drain started. 2017-12-24 18:53:52.361590 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.361596 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361597 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.361601 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361602 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.361601 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.361606 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361606 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361605 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.361608 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.361609 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361611 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.361612 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.361612 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.361612 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.361617 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361621 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.361623 7f01bbe38e00 30 stack drain end. 2017-12-24 18:53:52.361623 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.361625 7f01bbe38e00 10 -- 10.104.0.1:0/6033 wait: done. 2017-12-24 18:53:52.361628 7f01bbe38e00 1 -- 10.104.0.1:0/6033 wait complete. 2017-12-24 18:53:52.361636 7f01bbe38e00 10 -- 10.103.0.1:6802/6033 wait: waiting for dispatch queue 2017-12-24 18:53:52.361831 7f01bbe38e00 10 -- 10.103.0.1:6802/6033 wait: dispatch queue is stopped 2017-12-24 18:53:52.361840 7f01bbe38e00 1 -- 10.103.0.1:6802/6033 shutdown_connections 2017-12-24 18:53:52.361843 7f01bbe38e00 30 stack drain started. 2017-12-24 18:53:52.361846 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.361852 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361854 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.361858 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361859 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.361860 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.361864 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.361865 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361868 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.361869 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.361872 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.361872 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.361878 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361881 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.361881 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.361882 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.361889 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.361891 7f01bbe38e00 30 stack drain end. 2017-12-24 18:53:52.361891 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.361893 7f01bbe38e00 10 -- 10.103.0.1:6802/6033 wait: done. 2017-12-24 18:53:52.361896 7f01bbe38e00 1 -- 10.103.0.1:6802/6033 wait complete. 2017-12-24 18:53:52.361904 7f01bbe38e00 10 -- 10.104.0.1:6801/6033 wait: waiting for dispatch queue 2017-12-24 18:53:52.362191 7f01bbe38e00 10 -- 10.104.0.1:6801/6033 wait: dispatch queue is stopped 2017-12-24 18:53:52.362199 7f01bbe38e00 1 -- 10.104.0.1:6801/6033 shutdown_connections 2017-12-24 18:53:52.362202 7f01bbe38e00 30 stack drain started. 2017-12-24 18:53:52.362205 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.362210 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.362212 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.362214 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.362215 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.362217 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.362217 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.362219 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.362220 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.362220 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.362221 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.362225 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.362227 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.362228 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.362229 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.362234 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.362241 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.362242 7f01bbe38e00 30 stack drain end. 2017-12-24 18:53:52.362244 7f01bbe38e00 10 -- 10.104.0.1:6801/6033 wait: done. 2017-12-24 18:53:52.362246 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.362247 7f01bbe38e00 1 -- 10.104.0.1:6801/6033 wait complete. 2017-12-24 18:53:52.362259 7f01bbe38e00 10 -- 10.104.0.1:6800/6033 wait: waiting for dispatch queue 2017-12-24 18:53:52.365670 7f01bbe38e00 10 -- 10.104.0.1:6800/6033 wait: dispatch queue is stopped 2017-12-24 18:53:52.365680 7f01bbe38e00 1 -- 10.104.0.1:6800/6033 shutdown_connections 2017-12-24 18:53:52.365687 7f01bbe38e00 30 stack drain started. 2017-12-24 18:53:52.365690 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.365696 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.365698 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.365701 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.365703 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.365702 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.365706 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.365707 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.365710 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.365711 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.365709 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.365714 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.365713 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.365717 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.365718 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.365718 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.365725 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.365726 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.365727 7f01bbe38e00 30 stack drain end. 2017-12-24 18:53:52.365729 7f01bbe38e00 10 -- 10.104.0.1:6800/6033 wait: done. 2017-12-24 18:53:52.365732 7f01bbe38e00 1 -- 10.104.0.1:6800/6033 wait complete. 2017-12-24 18:53:52.365740 7f01bbe38e00 10 -- - wait: waiting for dispatch queue 2017-12-24 18:53:52.365978 7f01bbe38e00 10 -- - wait: dispatch queue is stopped 2017-12-24 18:53:52.365981 7f01bbe38e00 1 -- - shutdown_connections 2017-12-24 18:53:52.365984 7f01bbe38e00 30 stack drain started. 2017-12-24 18:53:52.365986 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.365995 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.365997 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.366002 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.366005 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.366010 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x7fff93d88b40 pending 1 2017-12-24 18:53:52.366009 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.366018 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.366022 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.366021 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.366024 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.366026 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.366030 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.366031 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.366199 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.366204 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x7fff93d88b40 2017-12-24 18:53:52.366208 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.366209 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.366211 7f01bbe38e00 30 stack drain end. 2017-12-24 18:53:52.366214 7f01bbe38e00 10 -- - wait: done. 2017-12-24 18:53:52.366215 7f01bbe38e00 1 -- - wait complete. 2017-12-24 18:53:52.370895 7f01bbe38e00 1 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6801/6033 conn(0x558118476000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down 2017-12-24 18:53:52.370909 7f01bbe38e00 2 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6801/6033 conn(0x558118476000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop 2017-12-24 18:53:52.370915 7f01bbe38e00 10 -- 10.103.0.1:6801/6033 >> 10.103.0.1:6801/6033 conn(0x558118476000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started 2017-12-24 18:53:52.370921 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.370928 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x558118166310 pending 1 2017-12-24 18:53:52.370934 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.370939 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x558118166310 2017-12-24 18:53:52.370945 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.370946 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.370946 7f01bbe38e00 1 -- 10.103.0.1:0/6033 >> 10.103.0.1:0/6033 conn(0x55811847a800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down 2017-12-24 18:53:52.370951 7f01bbe38e00 2 -- 10.103.0.1:0/6033 >> 10.103.0.1:0/6033 conn(0x55811847a800 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop 2017-12-24 18:53:52.370954 7f01bbe38e00 10 -- 10.103.0.1:0/6033 >> 10.103.0.1:0/6033 conn(0x55811847a800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started 2017-12-24 18:53:52.370958 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.370962 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x558118166440 pending 1 2017-12-24 18:53:52.370965 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.370968 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x558118166440 2017-12-24 18:53:52.370974 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.370975 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.370979 7f01bbe38e00 1 -- 10.104.0.1:0/6033 >> 10.104.0.1:0/6033 conn(0x558118479000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down 2017-12-24 18:53:52.370984 7f01bbe38e00 2 -- 10.104.0.1:0/6033 >> 10.104.0.1:0/6033 conn(0x558118479000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop 2017-12-24 18:53:52.370987 7f01bbe38e00 10 -- 10.104.0.1:0/6033 >> 10.104.0.1:0/6033 conn(0x558118479000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started 2017-12-24 18:53:52.370991 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.370995 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x5581181663e0 pending 1 2017-12-24 18:53:52.371004 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.371008 7f01bbe38e00 1 -- 10.103.0.1:6802/6033 >> 10.103.0.1:6802/6033 conn(0x55811851b800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down 2017-12-24 18:53:52.371010 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x5581181663e0 2017-12-24 18:53:52.371012 7f01bbe38e00 2 -- 10.103.0.1:6802/6033 >> 10.103.0.1:6802/6033 conn(0x55811851b800 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop 2017-12-24 18:53:52.371017 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.371016 7f01bbe38e00 10 -- 10.103.0.1:6802/6033 >> 10.103.0.1:6802/6033 conn(0x55811851b800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started 2017-12-24 18:53:52.371018 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.371020 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.371024 7f01bbe38e00 30 Event(0x55811816a980 nevent=5000 time_id=188674).dispatch_event_external 0x558118166500 pending 1 2017-12-24 18:53:52.371029 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.371032 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events do 0x558118166500 2017-12-24 18:53:52.371036 7f01b5b16700 30 stack operator() calling event process 2017-12-24 18:53:52.371038 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events wait second 30 usec 0 2017-12-24 18:53:52.371036 7f01bbe38e00 1 -- 10.104.0.1:6801/6033 >> 10.104.0.1:6801/6033 conn(0x55811851a000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down 2017-12-24 18:53:52.371040 7f01bbe38e00 2 -- 10.104.0.1:6801/6033 >> 10.104.0.1:6801/6033 conn(0x55811851a000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop 2017-12-24 18:53:52.371043 7f01bbe38e00 10 -- 10.104.0.1:6801/6033 >> 10.104.0.1:6801/6033 conn(0x55811851a000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started 2017-12-24 18:53:52.371047 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.371052 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x5581181664a0 pending 1 2017-12-24 18:53:52.371059 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.371062 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x5581181664a0 2017-12-24 18:53:52.371062 7f01bbe38e00 1 -- 10.104.0.1:6800/6033 >> 10.104.0.1:6800/6033 conn(0x558118477800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down 2017-12-24 18:53:52.371067 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.371066 7f01bbe38e00 2 -- 10.104.0.1:6800/6033 >> 10.104.0.1:6800/6033 conn(0x558118477800 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop 2017-12-24 18:53:52.371068 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.371070 7f01bbe38e00 10 -- 10.104.0.1:6800/6033 >> 10.104.0.1:6800/6033 conn(0x558118477800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started 2017-12-24 18:53:52.371077 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.371081 7f01bbe38e00 30 Event(0x55811816a740 nevent=5000 time_id=390552).dispatch_event_external 0x558118166380 pending 1 2017-12-24 18:53:52.371084 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.371086 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events do 0x558118166380 2017-12-24 18:53:52.371090 7f01b6317700 30 stack operator() calling event process 2017-12-24 18:53:52.371091 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events wait second 30 usec 0 2017-12-24 18:53:52.371092 7f01bbe38e00 1 -- - >> - conn(0x55811851d000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down 2017-12-24 18:53:52.371095 7f01bbe38e00 2 -- - >> - conn(0x55811851d000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop 2017-12-24 18:53:52.371097 7f01bbe38e00 10 -- - >> - conn(0x55811851d000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).discard_out_queue started 2017-12-24 18:53:52.371100 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.371103 7f01bbe38e00 30 Event(0x55811816a080 nevent=5000 time_id=135262).dispatch_event_external 0x558118166560 pending 1 2017-12-24 18:53:52.371107 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.371110 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events do 0x558118166560 2017-12-24 18:53:52.371115 7f01b6b18700 30 stack operator() calling event process 2017-12-24 18:53:52.371116 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events wait second 30 usec 0 2017-12-24 18:53:52.371338 7f01bbe38e00 20 Event(0x55811816a080 nevent=5000 time_id=135262).wakeup 2017-12-24 18:53:52.371350 7f01b6b18700 30 Event(0x55811816a080 nevent=5000 time_id=135262).process_events event_wq process is 5 mask is 1 2017-12-24 18:53:52.376976 7f01bbe38e00 20 Event(0x55811816a740 nevent=5000 time_id=390552).wakeup 2017-12-24 18:53:52.376997 7f01b6317700 30 Event(0x55811816a740 nevent=5000 time_id=390552).process_events event_wq process is 8 mask is 1 2017-12-24 18:53:52.388523 7f01bbe38e00 20 Event(0x55811816a980 nevent=5000 time_id=188674).wakeup 2017-12-24 18:53:52.388555 7f01b5b16700 30 Event(0x55811816a980 nevent=5000 time_id=188674).process_events event_wq process is 11 mask is 1 2017-12-24 18:53:52.784293 7fa6baf27e00 0 set uid:gid to 64045:64045 (ceph:ceph) 2017-12-24 18:53:52.784309 7fa6baf27e00 0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 140237 2017-12-24 18:53:52.791368 7fa6baf27e00 0 pidfile_write: ignore empty --pid-file 2017-12-24 18:53:52.798562 7fa6baf27e00 0 load: jerasure load: lrc load: isa 2017-12-24 18:53:52.798623 7fa6baf27e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2017-12-24 18:53:52.798633 7fa6baf27e00 1 bdev(0x55e8041a2b40 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2017-12-24 18:53:52.798876 7fa6baf27e00 1 bdev(0x55e8041a2b40 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational 2017-12-24 18:53:52.799091 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 2017-12-24 18:53:52.799112 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 2017-12-24 18:53:52.799117 7fa6baf27e00 1 bdev(0x55e8041a2b40 /var/lib/ceph/osd/ceph-0/block) close 2017-12-24 18:53:53.094013 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0 2017-12-24 18:53:53.094387 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck repair (deep) start 2017-12-24 18:53:53.094411 7fa6baf27e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2017-12-24 18:53:53.094417 7fa6baf27e00 1 bdev(0x55e8041a2d80 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2017-12-24 18:53:53.094600 7fa6baf27e00 1 bdev(0x55e8041a2d80 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational 2017-12-24 18:53:53.094804 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 2017-12-24 18:53:53.094877 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 2017-12-24 18:53:53.095305 7fa6baf27e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2017-12-24 18:53:53.095313 7fa6baf27e00 1 bdev(0x55e8041a2fc0 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2017-12-24 18:53:53.095449 7fa6baf27e00 1 bdev(0x55e8041a2fc0 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational 2017-12-24 18:53:53.095456 7fa6baf27e00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 1682 GB 2017-12-24 18:53:53.095504 7fa6baf27e00 1 bluefs mount 2017-12-24 18:53:53.155328 7fa6baf27e00 0 set rocksdb option compaction_readahead_size = 2097152 2017-12-24 18:53:53.155343 7fa6baf27e00 0 set rocksdb option compression = kNoCompression 2017-12-24 18:53:53.155348 7fa6baf27e00 0 set rocksdb option max_write_buffer_number = 4 2017-12-24 18:53:53.155352 7fa6baf27e00 0 set rocksdb option min_write_buffer_number_to_merge = 1 2017-12-24 18:53:53.155354 7fa6baf27e00 0 set rocksdb option recycle_log_file_num = 4 2017-12-24 18:53:53.155357 7fa6baf27e00 0 set rocksdb option writable_file_max_buffer_size = 0 2017-12-24 18:53:53.155361 7fa6baf27e00 0 set rocksdb option write_buffer_size = 268435456 2017-12-24 18:53:53.155376 7fa6baf27e00 0 set rocksdb option compaction_readahead_size = 2097152 2017-12-24 18:53:53.155380 7fa6baf27e00 0 set rocksdb option compression = kNoCompression 2017-12-24 18:53:53.155383 7fa6baf27e00 0 set rocksdb option max_write_buffer_number = 4 2017-12-24 18:53:53.155386 7fa6baf27e00 0 set rocksdb option min_write_buffer_number_to_merge = 1 2017-12-24 18:53:53.155388 7fa6baf27e00 0 set rocksdb option recycle_log_file_num = 4 2017-12-24 18:53:53.155391 7fa6baf27e00 0 set rocksdb option writable_file_max_buffer_size = 0 2017-12-24 18:53:53.155393 7fa6baf27e00 0 set rocksdb option write_buffer_size = 268435456 2017-12-24 18:53:53.155494 7fa6baf27e00 4 rocksdb: RocksDB version: 5.4.0 2017-12-24 18:53:53.155503 7fa6baf27e00 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2017-12-24 18:53:53.155509 7fa6baf27e00 4 rocksdb: Compile date Nov 30 2017 2017-12-24 18:53:53.155511 7fa6baf27e00 4 rocksdb: DB SUMMARY 2017-12-24 18:53:53.155534 7fa6baf27e00 4 rocksdb: CURRENT file: CURRENT 2017-12-24 18:53:53.155537 7fa6baf27e00 4 rocksdb: IDENTITY file: IDENTITY 2017-12-24 18:53:53.155539 7fa6baf27e00 4 rocksdb: MANIFEST file: MANIFEST-000431 size: 931685 Bytes 2017-12-24 18:53:53.155541 7fa6baf27e00 4 rocksdb: SST files in db dir, Total Num: 33, files: 005983.sst 005984.sst 005985.sst 006019.sst 006020.sst 006021.sst 009951.sst 009952.sst 009953.sst 2017-12-24 18:53:53.155543 7fa6baf27e00 4 rocksdb: Write Ahead Log file in db: 009948.log size: 243855567 ; 009977.log size: 126296612 ; 2017-12-24 18:53:53.155544 7fa6baf27e00 4 rocksdb: Options.error_if_exists: 0 2017-12-24 18:53:53.155545 7fa6baf27e00 4 rocksdb: Options.create_if_missing: 0 2017-12-24 18:53:53.155546 7fa6baf27e00 4 rocksdb: Options.paranoid_checks: 1 2017-12-24 18:53:53.155546 7fa6baf27e00 4 rocksdb: Options.env: 0x55e804684160 2017-12-24 18:53:53.155547 7fa6baf27e00 4 rocksdb: Options.info_log: 0x55e8046849c0 2017-12-24 18:53:53.155548 7fa6baf27e00 4 rocksdb: Options.max_open_files: -1 2017-12-24 18:53:53.155550 7fa6baf27e00 4 rocksdb: Options.max_file_opening_threads: 16 2017-12-24 18:53:53.155550 7fa6baf27e00 4 rocksdb: Options.use_fsync: 0 2017-12-24 18:53:53.155551 7fa6baf27e00 4 rocksdb: Options.max_log_file_size: 0 2017-12-24 18:53:53.155551 7fa6baf27e00 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2017-12-24 18:53:53.155553 7fa6baf27e00 4 rocksdb: Options.log_file_time_to_roll: 0 2017-12-24 18:53:53.155554 7fa6baf27e00 4 rocksdb: Options.keep_log_file_num: 1000 2017-12-24 18:53:53.155555 7fa6baf27e00 4 rocksdb: Options.recycle_log_file_num: 4 2017-12-24 18:53:53.155556 7fa6baf27e00 4 rocksdb: Options.allow_fallocate: 1 2017-12-24 18:53:53.155556 7fa6baf27e00 4 rocksdb: Options.allow_mmap_reads: 0 2017-12-24 18:53:53.155557 7fa6baf27e00 4 rocksdb: Options.allow_mmap_writes: 0 2017-12-24 18:53:53.155557 7fa6baf27e00 4 rocksdb: Options.use_direct_reads: 0 2017-12-24 18:53:53.155558 7fa6baf27e00 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2017-12-24 18:53:53.155559 7fa6baf27e00 4 rocksdb: Options.create_missing_column_families: 0 2017-12-24 18:53:53.155559 7fa6baf27e00 4 rocksdb: Options.db_log_dir: 2017-12-24 18:53:53.155560 7fa6baf27e00 4 rocksdb: Options.wal_dir: db 2017-12-24 18:53:53.155561 7fa6baf27e00 4 rocksdb: Options.table_cache_numshardbits: 6 2017-12-24 18:53:53.155561 7fa6baf27e00 4 rocksdb: Options.max_subcompactions: 1 2017-12-24 18:53:53.155562 7fa6baf27e00 4 rocksdb: Options.max_background_flushes: 1 2017-12-24 18:53:53.155563 7fa6baf27e00 4 rocksdb: Options.WAL_ttl_seconds: 0 2017-12-24 18:53:53.155563 7fa6baf27e00 4 rocksdb: Options.WAL_size_limit_MB: 0 2017-12-24 18:53:53.155564 7fa6baf27e00 4 rocksdb: Options.manifest_preallocation_size: 4194304 2017-12-24 18:53:53.155564 7fa6baf27e00 4 rocksdb: Options.is_fd_close_on_exec: 1 2017-12-24 18:53:53.155565 7fa6baf27e00 4 rocksdb: Options.advise_random_on_open: 1 2017-12-24 18:53:53.155565 7fa6baf27e00 4 rocksdb: Options.db_write_buffer_size: 0 2017-12-24 18:53:53.155566 7fa6baf27e00 4 rocksdb: Options.access_hint_on_compaction_start: 1 2017-12-24 18:53:53.155567 7fa6baf27e00 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2017-12-24 18:53:53.155567 7fa6baf27e00 4 rocksdb: Options.compaction_readahead_size: 2097152 2017-12-24 18:53:53.155568 7fa6baf27e00 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2017-12-24 18:53:53.155568 7fa6baf27e00 4 rocksdb: Options.writable_file_max_buffer_size: 0 2017-12-24 18:53:53.155569 7fa6baf27e00 4 rocksdb: Options.use_adaptive_mutex: 0 2017-12-24 18:53:53.155570 7fa6baf27e00 4 rocksdb: Options.rate_limiter: (nil) 2017-12-24 18:53:53.155571 7fa6baf27e00 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2017-12-24 18:53:53.155571 7fa6baf27e00 4 rocksdb: Options.bytes_per_sync: 0 2017-12-24 18:53:53.155572 7fa6baf27e00 4 rocksdb: Options.wal_bytes_per_sync: 0 2017-12-24 18:53:53.155572 7fa6baf27e00 4 rocksdb: Options.wal_recovery_mode: 0 2017-12-24 18:53:53.155573 7fa6baf27e00 4 rocksdb: Options.enable_thread_tracking: 0 2017-12-24 18:53:53.155574 7fa6baf27e00 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2017-12-24 18:53:53.155574 7fa6baf27e00 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2017-12-24 18:53:53.155575 7fa6baf27e00 4 rocksdb: Options.write_thread_max_yield_usec: 100 2017-12-24 18:53:53.155576 7fa6baf27e00 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2017-12-24 18:53:53.155577 7fa6baf27e00 4 rocksdb: Options.row_cache: None 2017-12-24 18:53:53.155577 7fa6baf27e00 4 rocksdb: Options.wal_filter: None 2017-12-24 18:53:53.155578 7fa6baf27e00 4 rocksdb: Options.avoid_flush_during_recovery: 0 2017-12-24 18:53:53.155579 7fa6baf27e00 4 rocksdb: Options.base_background_compactions: 1 2017-12-24 18:53:53.155579 7fa6baf27e00 4 rocksdb: Options.max_background_compactions: 1 2017-12-24 18:53:53.155580 7fa6baf27e00 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2017-12-24 18:53:53.155581 7fa6baf27e00 4 rocksdb: Options.delayed_write_rate : 16777216 2017-12-24 18:53:53.155582 7fa6baf27e00 4 rocksdb: Options.max_total_wal_size: 0 2017-12-24 18:53:53.155582 7fa6baf27e00 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2017-12-24 18:53:53.155583 7fa6baf27e00 4 rocksdb: Options.stats_dump_period_sec: 600 2017-12-24 18:53:53.155584 7fa6baf27e00 4 rocksdb: Compression algorithms supported: 2017-12-24 18:53:53.155584 7fa6baf27e00 4 rocksdb: Snappy supported: 0 2017-12-24 18:53:53.155585 7fa6baf27e00 4 rocksdb: Zlib supported: 0 2017-12-24 18:53:53.155586 7fa6baf27e00 4 rocksdb: Bzip supported: 0 2017-12-24 18:53:53.155586 7fa6baf27e00 4 rocksdb: LZ4 supported: 0 2017-12-24 18:53:53.155592 7fa6baf27e00 4 rocksdb: ZSTD supported: 0 2017-12-24 18:53:53.155593 7fa6baf27e00 4 rocksdb: Fast CRC32 supported: 0 2017-12-24 18:53:53.155755 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000431 2017-12-24 18:53:53.155784 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: 2017-12-24 18:53:53.155789 7fa6baf27e00 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2017-12-24 18:53:53.155794 7fa6baf27e00 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2017-12-24 18:53:53.155795 7fa6baf27e00 4 rocksdb: Options.compaction_filter: None 2017-12-24 18:53:53.155796 7fa6baf27e00 4 rocksdb: Options.compaction_filter_factory: None 2017-12-24 18:53:53.155797 7fa6baf27e00 4 rocksdb: Options.memtable_factory: SkipListFactory 2017-12-24 18:53:53.155797 7fa6baf27e00 4 rocksdb: Options.table_factory: BlockBasedTable 2017-12-24 18:53:53.155826 7fa6baf27e00 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e80419a1f0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55e804489570 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 2017-12-24 18:53:53.155832 7fa6baf27e00 4 rocksdb: Options.write_buffer_size: 268435456 2017-12-24 18:53:53.155834 7fa6baf27e00 4 rocksdb: Options.max_write_buffer_number: 4 2017-12-24 18:53:53.155836 7fa6baf27e00 4 rocksdb: Options.compression: NoCompression 2017-12-24 18:53:53.155838 7fa6baf27e00 4 rocksdb: Options.bottommost_compression: Disabled 2017-12-24 18:53:53.155839 7fa6baf27e00 4 rocksdb: Options.prefix_extractor: nullptr 2017-12-24 18:53:53.155841 7fa6baf27e00 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2017-12-24 18:53:53.155842 7fa6baf27e00 4 rocksdb: Options.num_levels: 7 2017-12-24 18:53:53.155844 7fa6baf27e00 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2017-12-24 18:53:53.155845 7fa6baf27e00 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2017-12-24 18:53:53.155847 7fa6baf27e00 4 rocksdb: Options.compression_opts.window_bits: -14 2017-12-24 18:53:53.155849 7fa6baf27e00 4 rocksdb: Options.compression_opts.level: -1 2017-12-24 18:53:53.155851 7fa6baf27e00 4 rocksdb: Options.compression_opts.strategy: 0 2017-12-24 18:53:53.155853 7fa6baf27e00 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2017-12-24 18:53:53.155854 7fa6baf27e00 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2017-12-24 18:53:53.155855 7fa6baf27e00 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2017-12-24 18:53:53.155857 7fa6baf27e00 4 rocksdb: Options.level0_stop_writes_trigger: 36 2017-12-24 18:53:53.155858 7fa6baf27e00 4 rocksdb: Options.target_file_size_base: 67108864 2017-12-24 18:53:53.155860 7fa6baf27e00 4 rocksdb: Options.target_file_size_multiplier: 1 2017-12-24 18:53:53.155861 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2017-12-24 18:53:53.155862 7fa6baf27e00 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2017-12-24 18:53:53.155864 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2017-12-24 18:53:53.155868 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2017-12-24 18:53:53.155870 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2017-12-24 18:53:53.155872 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2017-12-24 18:53:53.155873 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2017-12-24 18:53:53.155874 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2017-12-24 18:53:53.155876 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2017-12-24 18:53:53.155877 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2017-12-24 18:53:53.155879 7fa6baf27e00 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2017-12-24 18:53:53.155881 7fa6baf27e00 4 rocksdb: Options.max_compaction_bytes: 1677721600 2017-12-24 18:53:53.155883 7fa6baf27e00 4 rocksdb: Options.arena_block_size: 33554432 2017-12-24 18:53:53.155884 7fa6baf27e00 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2017-12-24 18:53:53.155885 7fa6baf27e00 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2017-12-24 18:53:53.155886 7fa6baf27e00 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2017-12-24 18:53:53.155888 7fa6baf27e00 4 rocksdb: Options.disable_auto_compactions: 0 2017-12-24 18:53:53.155890 7fa6baf27e00 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2017-12-24 18:53:53.155893 7fa6baf27e00 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2017-12-24 18:53:53.155894 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2017-12-24 18:53:53.155905 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2017-12-24 18:53:53.155906 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2017-12-24 18:53:53.155907 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2017-12-24 18:53:53.155908 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2017-12-24 18:53:53.155909 7fa6baf27e00 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2017-12-24 18:53:53.155910 7fa6baf27e00 4 rocksdb: Options.table_properties_collectors: 2017-12-24 18:53:53.155911 7fa6baf27e00 4 rocksdb: Options.inplace_update_support: 0 2017-12-24 18:53:53.155912 7fa6baf27e00 4 rocksdb: Options.inplace_update_num_locks: 10000 2017-12-24 18:53:53.155913 7fa6baf27e00 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2017-12-24 18:53:53.155915 7fa6baf27e00 4 rocksdb: Options.memtable_huge_page_size: 0 2017-12-24 18:53:53.155916 7fa6baf27e00 4 rocksdb: Options.bloom_locality: 0 2017-12-24 18:53:53.155917 7fa6baf27e00 4 rocksdb: Options.max_successive_merges: 0 2017-12-24 18:53:53.155918 7fa6baf27e00 4 rocksdb: Options.optimize_filters_for_hits: 0 2017-12-24 18:53:53.155918 7fa6baf27e00 4 rocksdb: Options.paranoid_file_checks: 0 2017-12-24 18:53:53.155919 7fa6baf27e00 4 rocksdb: Options.force_consistency_checks: 0 2017-12-24 18:53:53.155920 7fa6baf27e00 4 rocksdb: Options.report_bg_io_stats: 0 2017-12-24 18:53:53.170550 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000431 succeeded,manifest_file_number is 431, next_file_number is 9980, last_sequence is 1115432710, log_number is 0,prev_log_number is 0,max_column_family is 0 2017-12-24 18:53:53.170568 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 9977 2017-12-24 18:53:53.170790 7fa6baf27e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514123633170783, "job": 1, "event": "recovery_started", "log_files": [9948, 9977]} 2017-12-24 18:53:53.170796 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #9948 mode 0 2017-12-24 18:53:54.286692 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #9977 mode 0 2017-12-24 18:53:56.536626 7fa6baf27e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514123636536603, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 9980, "file_size": 5837670, "table_properties": {"data_size": 5555152, "index_size": 85920, "filter_size": 195609, "raw_key_size": 3756923, "raw_average_key_size": 51, "raw_value_size": 4326265, "raw_average_value_size": 59, "num_data_blocks": 1385, "num_entries": 73214, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "39853", "kMergeOperands": "2841"}} 2017-12-24 18:53:56.537735 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 9981 2017-12-24 18:53:56.538547 7fa6baf27e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514123636538543, "job": 1, "event": "recovery_finished"} 2017-12-24 18:53:56.539838 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55e8044a2100 2017-12-24 18:53:56.539956 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2017-12-24 18:53:56.577620 7fa6baf27e00 1 freelist init 2017-12-24 18:53:56.589384 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc opening allocation metadata 2017-12-24 18:54:07.082059 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc loaded 587 G in 1173199 extents 2017-12-24 18:54:07.158729 7fa6baf27e00 1 bluefs fsck 2017-12-24 18:54:07.158744 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck walking object keyspace 2017-12-24 19:40:08.939488 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking shared_blobs 2017-12-24 19:40:08.940645 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking for stray omap data 2017-12-24 19:40:14.739627 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking deferred events 2017-12-24 19:40:14.744003 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking freelist vs allocated 2017-12-24 19:40:26.217813 7fa6baf27e00 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_fsck(bool, bool)' thread 7fa6baf27e00 time 2017-12-24 19:40:26.215431 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 6122: FAILED assert(used_blocks.size() > count) ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e7f9f7e892] 2: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55e7f9e41c8c] 3: (BlueStore::_mount(bool)+0x20a) [0x55e7f9e4284a] 4: (OSD::init()+0x3e2) [0x55e7f99ae4e2] 5: (main()+0x2f07) [0x55e7f98c01d7] 6: (__libc_start_main()+0xf0) [0x7fa6b838f830] 7: (_start()+0x29) [0x55e7f994b7f9] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -214> 2017-12-24 18:53:52.777856 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command perfcounters_dump hook 0x55e80419e1c0 -213> 2017-12-24 18:53:52.777880 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command 1 hook 0x55e80419e1c0 -212> 2017-12-24 18:53:52.777884 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command perf dump hook 0x55e80419e1c0 -211> 2017-12-24 18:53:52.777887 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command perfcounters_schema hook 0x55e80419e1c0 -210> 2017-12-24 18:53:52.777892 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command perf histogram dump hook 0x55e80419e1c0 -209> 2017-12-24 18:53:52.777894 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command 2 hook 0x55e80419e1c0 -208> 2017-12-24 18:53:52.777898 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command perf schema hook 0x55e80419e1c0 -207> 2017-12-24 18:53:52.777900 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command perf histogram schema hook 0x55e80419e1c0 -206> 2017-12-24 18:53:52.777904 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command perf reset hook 0x55e80419e1c0 -205> 2017-12-24 18:53:52.777908 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command config show hook 0x55e80419e1c0 -204> 2017-12-24 18:53:52.777911 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command config help hook 0x55e80419e1c0 -203> 2017-12-24 18:53:52.777916 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command config set hook 0x55e80419e1c0 -202> 2017-12-24 18:53:52.777918 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command config get hook 0x55e80419e1c0 -201> 2017-12-24 18:53:52.777922 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command config diff hook 0x55e80419e1c0 -200> 2017-12-24 18:53:52.777925 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command config diff get hook 0x55e80419e1c0 -199> 2017-12-24 18:53:52.777930 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command log flush hook 0x55e80419e1c0 -198> 2017-12-24 18:53:52.777932 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command log dump hook 0x55e80419e1c0 -197> 2017-12-24 18:53:52.777936 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command log reopen hook 0x55e80419e1c0 -196> 2017-12-24 18:53:52.777945 7fa6baf27e00 5 asok(0x55e8041ef2c0) register_command dump_mempools hook 0x55e804452d68 -195> 2017-12-24 18:53:52.784293 7fa6baf27e00 0 set uid:gid to 64045:64045 (ceph:ceph) -194> 2017-12-24 18:53:52.784309 7fa6baf27e00 0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 140237 -193> 2017-12-24 18:53:52.791368 7fa6baf27e00 0 pidfile_write: ignore empty --pid-file -192> 2017-12-24 18:53:52.798562 7fa6baf27e00 0 load: jerasure load: lrc load: isa -191> 2017-12-24 18:53:52.798623 7fa6baf27e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel -190> 2017-12-24 18:53:52.798633 7fa6baf27e00 1 bdev(0x55e8041a2b40 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block -189> 2017-12-24 18:53:52.798876 7fa6baf27e00 1 bdev(0x55e8041a2b40 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational -188> 2017-12-24 18:53:52.799091 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 -187> 2017-12-24 18:53:52.799112 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 -186> 2017-12-24 18:53:52.799117 7fa6baf27e00 1 bdev(0x55e8041a2b40 /var/lib/ceph/osd/ceph-0/block) close -185> 2017-12-24 18:53:53.094013 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0 -184> 2017-12-24 18:53:53.094387 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck repair (deep) start -183> 2017-12-24 18:53:53.094411 7fa6baf27e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel -182> 2017-12-24 18:53:53.094417 7fa6baf27e00 1 bdev(0x55e8041a2d80 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block -181> 2017-12-24 18:53:53.094600 7fa6baf27e00 1 bdev(0x55e8041a2d80 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational -180> 2017-12-24 18:53:53.094804 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 -179> 2017-12-24 18:53:53.094877 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 -178> 2017-12-24 18:53:53.095305 7fa6baf27e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel -177> 2017-12-24 18:53:53.095313 7fa6baf27e00 1 bdev(0x55e8041a2fc0 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block -176> 2017-12-24 18:53:53.095449 7fa6baf27e00 1 bdev(0x55e8041a2fc0 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational -175> 2017-12-24 18:53:53.095456 7fa6baf27e00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 1682 GB -174> 2017-12-24 18:53:53.095504 7fa6baf27e00 1 bluefs mount -173> 2017-12-24 18:53:53.155328 7fa6baf27e00 0 set rocksdb option compaction_readahead_size = 2097152 -172> 2017-12-24 18:53:53.155343 7fa6baf27e00 0 set rocksdb option compression = kNoCompression -171> 2017-12-24 18:53:53.155348 7fa6baf27e00 0 set rocksdb option max_write_buffer_number = 4 -170> 2017-12-24 18:53:53.155352 7fa6baf27e00 0 set rocksdb option min_write_buffer_number_to_merge = 1 -169> 2017-12-24 18:53:53.155354 7fa6baf27e00 0 set rocksdb option recycle_log_file_num = 4 -168> 2017-12-24 18:53:53.155357 7fa6baf27e00 0 set rocksdb option writable_file_max_buffer_size = 0 -167> 2017-12-24 18:53:53.155361 7fa6baf27e00 0 set rocksdb option write_buffer_size = 268435456 -166> 2017-12-24 18:53:53.155376 7fa6baf27e00 0 set rocksdb option compaction_readahead_size = 2097152 -165> 2017-12-24 18:53:53.155380 7fa6baf27e00 0 set rocksdb option compression = kNoCompression -164> 2017-12-24 18:53:53.155383 7fa6baf27e00 0 set rocksdb option max_write_buffer_number = 4 -163> 2017-12-24 18:53:53.155386 7fa6baf27e00 0 set rocksdb option min_write_buffer_number_to_merge = 1 -162> 2017-12-24 18:53:53.155388 7fa6baf27e00 0 set rocksdb option recycle_log_file_num = 4 -161> 2017-12-24 18:53:53.155391 7fa6baf27e00 0 set rocksdb option writable_file_max_buffer_size = 0 -160> 2017-12-24 18:53:53.155393 7fa6baf27e00 0 set rocksdb option write_buffer_size = 268435456 -159> 2017-12-24 18:53:53.155494 7fa6baf27e00 4 rocksdb: RocksDB version: 5.4.0 -158> 2017-12-24 18:53:53.155503 7fa6baf27e00 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -157> 2017-12-24 18:53:53.155509 7fa6baf27e00 4 rocksdb: Compile date Nov 30 2017 -156> 2017-12-24 18:53:53.155511 7fa6baf27e00 4 rocksdb: DB SUMMARY -155> 2017-12-24 18:53:53.155534 7fa6baf27e00 4 rocksdb: CURRENT file: CURRENT -154> 2017-12-24 18:53:53.155537 7fa6baf27e00 4 rocksdb: IDENTITY file: IDENTITY -153> 2017-12-24 18:53:53.155539 7fa6baf27e00 4 rocksdb: MANIFEST file: MANIFEST-000431 size: 931685 Bytes -152> 2017-12-24 18:53:53.155541 7fa6baf27e00 4 rocksdb: SST files in db dir, Total Num: 33, files: 005983.sst 005984.sst 005985.sst 006019.sst 006020.sst 006021.sst 009951.sst 009952.sst 009953.sst -151> 2017-12-24 18:53:53.155543 7fa6baf27e00 4 rocksdb: Write Ahead Log file in db: 009948.log size: 243855567 ; 009977.log size: 126296612 ; -150> 2017-12-24 18:53:53.155544 7fa6baf27e00 4 rocksdb: Options.error_if_exists: 0 -149> 2017-12-24 18:53:53.155545 7fa6baf27e00 4 rocksdb: Options.create_if_missing: 0 -148> 2017-12-24 18:53:53.155546 7fa6baf27e00 4 rocksdb: Options.paranoid_checks: 1 -147> 2017-12-24 18:53:53.155546 7fa6baf27e00 4 rocksdb: Options.env: 0x55e804684160 -146> 2017-12-24 18:53:53.155547 7fa6baf27e00 4 rocksdb: Options.info_log: 0x55e8046849c0 -145> 2017-12-24 18:53:53.155548 7fa6baf27e00 4 rocksdb: Options.max_open_files: -1 -144> 2017-12-24 18:53:53.155550 7fa6baf27e00 4 rocksdb: Options.max_file_opening_threads: 16 -143> 2017-12-24 18:53:53.155550 7fa6baf27e00 4 rocksdb: Options.use_fsync: 0 -142> 2017-12-24 18:53:53.155551 7fa6baf27e00 4 rocksdb: Options.max_log_file_size: 0 -141> 2017-12-24 18:53:53.155551 7fa6baf27e00 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -140> 2017-12-24 18:53:53.155553 7fa6baf27e00 4 rocksdb: Options.log_file_time_to_roll: 0 -139> 2017-12-24 18:53:53.155554 7fa6baf27e00 4 rocksdb: Options.keep_log_file_num: 1000 -138> 2017-12-24 18:53:53.155555 7fa6baf27e00 4 rocksdb: Options.recycle_log_file_num: 4 -137> 2017-12-24 18:53:53.155556 7fa6baf27e00 4 rocksdb: Options.allow_fallocate: 1 -136> 2017-12-24 18:53:53.155556 7fa6baf27e00 4 rocksdb: Options.allow_mmap_reads: 0 -135> 2017-12-24 18:53:53.155557 7fa6baf27e00 4 rocksdb: Options.allow_mmap_writes: 0 -134> 2017-12-24 18:53:53.155557 7fa6baf27e00 4 rocksdb: Options.use_direct_reads: 0 -133> 2017-12-24 18:53:53.155558 7fa6baf27e00 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -132> 2017-12-24 18:53:53.155559 7fa6baf27e00 4 rocksdb: Options.create_missing_column_families: 0 -131> 2017-12-24 18:53:53.155559 7fa6baf27e00 4 rocksdb: Options.db_log_dir: -130> 2017-12-24 18:53:53.155560 7fa6baf27e00 4 rocksdb: Options.wal_dir: db -129> 2017-12-24 18:53:53.155561 7fa6baf27e00 4 rocksdb: Options.table_cache_numshardbits: 6 -128> 2017-12-24 18:53:53.155561 7fa6baf27e00 4 rocksdb: Options.max_subcompactions: 1 -127> 2017-12-24 18:53:53.155562 7fa6baf27e00 4 rocksdb: Options.max_background_flushes: 1 -126> 2017-12-24 18:53:53.155563 7fa6baf27e00 4 rocksdb: Options.WAL_ttl_seconds: 0 -125> 2017-12-24 18:53:53.155563 7fa6baf27e00 4 rocksdb: Options.WAL_size_limit_MB: 0 -124> 2017-12-24 18:53:53.155564 7fa6baf27e00 4 rocksdb: Options.manifest_preallocation_size: 4194304 -123> 2017-12-24 18:53:53.155564 7fa6baf27e00 4 rocksdb: Options.is_fd_close_on_exec: 1 -122> 2017-12-24 18:53:53.155565 7fa6baf27e00 4 rocksdb: Options.advise_random_on_open: 1 -121> 2017-12-24 18:53:53.155565 7fa6baf27e00 4 rocksdb: Options.db_write_buffer_size: 0 -120> 2017-12-24 18:53:53.155566 7fa6baf27e00 4 rocksdb: Options.access_hint_on_compaction_start: 1 -119> 2017-12-24 18:53:53.155567 7fa6baf27e00 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -118> 2017-12-24 18:53:53.155567 7fa6baf27e00 4 rocksdb: Options.compaction_readahead_size: 2097152 -117> 2017-12-24 18:53:53.155568 7fa6baf27e00 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -116> 2017-12-24 18:53:53.155568 7fa6baf27e00 4 rocksdb: Options.writable_file_max_buffer_size: 0 -115> 2017-12-24 18:53:53.155569 7fa6baf27e00 4 rocksdb: Options.use_adaptive_mutex: 0 -114> 2017-12-24 18:53:53.155570 7fa6baf27e00 4 rocksdb: Options.rate_limiter: (nil) -113> 2017-12-24 18:53:53.155571 7fa6baf27e00 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -112> 2017-12-24 18:53:53.155571 7fa6baf27e00 4 rocksdb: Options.bytes_per_sync: 0 -111> 2017-12-24 18:53:53.155572 7fa6baf27e00 4 rocksdb: Options.wal_bytes_per_sync: 0 -110> 2017-12-24 18:53:53.155572 7fa6baf27e00 4 rocksdb: Options.wal_recovery_mode: 0 -109> 2017-12-24 18:53:53.155573 7fa6baf27e00 4 rocksdb: Options.enable_thread_tracking: 0 -108> 2017-12-24 18:53:53.155574 7fa6baf27e00 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -107> 2017-12-24 18:53:53.155574 7fa6baf27e00 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -106> 2017-12-24 18:53:53.155575 7fa6baf27e00 4 rocksdb: Options.write_thread_max_yield_usec: 100 -105> 2017-12-24 18:53:53.155576 7fa6baf27e00 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -104> 2017-12-24 18:53:53.155577 7fa6baf27e00 4 rocksdb: Options.row_cache: None -103> 2017-12-24 18:53:53.155577 7fa6baf27e00 4 rocksdb: Options.wal_filter: None -102> 2017-12-24 18:53:53.155578 7fa6baf27e00 4 rocksdb: Options.avoid_flush_during_recovery: 0 -101> 2017-12-24 18:53:53.155579 7fa6baf27e00 4 rocksdb: Options.base_background_compactions: 1 -100> 2017-12-24 18:53:53.155579 7fa6baf27e00 4 rocksdb: Options.max_background_compactions: 1 -99> 2017-12-24 18:53:53.155580 7fa6baf27e00 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -98> 2017-12-24 18:53:53.155581 7fa6baf27e00 4 rocksdb: Options.delayed_write_rate : 16777216 -97> 2017-12-24 18:53:53.155582 7fa6baf27e00 4 rocksdb: Options.max_total_wal_size: 0 -96> 2017-12-24 18:53:53.155582 7fa6baf27e00 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -95> 2017-12-24 18:53:53.155583 7fa6baf27e00 4 rocksdb: Options.stats_dump_period_sec: 600 -94> 2017-12-24 18:53:53.155584 7fa6baf27e00 4 rocksdb: Compression algorithms supported: -93> 2017-12-24 18:53:53.155584 7fa6baf27e00 4 rocksdb: Snappy supported: 0 -92> 2017-12-24 18:53:53.155585 7fa6baf27e00 4 rocksdb: Zlib supported: 0 -91> 2017-12-24 18:53:53.155586 7fa6baf27e00 4 rocksdb: Bzip supported: 0 -90> 2017-12-24 18:53:53.155586 7fa6baf27e00 4 rocksdb: LZ4 supported: 0 -89> 2017-12-24 18:53:53.155592 7fa6baf27e00 4 rocksdb: ZSTD supported: 0 -88> 2017-12-24 18:53:53.155593 7fa6baf27e00 4 rocksdb: Fast CRC32 supported: 0 -87> 2017-12-24 18:53:53.155755 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000431 -86> 2017-12-24 18:53:53.155784 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: -85> 2017-12-24 18:53:53.155789 7fa6baf27e00 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -84> 2017-12-24 18:53:53.155794 7fa6baf27e00 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -83> 2017-12-24 18:53:53.155795 7fa6baf27e00 4 rocksdb: Options.compaction_filter: None -82> 2017-12-24 18:53:53.155796 7fa6baf27e00 4 rocksdb: Options.compaction_filter_factory: None -81> 2017-12-24 18:53:53.155797 7fa6baf27e00 4 rocksdb: Options.memtable_factory: SkipListFactory -80> 2017-12-24 18:53:53.155797 7fa6baf27e00 4 rocksdb: Options.table_factory: BlockBasedTable -79> 2017-12-24 18:53:53.155826 7fa6baf27e00 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e80419a1f0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55e804489570 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 -78> 2017-12-24 18:53:53.155832 7fa6baf27e00 4 rocksdb: Options.write_buffer_size: 268435456 -77> 2017-12-24 18:53:53.155834 7fa6baf27e00 4 rocksdb: Options.max_write_buffer_number: 4 -76> 2017-12-24 18:53:53.155836 7fa6baf27e00 4 rocksdb: Options.compression: NoCompression -75> 2017-12-24 18:53:53.155838 7fa6baf27e00 4 rocksdb: Options.bottommost_compression: Disabled -74> 2017-12-24 18:53:53.155839 7fa6baf27e00 4 rocksdb: Options.prefix_extractor: nullptr -73> 2017-12-24 18:53:53.155841 7fa6baf27e00 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -72> 2017-12-24 18:53:53.155842 7fa6baf27e00 4 rocksdb: Options.num_levels: 7 -71> 2017-12-24 18:53:53.155844 7fa6baf27e00 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -70> 2017-12-24 18:53:53.155845 7fa6baf27e00 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -69> 2017-12-24 18:53:53.155847 7fa6baf27e00 4 rocksdb: Options.compression_opts.window_bits: -14 -68> 2017-12-24 18:53:53.155849 7fa6baf27e00 4 rocksdb: Options.compression_opts.level: -1 -67> 2017-12-24 18:53:53.155851 7fa6baf27e00 4 rocksdb: Options.compression_opts.strategy: 0 -66> 2017-12-24 18:53:53.155853 7fa6baf27e00 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -65> 2017-12-24 18:53:53.155854 7fa6baf27e00 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -64> 2017-12-24 18:53:53.155855 7fa6baf27e00 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -63> 2017-12-24 18:53:53.155857 7fa6baf27e00 4 rocksdb: Options.level0_stop_writes_trigger: 36 -62> 2017-12-24 18:53:53.155858 7fa6baf27e00 4 rocksdb: Options.target_file_size_base: 67108864 -61> 2017-12-24 18:53:53.155860 7fa6baf27e00 4 rocksdb: Options.target_file_size_multiplier: 1 -60> 2017-12-24 18:53:53.155861 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -59> 2017-12-24 18:53:53.155862 7fa6baf27e00 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -58> 2017-12-24 18:53:53.155864 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -57> 2017-12-24 18:53:53.155868 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -56> 2017-12-24 18:53:53.155870 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -55> 2017-12-24 18:53:53.155872 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -54> 2017-12-24 18:53:53.155873 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -53> 2017-12-24 18:53:53.155874 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -52> 2017-12-24 18:53:53.155876 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -51> 2017-12-24 18:53:53.155877 7fa6baf27e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -50> 2017-12-24 18:53:53.155879 7fa6baf27e00 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -49> 2017-12-24 18:53:53.155881 7fa6baf27e00 4 rocksdb: Options.max_compaction_bytes: 1677721600 -48> 2017-12-24 18:53:53.155883 7fa6baf27e00 4 rocksdb: Options.arena_block_size: 33554432 -47> 2017-12-24 18:53:53.155884 7fa6baf27e00 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -46> 2017-12-24 18:53:53.155885 7fa6baf27e00 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -45> 2017-12-24 18:53:53.155886 7fa6baf27e00 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -44> 2017-12-24 18:53:53.155888 7fa6baf27e00 4 rocksdb: Options.disable_auto_compactions: 0 -43> 2017-12-24 18:53:53.155890 7fa6baf27e00 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -42> 2017-12-24 18:53:53.155893 7fa6baf27e00 4 rocksdb: Options.compaction_pri: kByCompensatedSize -41> 2017-12-24 18:53:53.155894 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -40> 2017-12-24 18:53:53.155905 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -39> 2017-12-24 18:53:53.155906 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -38> 2017-12-24 18:53:53.155907 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -37> 2017-12-24 18:53:53.155908 7fa6baf27e00 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -36> 2017-12-24 18:53:53.155909 7fa6baf27e00 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -35> 2017-12-24 18:53:53.155910 7fa6baf27e00 4 rocksdb: Options.table_properties_collectors: -34> 2017-12-24 18:53:53.155911 7fa6baf27e00 4 rocksdb: Options.inplace_update_support: 0 -33> 2017-12-24 18:53:53.155912 7fa6baf27e00 4 rocksdb: Options.inplace_update_num_locks: 10000 -32> 2017-12-24 18:53:53.155913 7fa6baf27e00 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -31> 2017-12-24 18:53:53.155915 7fa6baf27e00 4 rocksdb: Options.memtable_huge_page_size: 0 -30> 2017-12-24 18:53:53.155916 7fa6baf27e00 4 rocksdb: Options.bloom_locality: 0 -29> 2017-12-24 18:53:53.155917 7fa6baf27e00 4 rocksdb: Options.max_successive_merges: 0 -28> 2017-12-24 18:53:53.155918 7fa6baf27e00 4 rocksdb: Options.optimize_filters_for_hits: 0 -27> 2017-12-24 18:53:53.155918 7fa6baf27e00 4 rocksdb: Options.paranoid_file_checks: 0 -26> 2017-12-24 18:53:53.155919 7fa6baf27e00 4 rocksdb: Options.force_consistency_checks: 0 -25> 2017-12-24 18:53:53.155920 7fa6baf27e00 4 rocksdb: Options.report_bg_io_stats: 0 -24> 2017-12-24 18:53:53.170550 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000431 succeeded,manifest_file_number is 431, next_file_number is 9980, last_sequence is 1115432710, log_number is 0,prev_log_number is 0,max_column_family is 0 -23> 2017-12-24 18:53:53.170568 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 9977 -22> 2017-12-24 18:53:53.170790 7fa6baf27e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514123633170783, "job": 1, "event": "recovery_started", "log_files": [9948, 9977]} -21> 2017-12-24 18:53:53.170796 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #9948 mode 0 -20> 2017-12-24 18:53:54.286692 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #9977 mode 0 -19> 2017-12-24 18:53:56.396513 7fa6baf27e00 5 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:815] [default] [WriteLevel0TableForRecovery] Level-0 table #9980: started -18> 2017-12-24 18:53:56.536626 7fa6baf27e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514123636536603, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 9980, "file_size": 5837670, "table_properties": {"data_size": 5555152, "index_size": 85920, "filter_size": 195609, "raw_key_size": 3756923, "raw_average_key_size": 51, "raw_value_size": 4326265, "raw_average_value_size": 59, "num_data_blocks": 1385, "num_entries": 73214, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "39853", "kMergeOperands": "2841"}} -17> 2017-12-24 18:53:56.536639 7fa6baf27e00 5 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:847] [default] [WriteLevel0TableForRecovery] Level-0 table #9980: 5837670 bytes OK -16> 2017-12-24 18:53:56.537735 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 9981 -15> 2017-12-24 18:53:56.538547 7fa6baf27e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514123636538543, "job": 1, "event": "recovery_finished"} -14> 2017-12-24 18:53:56.538663 7fa6baf27e00 5 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete db//MANIFEST-000431 type=3 #431 -- OK -13> 2017-12-24 18:53:56.538677 7fa6baf27e00 5 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete db//009977.log type=0 #9977 -- OK -12> 2017-12-24 18:53:56.538689 7fa6baf27e00 5 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete db//009948.log type=0 #9948 -- OK -11> 2017-12-24 18:53:56.539838 7fa6baf27e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55e8044a2100 -10> 2017-12-24 18:53:56.539956 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -9> 2017-12-24 18:53:56.577620 7fa6baf27e00 1 freelist init -8> 2017-12-24 18:53:56.589384 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc opening allocation metadata -7> 2017-12-24 18:54:07.082059 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc loaded 587 G in 1173199 extents -6> 2017-12-24 18:54:07.158729 7fa6baf27e00 1 bluefs fsck -5> 2017-12-24 18:54:07.158744 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck walking object keyspace -4> 2017-12-24 19:40:08.939488 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking shared_blobs -3> 2017-12-24 19:40:08.940645 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking for stray omap data -2> 2017-12-24 19:40:14.739627 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking deferred events -1> 2017-12-24 19:40:14.744003 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking freelist vs allocated 0> 2017-12-24 19:40:26.217813 7fa6baf27e00 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_fsck(bool, bool)' thread 7fa6baf27e00 time 2017-12-24 19:40:26.215431 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 6122: FAILED assert(used_blocks.size() > count) ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e7f9f7e892] 2: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55e7f9e41c8c] 3: (BlueStore::_mount(bool)+0x20a) [0x55e7f9e4284a] 4: (OSD::init()+0x3e2) [0x55e7f99ae4e2] 5: (main()+0x2f07) [0x55e7f98c01d7] 6: (__libc_start_main()+0xf0) [0x7fa6b838f830] 7: (_start()+0x29) [0x55e7f994b7f9] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 1/ 1 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events --- 2017-12-24 19:40:26.221532 7fa6baf27e00 -1 *** Caught signal (Aborted) ** in thread 7fa6baf27e00 thread_name:ceph-osd ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (()+0xa65824) [0x55e7f9f3b824] 2: (()+0x11390) [0x7fa6b9409390] 3: (gsignal()+0x38) [0x7fa6b83a4428] 4: (abort()+0x16a) [0x7fa6b83a602a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55e7f9f7ea1e] 6: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55e7f9e41c8c] 7: (BlueStore::_mount(bool)+0x20a) [0x55e7f9e4284a] 8: (OSD::init()+0x3e2) [0x55e7f99ae4e2] 9: (main()+0x2f07) [0x55e7f98c01d7] 10: (__libc_start_main()+0xf0) [0x7fa6b838f830] 11: (_start()+0x29) [0x55e7f994b7f9] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2017-12-24 19:40:26.221532 7fa6baf27e00 -1 *** Caught signal (Aborted) ** in thread 7fa6baf27e00 thread_name:ceph-osd ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (()+0xa65824) [0x55e7f9f3b824] 2: (()+0x11390) [0x7fa6b9409390] 3: (gsignal()+0x38) [0x7fa6b83a4428] 4: (abort()+0x16a) [0x7fa6b83a602a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55e7f9f7ea1e] 6: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55e7f9e41c8c] 7: (BlueStore::_mount(bool)+0x20a) [0x55e7f9e4284a] 8: (OSD::init()+0x3e2) [0x55e7f99ae4e2] 9: (main()+0x2f07) [0x55e7f98c01d7] 10: (__libc_start_main()+0xf0) [0x7fa6b838f830] 11: (_start()+0x29) [0x55e7f994b7f9] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 1/ 1 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events --- 2017-12-24 19:40:47.107308 7faaab3d4e00 0 set uid:gid to 64045:64045 (ceph:ceph) 2017-12-24 19:40:47.107323 7faaab3d4e00 0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 60889 2017-12-24 19:40:47.114220 7faaab3d4e00 0 pidfile_write: ignore empty --pid-file 2017-12-24 19:40:47.120897 7faaab3d4e00 0 load: jerasure load: lrc load: isa 2017-12-24 19:40:47.120968 7faaab3d4e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2017-12-24 19:40:47.120978 7faaab3d4e00 1 bdev(0x559895cd8b40 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2017-12-24 19:40:47.121247 7faaab3d4e00 1 bdev(0x559895cd8b40 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational 2017-12-24 19:40:47.121498 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 2017-12-24 19:40:47.121520 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 2017-12-24 19:40:47.121526 7faaab3d4e00 1 bdev(0x559895cd8b40 /var/lib/ceph/osd/ceph-0/block) close 2017-12-24 19:40:47.422060 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0 2017-12-24 19:40:47.422496 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck repair (deep) start 2017-12-24 19:40:47.422523 7faaab3d4e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2017-12-24 19:40:47.422527 7faaab3d4e00 1 bdev(0x559895cd8d80 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2017-12-24 19:40:47.422719 7faaab3d4e00 1 bdev(0x559895cd8d80 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational 2017-12-24 19:40:47.422930 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 2017-12-24 19:40:47.422954 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 2017-12-24 19:40:47.423032 7faaab3d4e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2017-12-24 19:40:47.423039 7faaab3d4e00 1 bdev(0x559895cd8fc0 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2017-12-24 19:40:47.423192 7faaab3d4e00 1 bdev(0x559895cd8fc0 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational 2017-12-24 19:40:47.423201 7faaab3d4e00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 1682 GB 2017-12-24 19:40:47.423258 7faaab3d4e00 1 bluefs mount 2017-12-24 19:40:47.484696 7faaab3d4e00 0 set rocksdb option compaction_readahead_size = 2097152 2017-12-24 19:40:47.484715 7faaab3d4e00 0 set rocksdb option compression = kNoCompression 2017-12-24 19:40:47.484722 7faaab3d4e00 0 set rocksdb option max_write_buffer_number = 4 2017-12-24 19:40:47.484726 7faaab3d4e00 0 set rocksdb option min_write_buffer_number_to_merge = 1 2017-12-24 19:40:47.484729 7faaab3d4e00 0 set rocksdb option recycle_log_file_num = 4 2017-12-24 19:40:47.484732 7faaab3d4e00 0 set rocksdb option writable_file_max_buffer_size = 0 2017-12-24 19:40:47.484736 7faaab3d4e00 0 set rocksdb option write_buffer_size = 268435456 2017-12-24 19:40:47.484760 7faaab3d4e00 0 set rocksdb option compaction_readahead_size = 2097152 2017-12-24 19:40:47.484764 7faaab3d4e00 0 set rocksdb option compression = kNoCompression 2017-12-24 19:40:47.484768 7faaab3d4e00 0 set rocksdb option max_write_buffer_number = 4 2017-12-24 19:40:47.484771 7faaab3d4e00 0 set rocksdb option min_write_buffer_number_to_merge = 1 2017-12-24 19:40:47.484774 7faaab3d4e00 0 set rocksdb option recycle_log_file_num = 4 2017-12-24 19:40:47.484777 7faaab3d4e00 0 set rocksdb option writable_file_max_buffer_size = 0 2017-12-24 19:40:47.484781 7faaab3d4e00 0 set rocksdb option write_buffer_size = 268435456 2017-12-24 19:40:47.484888 7faaab3d4e00 4 rocksdb: RocksDB version: 5.4.0 2017-12-24 19:40:47.484898 7faaab3d4e00 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2017-12-24 19:40:47.484900 7faaab3d4e00 4 rocksdb: Compile date Nov 30 2017 2017-12-24 19:40:47.484902 7faaab3d4e00 4 rocksdb: DB SUMMARY 2017-12-24 19:40:47.484926 7faaab3d4e00 4 rocksdb: CURRENT file: CURRENT 2017-12-24 19:40:47.484928 7faaab3d4e00 4 rocksdb: IDENTITY file: IDENTITY 2017-12-24 19:40:47.484962 7faaab3d4e00 4 rocksdb: MANIFEST file: MANIFEST-009981 size: 5579 Bytes 2017-12-24 19:40:47.484970 7faaab3d4e00 4 rocksdb: SST files in db dir, Total Num: 34, files: 005983.sst 005984.sst 005985.sst 006019.sst 006020.sst 006021.sst 009951.sst 009952.sst 009953.sst 2017-12-24 19:40:47.484973 7faaab3d4e00 4 rocksdb: Write Ahead Log file in db: 009982.log size: 0 ; 2017-12-24 19:40:47.484974 7faaab3d4e00 4 rocksdb: Options.error_if_exists: 0 2017-12-24 19:40:47.484975 7faaab3d4e00 4 rocksdb: Options.create_if_missing: 0 2017-12-24 19:40:47.484976 7faaab3d4e00 4 rocksdb: Options.paranoid_checks: 1 2017-12-24 19:40:47.484976 7faaab3d4e00 4 rocksdb: Options.env: 0x5598961ba160 2017-12-24 19:40:47.484977 7faaab3d4e00 4 rocksdb: Options.info_log: 0x5598961ba9e0 2017-12-24 19:40:47.484978 7faaab3d4e00 4 rocksdb: Options.max_open_files: -1 2017-12-24 19:40:47.484979 7faaab3d4e00 4 rocksdb: Options.max_file_opening_threads: 16 2017-12-24 19:40:47.484979 7faaab3d4e00 4 rocksdb: Options.use_fsync: 0 2017-12-24 19:40:47.484980 7faaab3d4e00 4 rocksdb: Options.max_log_file_size: 0 2017-12-24 19:40:47.484981 7faaab3d4e00 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2017-12-24 19:40:47.484983 7faaab3d4e00 4 rocksdb: Options.log_file_time_to_roll: 0 2017-12-24 19:40:47.484984 7faaab3d4e00 4 rocksdb: Options.keep_log_file_num: 1000 2017-12-24 19:40:47.484985 7faaab3d4e00 4 rocksdb: Options.recycle_log_file_num: 4 2017-12-24 19:40:47.484986 7faaab3d4e00 4 rocksdb: Options.allow_fallocate: 1 2017-12-24 19:40:47.484987 7faaab3d4e00 4 rocksdb: Options.allow_mmap_reads: 0 2017-12-24 19:40:47.484988 7faaab3d4e00 4 rocksdb: Options.allow_mmap_writes: 0 2017-12-24 19:40:47.484989 7faaab3d4e00 4 rocksdb: Options.use_direct_reads: 0 2017-12-24 19:40:47.484989 7faaab3d4e00 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2017-12-24 19:40:47.484990 7faaab3d4e00 4 rocksdb: Options.create_missing_column_families: 0 2017-12-24 19:40:47.484991 7faaab3d4e00 4 rocksdb: Options.db_log_dir: 2017-12-24 19:40:47.484992 7faaab3d4e00 4 rocksdb: Options.wal_dir: db 2017-12-24 19:40:47.484992 7faaab3d4e00 4 rocksdb: Options.table_cache_numshardbits: 6 2017-12-24 19:40:47.484994 7faaab3d4e00 4 rocksdb: Options.max_subcompactions: 1 2017-12-24 19:40:47.484994 7faaab3d4e00 4 rocksdb: Options.max_background_flushes: 1 2017-12-24 19:40:47.484995 7faaab3d4e00 4 rocksdb: Options.WAL_ttl_seconds: 0 2017-12-24 19:40:47.484996 7faaab3d4e00 4 rocksdb: Options.WAL_size_limit_MB: 0 2017-12-24 19:40:47.484996 7faaab3d4e00 4 rocksdb: Options.manifest_preallocation_size: 4194304 2017-12-24 19:40:47.484997 7faaab3d4e00 4 rocksdb: Options.is_fd_close_on_exec: 1 2017-12-24 19:40:47.484998 7faaab3d4e00 4 rocksdb: Options.advise_random_on_open: 1 2017-12-24 19:40:47.484998 7faaab3d4e00 4 rocksdb: Options.db_write_buffer_size: 0 2017-12-24 19:40:47.484999 7faaab3d4e00 4 rocksdb: Options.access_hint_on_compaction_start: 1 2017-12-24 19:40:47.485000 7faaab3d4e00 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2017-12-24 19:40:47.485000 7faaab3d4e00 4 rocksdb: Options.compaction_readahead_size: 2097152 2017-12-24 19:40:47.485001 7faaab3d4e00 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2017-12-24 19:40:47.485001 7faaab3d4e00 4 rocksdb: Options.writable_file_max_buffer_size: 0 2017-12-24 19:40:47.485002 7faaab3d4e00 4 rocksdb: Options.use_adaptive_mutex: 0 2017-12-24 19:40:47.485003 7faaab3d4e00 4 rocksdb: Options.rate_limiter: (nil) 2017-12-24 19:40:47.485004 7faaab3d4e00 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2017-12-24 19:40:47.485004 7faaab3d4e00 4 rocksdb: Options.bytes_per_sync: 0 2017-12-24 19:40:47.485005 7faaab3d4e00 4 rocksdb: Options.wal_bytes_per_sync: 0 2017-12-24 19:40:47.485006 7faaab3d4e00 4 rocksdb: Options.wal_recovery_mode: 0 2017-12-24 19:40:47.485006 7faaab3d4e00 4 rocksdb: Options.enable_thread_tracking: 0 2017-12-24 19:40:47.485007 7faaab3d4e00 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2017-12-24 19:40:47.485008 7faaab3d4e00 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2017-12-24 19:40:47.485008 7faaab3d4e00 4 rocksdb: Options.write_thread_max_yield_usec: 100 2017-12-24 19:40:47.485009 7faaab3d4e00 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2017-12-24 19:40:47.485010 7faaab3d4e00 4 rocksdb: Options.row_cache: None 2017-12-24 19:40:47.485010 7faaab3d4e00 4 rocksdb: Options.wal_filter: None 2017-12-24 19:40:47.485011 7faaab3d4e00 4 rocksdb: Options.avoid_flush_during_recovery: 0 2017-12-24 19:40:47.485012 7faaab3d4e00 4 rocksdb: Options.base_background_compactions: 1 2017-12-24 19:40:47.485012 7faaab3d4e00 4 rocksdb: Options.max_background_compactions: 1 2017-12-24 19:40:47.485013 7faaab3d4e00 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2017-12-24 19:40:47.485014 7faaab3d4e00 4 rocksdb: Options.delayed_write_rate : 16777216 2017-12-24 19:40:47.485015 7faaab3d4e00 4 rocksdb: Options.max_total_wal_size: 0 2017-12-24 19:40:47.485016 7faaab3d4e00 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2017-12-24 19:40:47.485017 7faaab3d4e00 4 rocksdb: Options.stats_dump_period_sec: 600 2017-12-24 19:40:47.485056 7faaab3d4e00 4 rocksdb: Compression algorithms supported: 2017-12-24 19:40:47.485057 7faaab3d4e00 4 rocksdb: Snappy supported: 0 2017-12-24 19:40:47.485059 7faaab3d4e00 4 rocksdb: Zlib supported: 0 2017-12-24 19:40:47.485060 7faaab3d4e00 4 rocksdb: Bzip supported: 0 2017-12-24 19:40:47.485063 7faaab3d4e00 4 rocksdb: LZ4 supported: 0 2017-12-24 19:40:47.485064 7faaab3d4e00 4 rocksdb: ZSTD supported: 0 2017-12-24 19:40:47.485065 7faaab3d4e00 4 rocksdb: Fast CRC32 supported: 0 2017-12-24 19:40:47.485296 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-009981 2017-12-24 19:40:47.485331 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: 2017-12-24 19:40:47.485334 7faaab3d4e00 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2017-12-24 19:40:47.485338 7faaab3d4e00 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2017-12-24 19:40:47.485339 7faaab3d4e00 4 rocksdb: Options.compaction_filter: None 2017-12-24 19:40:47.485340 7faaab3d4e00 4 rocksdb: Options.compaction_filter_factory: None 2017-12-24 19:40:47.485341 7faaab3d4e00 4 rocksdb: Options.memtable_factory: SkipListFactory 2017-12-24 19:40:47.485342 7faaab3d4e00 4 rocksdb: Options.table_factory: BlockBasedTable 2017-12-24 19:40:47.485360 7faaab3d4e00 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559895cd01f0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x559895fbf570 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 2017-12-24 19:40:47.485364 7faaab3d4e00 4 rocksdb: Options.write_buffer_size: 268435456 2017-12-24 19:40:47.485365 7faaab3d4e00 4 rocksdb: Options.max_write_buffer_number: 4 2017-12-24 19:40:47.485366 7faaab3d4e00 4 rocksdb: Options.compression: NoCompression 2017-12-24 19:40:47.485367 7faaab3d4e00 4 rocksdb: Options.bottommost_compression: Disabled 2017-12-24 19:40:47.485368 7faaab3d4e00 4 rocksdb: Options.prefix_extractor: nullptr 2017-12-24 19:40:47.485369 7faaab3d4e00 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2017-12-24 19:40:47.485370 7faaab3d4e00 4 rocksdb: Options.num_levels: 7 2017-12-24 19:40:47.485371 7faaab3d4e00 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2017-12-24 19:40:47.485372 7faaab3d4e00 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2017-12-24 19:40:47.485373 7faaab3d4e00 4 rocksdb: Options.compression_opts.window_bits: -14 2017-12-24 19:40:47.485373 7faaab3d4e00 4 rocksdb: Options.compression_opts.level: -1 2017-12-24 19:40:47.485374 7faaab3d4e00 4 rocksdb: Options.compression_opts.strategy: 0 2017-12-24 19:40:47.485375 7faaab3d4e00 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2017-12-24 19:40:47.485376 7faaab3d4e00 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2017-12-24 19:40:47.485377 7faaab3d4e00 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2017-12-24 19:40:47.485378 7faaab3d4e00 4 rocksdb: Options.level0_stop_writes_trigger: 36 2017-12-24 19:40:47.485379 7faaab3d4e00 4 rocksdb: Options.target_file_size_base: 67108864 2017-12-24 19:40:47.485380 7faaab3d4e00 4 rocksdb: Options.target_file_size_multiplier: 1 2017-12-24 19:40:47.485407 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2017-12-24 19:40:47.485409 7faaab3d4e00 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2017-12-24 19:40:47.485411 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2017-12-24 19:40:47.485414 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2017-12-24 19:40:47.485415 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2017-12-24 19:40:47.485416 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2017-12-24 19:40:47.485416 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2017-12-24 19:40:47.485417 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2017-12-24 19:40:47.485418 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2017-12-24 19:40:47.485419 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2017-12-24 19:40:47.485420 7faaab3d4e00 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2017-12-24 19:40:47.485421 7faaab3d4e00 4 rocksdb: Options.max_compaction_bytes: 1677721600 2017-12-24 19:40:47.485422 7faaab3d4e00 4 rocksdb: Options.arena_block_size: 33554432 2017-12-24 19:40:47.485423 7faaab3d4e00 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2017-12-24 19:40:47.485423 7faaab3d4e00 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2017-12-24 19:40:47.485424 7faaab3d4e00 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2017-12-24 19:40:47.485425 7faaab3d4e00 4 rocksdb: Options.disable_auto_compactions: 0 2017-12-24 19:40:47.485426 7faaab3d4e00 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2017-12-24 19:40:47.485428 7faaab3d4e00 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2017-12-24 19:40:47.485429 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2017-12-24 19:40:47.485430 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2017-12-24 19:40:47.485430 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2017-12-24 19:40:47.485431 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2017-12-24 19:40:47.485432 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2017-12-24 19:40:47.485433 7faaab3d4e00 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2017-12-24 19:40:47.485434 7faaab3d4e00 4 rocksdb: Options.table_properties_collectors: 2017-12-24 19:40:47.485435 7faaab3d4e00 4 rocksdb: Options.inplace_update_support: 0 2017-12-24 19:40:47.485436 7faaab3d4e00 4 rocksdb: Options.inplace_update_num_locks: 10000 2017-12-24 19:40:47.485437 7faaab3d4e00 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2017-12-24 19:40:47.485438 7faaab3d4e00 4 rocksdb: Options.memtable_huge_page_size: 0 2017-12-24 19:40:47.485439 7faaab3d4e00 4 rocksdb: Options.bloom_locality: 0 2017-12-24 19:40:47.485439 7faaab3d4e00 4 rocksdb: Options.max_successive_merges: 0 2017-12-24 19:40:47.485440 7faaab3d4e00 4 rocksdb: Options.optimize_filters_for_hits: 0 2017-12-24 19:40:47.485441 7faaab3d4e00 4 rocksdb: Options.paranoid_file_checks: 0 2017-12-24 19:40:47.485441 7faaab3d4e00 4 rocksdb: Options.force_consistency_checks: 0 2017-12-24 19:40:47.485442 7faaab3d4e00 4 rocksdb: Options.report_bg_io_stats: 0 2017-12-24 19:40:47.490286 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-009981 succeeded,manifest_file_number is 9981, next_file_number is 9983, last_sequence is 1115879236, log_number is 0,prev_log_number is 0,max_column_family is 0 2017-12-24 19:40:47.490312 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 9978 2017-12-24 19:40:47.490410 7faaab3d4e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514126447490403, "job": 1, "event": "recovery_started", "log_files": [9982]} 2017-12-24 19:40:47.490417 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #9982 mode 0 2017-12-24 19:40:47.490436 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 9984 2017-12-24 19:40:47.491091 7faaab3d4e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514126447491088, "job": 1, "event": "recovery_finished"} 2017-12-24 19:40:47.491961 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x559895fd4100 2017-12-24 19:40:47.491990 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2017-12-24 19:40:47.529660 7faaab3d4e00 1 freelist init 2017-12-24 19:40:47.540795 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc opening allocation metadata 2017-12-24 19:40:58.564400 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc loaded 587 G in 1173199 extents 2017-12-24 19:40:58.645847 7faaab3d4e00 1 bluefs fsck 2017-12-24 19:40:58.645865 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck walking object keyspace 2017-12-24 20:27:01.456912 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking shared_blobs 2017-12-24 20:27:01.458261 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking for stray omap data 2017-12-24 20:27:07.297735 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking deferred events 2017-12-24 20:27:07.302892 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking freelist vs allocated 2017-12-24 20:27:18.627676 7faaab3d4e00 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_fsck(bool, bool)' thread 7faaab3d4e00 time 2017-12-24 20:27:18.625491 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 6122: FAILED assert(used_blocks.size() > count) ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55988ba70892] 2: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55988b933c8c] 3: (BlueStore::_mount(bool)+0x20a) [0x55988b93484a] 4: (OSD::init()+0x3e2) [0x55988b4a04e2] 5: (main()+0x2f07) [0x55988b3b21d7] 6: (__libc_start_main()+0xf0) [0x7faaa883c830] 7: (_start()+0x29) [0x55988b43d7f9] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -209> 2017-12-24 19:40:47.100859 7faaab3d4e00 5 asok(0x559895d252c0) register_command perfcounters_dump hook 0x559895cd41c0 -208> 2017-12-24 19:40:47.100888 7faaab3d4e00 5 asok(0x559895d252c0) register_command 1 hook 0x559895cd41c0 -207> 2017-12-24 19:40:47.100891 7faaab3d4e00 5 asok(0x559895d252c0) register_command perf dump hook 0x559895cd41c0 -206> 2017-12-24 19:40:47.100896 7faaab3d4e00 5 asok(0x559895d252c0) register_command perfcounters_schema hook 0x559895cd41c0 -205> 2017-12-24 19:40:47.100901 7faaab3d4e00 5 asok(0x559895d252c0) register_command perf histogram dump hook 0x559895cd41c0 -204> 2017-12-24 19:40:47.100903 7faaab3d4e00 5 asok(0x559895d252c0) register_command 2 hook 0x559895cd41c0 -203> 2017-12-24 19:40:47.100907 7faaab3d4e00 5 asok(0x559895d252c0) register_command perf schema hook 0x559895cd41c0 -202> 2017-12-24 19:40:47.100911 7faaab3d4e00 5 asok(0x559895d252c0) register_command perf histogram schema hook 0x559895cd41c0 -201> 2017-12-24 19:40:47.100920 7faaab3d4e00 5 asok(0x559895d252c0) register_command perf reset hook 0x559895cd41c0 -200> 2017-12-24 19:40:47.100922 7faaab3d4e00 5 asok(0x559895d252c0) register_command config show hook 0x559895cd41c0 -199> 2017-12-24 19:40:47.100926 7faaab3d4e00 5 asok(0x559895d252c0) register_command config help hook 0x559895cd41c0 -198> 2017-12-24 19:40:47.100929 7faaab3d4e00 5 asok(0x559895d252c0) register_command config set hook 0x559895cd41c0 -197> 2017-12-24 19:40:47.100933 7faaab3d4e00 5 asok(0x559895d252c0) register_command config get hook 0x559895cd41c0 -196> 2017-12-24 19:40:47.100936 7faaab3d4e00 5 asok(0x559895d252c0) register_command config diff hook 0x559895cd41c0 -195> 2017-12-24 19:40:47.100940 7faaab3d4e00 5 asok(0x559895d252c0) register_command config diff get hook 0x559895cd41c0 -194> 2017-12-24 19:40:47.100943 7faaab3d4e00 5 asok(0x559895d252c0) register_command log flush hook 0x559895cd41c0 -193> 2017-12-24 19:40:47.100946 7faaab3d4e00 5 asok(0x559895d252c0) register_command log dump hook 0x559895cd41c0 -192> 2017-12-24 19:40:47.100948 7faaab3d4e00 5 asok(0x559895d252c0) register_command log reopen hook 0x559895cd41c0 -191> 2017-12-24 19:40:47.100958 7faaab3d4e00 5 asok(0x559895d252c0) register_command dump_mempools hook 0x559895f88d68 -190> 2017-12-24 19:40:47.107308 7faaab3d4e00 0 set uid:gid to 64045:64045 (ceph:ceph) -189> 2017-12-24 19:40:47.107323 7faaab3d4e00 0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 60889 -188> 2017-12-24 19:40:47.114220 7faaab3d4e00 0 pidfile_write: ignore empty --pid-file -187> 2017-12-24 19:40:47.120897 7faaab3d4e00 0 load: jerasure load: lrc load: isa -186> 2017-12-24 19:40:47.120968 7faaab3d4e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel -185> 2017-12-24 19:40:47.120978 7faaab3d4e00 1 bdev(0x559895cd8b40 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block -184> 2017-12-24 19:40:47.121247 7faaab3d4e00 1 bdev(0x559895cd8b40 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational -183> 2017-12-24 19:40:47.121498 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 -182> 2017-12-24 19:40:47.121520 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 -181> 2017-12-24 19:40:47.121526 7faaab3d4e00 1 bdev(0x559895cd8b40 /var/lib/ceph/osd/ceph-0/block) close -180> 2017-12-24 19:40:47.422060 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0 -179> 2017-12-24 19:40:47.422496 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck repair (deep) start -178> 2017-12-24 19:40:47.422523 7faaab3d4e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel -177> 2017-12-24 19:40:47.422527 7faaab3d4e00 1 bdev(0x559895cd8d80 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block -176> 2017-12-24 19:40:47.422719 7faaab3d4e00 1 bdev(0x559895cd8d80 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational -175> 2017-12-24 19:40:47.422930 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 -174> 2017-12-24 19:40:47.422954 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 -173> 2017-12-24 19:40:47.423032 7faaab3d4e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel -172> 2017-12-24 19:40:47.423039 7faaab3d4e00 1 bdev(0x559895cd8fc0 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block -171> 2017-12-24 19:40:47.423192 7faaab3d4e00 1 bdev(0x559895cd8fc0 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational -170> 2017-12-24 19:40:47.423201 7faaab3d4e00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 1682 GB -169> 2017-12-24 19:40:47.423258 7faaab3d4e00 1 bluefs mount -168> 2017-12-24 19:40:47.484696 7faaab3d4e00 0 set rocksdb option compaction_readahead_size = 2097152 -167> 2017-12-24 19:40:47.484715 7faaab3d4e00 0 set rocksdb option compression = kNoCompression -166> 2017-12-24 19:40:47.484722 7faaab3d4e00 0 set rocksdb option max_write_buffer_number = 4 -165> 2017-12-24 19:40:47.484726 7faaab3d4e00 0 set rocksdb option min_write_buffer_number_to_merge = 1 -164> 2017-12-24 19:40:47.484729 7faaab3d4e00 0 set rocksdb option recycle_log_file_num = 4 -163> 2017-12-24 19:40:47.484732 7faaab3d4e00 0 set rocksdb option writable_file_max_buffer_size = 0 -162> 2017-12-24 19:40:47.484736 7faaab3d4e00 0 set rocksdb option write_buffer_size = 268435456 -161> 2017-12-24 19:40:47.484760 7faaab3d4e00 0 set rocksdb option compaction_readahead_size = 2097152 -160> 2017-12-24 19:40:47.484764 7faaab3d4e00 0 set rocksdb option compression = kNoCompression -159> 2017-12-24 19:40:47.484768 7faaab3d4e00 0 set rocksdb option max_write_buffer_number = 4 -158> 2017-12-24 19:40:47.484771 7faaab3d4e00 0 set rocksdb option min_write_buffer_number_to_merge = 1 -157> 2017-12-24 19:40:47.484774 7faaab3d4e00 0 set rocksdb option recycle_log_file_num = 4 -156> 2017-12-24 19:40:47.484777 7faaab3d4e00 0 set rocksdb option writable_file_max_buffer_size = 0 -155> 2017-12-24 19:40:47.484781 7faaab3d4e00 0 set rocksdb option write_buffer_size = 268435456 -154> 2017-12-24 19:40:47.484888 7faaab3d4e00 4 rocksdb: RocksDB version: 5.4.0 -153> 2017-12-24 19:40:47.484898 7faaab3d4e00 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ -152> 2017-12-24 19:40:47.484900 7faaab3d4e00 4 rocksdb: Compile date Nov 30 2017 -151> 2017-12-24 19:40:47.484902 7faaab3d4e00 4 rocksdb: DB SUMMARY -150> 2017-12-24 19:40:47.484926 7faaab3d4e00 4 rocksdb: CURRENT file: CURRENT -149> 2017-12-24 19:40:47.484928 7faaab3d4e00 4 rocksdb: IDENTITY file: IDENTITY -148> 2017-12-24 19:40:47.484962 7faaab3d4e00 4 rocksdb: MANIFEST file: MANIFEST-009981 size: 5579 Bytes -147> 2017-12-24 19:40:47.484970 7faaab3d4e00 4 rocksdb: SST files in db dir, Total Num: 34, files: 005983.sst 005984.sst 005985.sst 006019.sst 006020.sst 006021.sst 009951.sst 009952.sst 009953.sst -146> 2017-12-24 19:40:47.484973 7faaab3d4e00 4 rocksdb: Write Ahead Log file in db: 009982.log size: 0 ; -145> 2017-12-24 19:40:47.484974 7faaab3d4e00 4 rocksdb: Options.error_if_exists: 0 -144> 2017-12-24 19:40:47.484975 7faaab3d4e00 4 rocksdb: Options.create_if_missing: 0 -143> 2017-12-24 19:40:47.484976 7faaab3d4e00 4 rocksdb: Options.paranoid_checks: 1 -142> 2017-12-24 19:40:47.484976 7faaab3d4e00 4 rocksdb: Options.env: 0x5598961ba160 -141> 2017-12-24 19:40:47.484977 7faaab3d4e00 4 rocksdb: Options.info_log: 0x5598961ba9e0 -140> 2017-12-24 19:40:47.484978 7faaab3d4e00 4 rocksdb: Options.max_open_files: -1 -139> 2017-12-24 19:40:47.484979 7faaab3d4e00 4 rocksdb: Options.max_file_opening_threads: 16 -138> 2017-12-24 19:40:47.484979 7faaab3d4e00 4 rocksdb: Options.use_fsync: 0 -137> 2017-12-24 19:40:47.484980 7faaab3d4e00 4 rocksdb: Options.max_log_file_size: 0 -136> 2017-12-24 19:40:47.484981 7faaab3d4e00 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 -135> 2017-12-24 19:40:47.484983 7faaab3d4e00 4 rocksdb: Options.log_file_time_to_roll: 0 -134> 2017-12-24 19:40:47.484984 7faaab3d4e00 4 rocksdb: Options.keep_log_file_num: 1000 -133> 2017-12-24 19:40:47.484985 7faaab3d4e00 4 rocksdb: Options.recycle_log_file_num: 4 -132> 2017-12-24 19:40:47.484986 7faaab3d4e00 4 rocksdb: Options.allow_fallocate: 1 -131> 2017-12-24 19:40:47.484987 7faaab3d4e00 4 rocksdb: Options.allow_mmap_reads: 0 -130> 2017-12-24 19:40:47.484988 7faaab3d4e00 4 rocksdb: Options.allow_mmap_writes: 0 -129> 2017-12-24 19:40:47.484989 7faaab3d4e00 4 rocksdb: Options.use_direct_reads: 0 -128> 2017-12-24 19:40:47.484989 7faaab3d4e00 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 -127> 2017-12-24 19:40:47.484990 7faaab3d4e00 4 rocksdb: Options.create_missing_column_families: 0 -126> 2017-12-24 19:40:47.484991 7faaab3d4e00 4 rocksdb: Options.db_log_dir: -125> 2017-12-24 19:40:47.484992 7faaab3d4e00 4 rocksdb: Options.wal_dir: db -124> 2017-12-24 19:40:47.484992 7faaab3d4e00 4 rocksdb: Options.table_cache_numshardbits: 6 -123> 2017-12-24 19:40:47.484994 7faaab3d4e00 4 rocksdb: Options.max_subcompactions: 1 -122> 2017-12-24 19:40:47.484994 7faaab3d4e00 4 rocksdb: Options.max_background_flushes: 1 -121> 2017-12-24 19:40:47.484995 7faaab3d4e00 4 rocksdb: Options.WAL_ttl_seconds: 0 -120> 2017-12-24 19:40:47.484996 7faaab3d4e00 4 rocksdb: Options.WAL_size_limit_MB: 0 -119> 2017-12-24 19:40:47.484996 7faaab3d4e00 4 rocksdb: Options.manifest_preallocation_size: 4194304 -118> 2017-12-24 19:40:47.484997 7faaab3d4e00 4 rocksdb: Options.is_fd_close_on_exec: 1 -117> 2017-12-24 19:40:47.484998 7faaab3d4e00 4 rocksdb: Options.advise_random_on_open: 1 -116> 2017-12-24 19:40:47.484998 7faaab3d4e00 4 rocksdb: Options.db_write_buffer_size: 0 -115> 2017-12-24 19:40:47.484999 7faaab3d4e00 4 rocksdb: Options.access_hint_on_compaction_start: 1 -114> 2017-12-24 19:40:47.485000 7faaab3d4e00 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 -113> 2017-12-24 19:40:47.485000 7faaab3d4e00 4 rocksdb: Options.compaction_readahead_size: 2097152 -112> 2017-12-24 19:40:47.485001 7faaab3d4e00 4 rocksdb: Options.random_access_max_buffer_size: 1048576 -111> 2017-12-24 19:40:47.485001 7faaab3d4e00 4 rocksdb: Options.writable_file_max_buffer_size: 0 -110> 2017-12-24 19:40:47.485002 7faaab3d4e00 4 rocksdb: Options.use_adaptive_mutex: 0 -109> 2017-12-24 19:40:47.485003 7faaab3d4e00 4 rocksdb: Options.rate_limiter: (nil) -108> 2017-12-24 19:40:47.485004 7faaab3d4e00 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 -107> 2017-12-24 19:40:47.485004 7faaab3d4e00 4 rocksdb: Options.bytes_per_sync: 0 -106> 2017-12-24 19:40:47.485005 7faaab3d4e00 4 rocksdb: Options.wal_bytes_per_sync: 0 -105> 2017-12-24 19:40:47.485006 7faaab3d4e00 4 rocksdb: Options.wal_recovery_mode: 0 -104> 2017-12-24 19:40:47.485006 7faaab3d4e00 4 rocksdb: Options.enable_thread_tracking: 0 -103> 2017-12-24 19:40:47.485007 7faaab3d4e00 4 rocksdb: Options.allow_concurrent_memtable_write: 1 -102> 2017-12-24 19:40:47.485008 7faaab3d4e00 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 -101> 2017-12-24 19:40:47.485008 7faaab3d4e00 4 rocksdb: Options.write_thread_max_yield_usec: 100 -100> 2017-12-24 19:40:47.485009 7faaab3d4e00 4 rocksdb: Options.write_thread_slow_yield_usec: 3 -99> 2017-12-24 19:40:47.485010 7faaab3d4e00 4 rocksdb: Options.row_cache: None -98> 2017-12-24 19:40:47.485010 7faaab3d4e00 4 rocksdb: Options.wal_filter: None -97> 2017-12-24 19:40:47.485011 7faaab3d4e00 4 rocksdb: Options.avoid_flush_during_recovery: 0 -96> 2017-12-24 19:40:47.485012 7faaab3d4e00 4 rocksdb: Options.base_background_compactions: 1 -95> 2017-12-24 19:40:47.485012 7faaab3d4e00 4 rocksdb: Options.max_background_compactions: 1 -94> 2017-12-24 19:40:47.485013 7faaab3d4e00 4 rocksdb: Options.avoid_flush_during_shutdown: 0 -93> 2017-12-24 19:40:47.485014 7faaab3d4e00 4 rocksdb: Options.delayed_write_rate : 16777216 -92> 2017-12-24 19:40:47.485015 7faaab3d4e00 4 rocksdb: Options.max_total_wal_size: 0 -91> 2017-12-24 19:40:47.485016 7faaab3d4e00 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 -90> 2017-12-24 19:40:47.485017 7faaab3d4e00 4 rocksdb: Options.stats_dump_period_sec: 600 -89> 2017-12-24 19:40:47.485056 7faaab3d4e00 4 rocksdb: Compression algorithms supported: -88> 2017-12-24 19:40:47.485057 7faaab3d4e00 4 rocksdb: Snappy supported: 0 -87> 2017-12-24 19:40:47.485059 7faaab3d4e00 4 rocksdb: Zlib supported: 0 -86> 2017-12-24 19:40:47.485060 7faaab3d4e00 4 rocksdb: Bzip supported: 0 -85> 2017-12-24 19:40:47.485063 7faaab3d4e00 4 rocksdb: LZ4 supported: 0 -84> 2017-12-24 19:40:47.485064 7faaab3d4e00 4 rocksdb: ZSTD supported: 0 -83> 2017-12-24 19:40:47.485065 7faaab3d4e00 4 rocksdb: Fast CRC32 supported: 0 -82> 2017-12-24 19:40:47.485296 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-009981 -81> 2017-12-24 19:40:47.485331 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: -80> 2017-12-24 19:40:47.485334 7faaab3d4e00 4 rocksdb: Options.comparator: leveldb.BytewiseComparator -79> 2017-12-24 19:40:47.485338 7faaab3d4e00 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor -78> 2017-12-24 19:40:47.485339 7faaab3d4e00 4 rocksdb: Options.compaction_filter: None -77> 2017-12-24 19:40:47.485340 7faaab3d4e00 4 rocksdb: Options.compaction_filter_factory: None -76> 2017-12-24 19:40:47.485341 7faaab3d4e00 4 rocksdb: Options.memtable_factory: SkipListFactory -75> 2017-12-24 19:40:47.485342 7faaab3d4e00 4 rocksdb: Options.table_factory: BlockBasedTable -74> 2017-12-24 19:40:47.485360 7faaab3d4e00 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559895cd01f0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 1 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x559895fbf570 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 format_version: 2 -73> 2017-12-24 19:40:47.485364 7faaab3d4e00 4 rocksdb: Options.write_buffer_size: 268435456 -72> 2017-12-24 19:40:47.485365 7faaab3d4e00 4 rocksdb: Options.max_write_buffer_number: 4 -71> 2017-12-24 19:40:47.485366 7faaab3d4e00 4 rocksdb: Options.compression: NoCompression -70> 2017-12-24 19:40:47.485367 7faaab3d4e00 4 rocksdb: Options.bottommost_compression: Disabled -69> 2017-12-24 19:40:47.485368 7faaab3d4e00 4 rocksdb: Options.prefix_extractor: nullptr -68> 2017-12-24 19:40:47.485369 7faaab3d4e00 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr -67> 2017-12-24 19:40:47.485370 7faaab3d4e00 4 rocksdb: Options.num_levels: 7 -66> 2017-12-24 19:40:47.485371 7faaab3d4e00 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 -65> 2017-12-24 19:40:47.485372 7faaab3d4e00 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 -64> 2017-12-24 19:40:47.485373 7faaab3d4e00 4 rocksdb: Options.compression_opts.window_bits: -14 -63> 2017-12-24 19:40:47.485373 7faaab3d4e00 4 rocksdb: Options.compression_opts.level: -1 -62> 2017-12-24 19:40:47.485374 7faaab3d4e00 4 rocksdb: Options.compression_opts.strategy: 0 -61> 2017-12-24 19:40:47.485375 7faaab3d4e00 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 -60> 2017-12-24 19:40:47.485376 7faaab3d4e00 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 -59> 2017-12-24 19:40:47.485377 7faaab3d4e00 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 -58> 2017-12-24 19:40:47.485378 7faaab3d4e00 4 rocksdb: Options.level0_stop_writes_trigger: 36 -57> 2017-12-24 19:40:47.485379 7faaab3d4e00 4 rocksdb: Options.target_file_size_base: 67108864 -56> 2017-12-24 19:40:47.485380 7faaab3d4e00 4 rocksdb: Options.target_file_size_multiplier: 1 -55> 2017-12-24 19:40:47.485407 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_base: 268435456 -54> 2017-12-24 19:40:47.485409 7faaab3d4e00 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 -53> 2017-12-24 19:40:47.485411 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 -52> 2017-12-24 19:40:47.485414 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 -51> 2017-12-24 19:40:47.485415 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 -50> 2017-12-24 19:40:47.485416 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 -49> 2017-12-24 19:40:47.485416 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 -48> 2017-12-24 19:40:47.485417 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 -47> 2017-12-24 19:40:47.485418 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 -46> 2017-12-24 19:40:47.485419 7faaab3d4e00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 -45> 2017-12-24 19:40:47.485420 7faaab3d4e00 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 -44> 2017-12-24 19:40:47.485421 7faaab3d4e00 4 rocksdb: Options.max_compaction_bytes: 1677721600 -43> 2017-12-24 19:40:47.485422 7faaab3d4e00 4 rocksdb: Options.arena_block_size: 33554432 -42> 2017-12-24 19:40:47.485423 7faaab3d4e00 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 -41> 2017-12-24 19:40:47.485423 7faaab3d4e00 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 -40> 2017-12-24 19:40:47.485424 7faaab3d4e00 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 -39> 2017-12-24 19:40:47.485425 7faaab3d4e00 4 rocksdb: Options.disable_auto_compactions: 0 -38> 2017-12-24 19:40:47.485426 7faaab3d4e00 4 rocksdb: Options.compaction_style: kCompactionStyleLevel -37> 2017-12-24 19:40:47.485428 7faaab3d4e00 4 rocksdb: Options.compaction_pri: kByCompensatedSize -36> 2017-12-24 19:40:47.485429 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 -35> 2017-12-24 19:40:47.485430 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 -34> 2017-12-24 19:40:47.485430 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 -33> 2017-12-24 19:40:47.485431 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 -32> 2017-12-24 19:40:47.485432 7faaab3d4e00 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 -31> 2017-12-24 19:40:47.485433 7faaab3d4e00 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 -30> 2017-12-24 19:40:47.485434 7faaab3d4e00 4 rocksdb: Options.table_properties_collectors: -29> 2017-12-24 19:40:47.485435 7faaab3d4e00 4 rocksdb: Options.inplace_update_support: 0 -28> 2017-12-24 19:40:47.485436 7faaab3d4e00 4 rocksdb: Options.inplace_update_num_locks: 10000 -27> 2017-12-24 19:40:47.485437 7faaab3d4e00 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 -26> 2017-12-24 19:40:47.485438 7faaab3d4e00 4 rocksdb: Options.memtable_huge_page_size: 0 -25> 2017-12-24 19:40:47.485439 7faaab3d4e00 4 rocksdb: Options.bloom_locality: 0 -24> 2017-12-24 19:40:47.485439 7faaab3d4e00 4 rocksdb: Options.max_successive_merges: 0 -23> 2017-12-24 19:40:47.485440 7faaab3d4e00 4 rocksdb: Options.optimize_filters_for_hits: 0 -22> 2017-12-24 19:40:47.485441 7faaab3d4e00 4 rocksdb: Options.paranoid_file_checks: 0 -21> 2017-12-24 19:40:47.485441 7faaab3d4e00 4 rocksdb: Options.force_consistency_checks: 0 -20> 2017-12-24 19:40:47.485442 7faaab3d4e00 4 rocksdb: Options.report_bg_io_stats: 0 -19> 2017-12-24 19:40:47.490286 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-009981 succeeded,manifest_file_number is 9981, next_file_number is 9983, last_sequence is 1115879236, log_number is 0,prev_log_number is 0,max_column_family is 0 -18> 2017-12-24 19:40:47.490312 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 9978 -17> 2017-12-24 19:40:47.490410 7faaab3d4e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514126447490403, "job": 1, "event": "recovery_started", "log_files": [9982]} -16> 2017-12-24 19:40:47.490417 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:482] Recovering log #9982 mode 0 -15> 2017-12-24 19:40:47.490436 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/version_set.cc:2395] Creating manifest 9984 -14> 2017-12-24 19:40:47.491091 7faaab3d4e00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1514126447491088, "job": 1, "event": "recovery_finished"} -13> 2017-12-24 19:40:47.491158 7faaab3d4e00 5 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete db//MANIFEST-009981 type=3 #9981 -- OK -12> 2017-12-24 19:40:47.491168 7faaab3d4e00 5 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete db//009982.log type=0 #9982 -- OK -11> 2017-12-24 19:40:47.491961 7faaab3d4e00 4 rocksdb: [/build/ceph-12.2.2/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x559895fd4100 -10> 2017-12-24 19:40:47.491990 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 -9> 2017-12-24 19:40:47.529660 7faaab3d4e00 1 freelist init -8> 2017-12-24 19:40:47.540795 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc opening allocation metadata -7> 2017-12-24 19:40:58.564400 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc loaded 587 G in 1173199 extents -6> 2017-12-24 19:40:58.645847 7faaab3d4e00 1 bluefs fsck -5> 2017-12-24 19:40:58.645865 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck walking object keyspace -4> 2017-12-24 20:27:01.456912 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking shared_blobs -3> 2017-12-24 20:27:01.458261 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking for stray omap data -2> 2017-12-24 20:27:07.297735 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking deferred events -1> 2017-12-24 20:27:07.302892 7faaab3d4e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking freelist vs allocated 0> 2017-12-24 20:27:18.627676 7faaab3d4e00 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_fsck(bool, bool)' thread 7faaab3d4e00 time 2017-12-24 20:27:18.625491 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 6122: FAILED assert(used_blocks.size() > count) ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55988ba70892] 2: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55988b933c8c] 3: (BlueStore::_mount(bool)+0x20a) [0x55988b93484a] 4: (OSD::init()+0x3e2) [0x55988b4a04e2] 5: (main()+0x2f07) [0x55988b3b21d7] 6: (__libc_start_main()+0xf0) [0x7faaa883c830] 7: (_start()+0x29) [0x55988b43d7f9] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 1/ 1 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events --- 2017-12-24 20:27:18.630849 7faaab3d4e00 -1 *** Caught signal (Aborted) ** in thread 7faaab3d4e00 thread_name:ceph-osd ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (()+0xa65824) [0x55988ba2d824] 2: (()+0x11390) [0x7faaa98b6390] 3: (gsignal()+0x38) [0x7faaa8851428] 4: (abort()+0x16a) [0x7faaa885302a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55988ba70a1e] 6: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55988b933c8c] 7: (BlueStore::_mount(bool)+0x20a) [0x55988b93484a] 8: (OSD::init()+0x3e2) [0x55988b4a04e2] 9: (main()+0x2f07) [0x55988b3b21d7] 10: (__libc_start_main()+0xf0) [0x7faaa883c830] 11: (_start()+0x29) [0x55988b43d7f9] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2017-12-24 20:27:18.630849 7faaab3d4e00 -1 *** Caught signal (Aborted) ** in thread 7faaab3d4e00 thread_name:ceph-osd ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) 1: (()+0xa65824) [0x55988ba2d824] 2: (()+0x11390) [0x7faaa98b6390] 3: (gsignal()+0x38) [0x7faaa8851428] 4: (abort()+0x16a) [0x7faaa885302a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55988ba70a1e] 6: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55988b933c8c] 7: (BlueStore::_mount(bool)+0x20a) [0x55988b93484a] 8: (OSD::init()+0x3e2) [0x55988b4a04e2] 9: (main()+0x2f07) [0x55988b3b21d7] 10: (__libc_start_main()+0xf0) [0x7faaa883c830] 11: (_start()+0x29) [0x55988b43d7f9] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 1/ 1 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events --- 2017-12-24 20:27:39.630976 7f642f3a6e00 0 set uid:gid to 64045:64045 (ceph:ceph) 2017-12-24 20:27:39.630993 7f642f3a6e00 0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 132904 2017-12-24 20:27:39.637485 7f642f3a6e00 0 pidfile_write: ignore empty --pid-file 2017-12-24 20:27:39.644133 7f642f3a6e00 0 load: jerasure load: lrc load: isa 2017-12-24 20:27:39.644190 7f642f3a6e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2017-12-24 20:27:39.644197 7f642f3a6e00 1 bdev(0x5629b8dc2b40 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2017-12-24 20:27:39.644464 7f642f3a6e00 1 bdev(0x5629b8dc2b40 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational 2017-12-24 20:27:39.644704 7f642f3a6e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 2017-12-24 20:27:39.644726 7f642f3a6e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 2017-12-24 20:27:39.644731 7f642f3a6e00 1 bdev(0x5629b8dc2b40 /var/lib/ceph/osd/ceph-0/block) close 2017-12-24 20:27:39.966197 7f642f3a6e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0 2017-12-24 20:27:39.966574 7f642f3a6e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck repair (deep) start 2017-12-24 20:27:39.966609 7f642f3a6e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel 2017-12-24 20:27:39.966614 7f642f3a6e00 1 bdev(0x5629b8dc2d80 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2017-12-24 20:27:39.966806 7f642f3a6e00 1 bdev(0x5629b8dc2d80 /var/lib/ceph/osd/ceph-0/block) open size 1806155714560 (0x1a487451000, 1682 GB) block_size 4096 (4096 B) non-rotational 2017-12-24 20:27:39.967022 7f642f3a6e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes max 0.166667 < ratio 0.99 2017-12-24 20:27:39.967039 7f642f3a6e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.833333 kv 0.166667 data 1.49012e-08 2017-12-24 20:27:39.967126 7f642f3a6e00 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel